AI-responsive layout for cross-platform environments

Abstract

The method includes running a video game to generate video frames for presentation on a device of a first platform in response to gameplay by a user. [Solution] The method includes determining a target device for a second platform. The method includes mapping a video frame to the target device. The method includes determining a focal region within a scene in the video frame based on the game context. The method includes classifying assets within the focal region using a computer vision model that implements artificial intelligence. The method includes determining, using the computer vision model, that an asset is important to gameplay. The method includes determining that an asset in the mapped video frame does not meet a visibility threshold. The method includes modifying the mapped video frame so that the asset meets the visibility threshold.

Description

Technical Field

[0001] The present disclosure relates to modifying video game content when delivering video frames for display on cross-platform devices using artificial intelligence technologies such as computer vision.

Background Art

[0002] Video games and / or game applications, as well as their related industries (e.g., the video game industry), are extremely popular and account for a large proportion of the global entertainment market. Video games are played anytime and anywhere using various types of platforms, including game consoles, desktop computers, laptop computers, mobile phones, and the like.

[0003] A user may play a video game designed for presentation on a specific platform without issues, but is playing the game on another more restricted platform, such as one with a smaller display. For example, a video game may be running on a game console to display video frames on a large-screen television or screen monitor, but in practice, the user is viewing the gameplay on a mobile device (e.g., a tablet, a phone, etc.). This may occur when the video game is running on a game console and the video frames are transmitted to a mobile device for display, or when the video frames are executed on a mobile device for display. In these cases, the content within the video frames may be compromised, such that one or more objects may not be displayed in a manner satisfactory to the user, including instances where the user cannot easily view the content or part of the content because the content is excessively scaled down to fit the screen of the mobile device.

[0004] Embodiments of the present disclosure arise under such circumstances.

Summary of the Invention

[0005] Embodiments of this disclosure relate to cross-platform modification of content using artificial intelligence, such that critical assets meet visibility requirements. Identifying issues when displaying content on cross-platform devices using artificial intelligence can be done in real time during video game development or during video game gameplay.

[0006] In one embodiment, a method is disclosed. The method includes running a video game to generate video frames for presentation on a first device on a first platform, the video frames being generated for gameplay of the video game by a user. The method includes determining a target device, the target device being on a second platform. The method includes mapping the video frames from the first device to the target device, the mapped video frames being presented on the target device. The method includes determining a focal region within the gameplay scene presented in the video frame based on the game context of the scene in the gameplay. The method includes classifying assets within the focal region using a computer vision model that implements artificial intelligence. The method includes determining, using the computer vision model, that an asset is important in the gameplay of the video game. The method includes determining that an asset in the video frame mapped to the target device does not meet a visibility threshold. The method includes modifying the mapped video frame so that the asset meets the visibility threshold.

[0007] In another embodiment, a non-temporary computer-readable medium is disclosed that stores a computer program for execution by a processor to perform a method. The computer-readable medium includes program instructions for running a video game to generate video frames for presentation on a first device of a first platform, the video frames including program instructions for execution that are generated for gameplay of the video game by a user. The computer-readable medium includes program instructions for determining a target device, the target device being of a second platform. The computer-readable medium includes program instructions for mapping video frames from a first device to a target device, the mapped video frames including program instructions for mapping that are presented on the target device. The computer-readable medium includes program instructions for determining a focal region in a gameplay scene presented in a video frame based on the game context of the scene in the gameplay. The computer-readable medium includes program instructions for classifying assets in the focal region using a computer vision model that implements artificial intelligence. The computer-readable medium includes program instructions for determining, using the computer vision model, that an asset is important in the gameplay of the video game. The computer-readable medium includes program instructions for determining whether an asset in a video frame mapped to a target device does not meet the visibility threshold. The computer-readable medium also includes program instructions for modifying the mapped video frame so that the asset meets the visibility threshold.

[0008] In yet another embodiment, a computer system is disclosed, the computer system comprising a processor and a memory coupled to the processor, which stores instructions, the instructions, when executed by the computer system, cause the computer system to execute a method. The method includes executing a video game to generate video frames for presentation on a first device of a first platform, the video frames being generated for gameplay of the video game by a user. The method includes determining a target device, the target device being of a second platform. The method includes mapping video frames from a first device to a target device, the mapped video frames being presented on the target device. The method includes determining a focal region in a gameplay scene presented in a video frame based on the game context of the scene in the gameplay. The method includes classifying assets in the focal region using a computer vision model that implements artificial intelligence. The method includes determining, using the computer vision model, that an asset is important in the gameplay of the video game. The method includes determining that an asset in a video frame mapped to a target device does not meet a visibility threshold. The method involves modifying the mapped video frames so that the asset meets the visibility threshold.

[0009] Other aspects of this disclosure will become apparent from the following embodiments for carrying out the invention, which are shown as examples of the principles of this disclosure, together with the accompanying drawings.

[0010] This disclosure can be best understood by referring to the following description in conjunction with the attached drawings. [Brief explanation of the drawing]

[0011] [Figure 1]This disclosure describes a system configured for cross-platform modification of content using artificial intelligence (AI) according to one embodiment of this disclosure. [Figure 2] This flowchart illustrates a method for cross-platform modification of content, according to one embodiment of the present disclosure, using an AI model configured to identify key assets within video content and ensure that the display of those key assets meets visibility requirements. [Figure 3] This data flow diagram illustrates cross-platform modification of content using an AI model when streaming video content to a client device over a network and / or delivering video content locally to a client device, according to embodiments of the present disclosure. [Figure 4] This figure shows a platform transformation engine that implements an AI model configured to classify and identify critical assets within video content and modifications to the display of those assets to meet visibility requirements in a cross-platform environment, according to one embodiment of the present disclosure. [Figure 5A] This invention illustrates the mapping and / or scaling of image frames for display on a target device according to one embodiment of the present disclosure. [Figure 5B] This invention illustrates the transformation of the image frame introduced in Figure 5A for display on a target device, according to one embodiment of this disclosure. [Figure 6] The following are exemplary device components that may be used to perform various embodiments of the present disclosure. [Modes for carrying out the invention]

[0012] The following detailed description includes many specific details for illustrative purposes, but those skilled in the art will understand that many variations and modifications of the following details are within the scope of this disclosure. Thus, aspects of this disclosure are shown without any loss of generality to the claims that follow this description, and without imposing any limitations on those claims.

[0013] Generally speaking, various embodiments of this disclosure describe systems and methods for modifying video game content when delivering video frames for display on cross-platform devices using artificial intelligence (AI) techniques such as computer vision. The overall process may include, in no particular order, one or more of the following: scaling video frames to fit the display of the target platform (e.g., dumb scaling, scaling everything to a smaller size); analyzing the scaled video frames to determine whether assets are visible; transforming assets that would be visually impaired if they are the focus of the game context; and making field of view (FOV) changes as necessary (e.g., dumb zooming to a focal region surrounding the assets). In one embodiment, the content is modified at runtime by adjusting the display of the content for the detected platform. For example, the display of the content dynamically adapts based on the platform on which it is being viewed (i.e., a cross-platform platform different from the game design platform on which the video frames are generated during the video game's execution). For example, if the content is being accessed on a mobile device rather than a full display, the layout and format are adjusted to fit the smaller screen size to ensure an optimal user experience. This adjustment is made in real time as the platform is detected. In particular, computer vision implemented with AI is used to identify layout issues when content is displayed on different platforms or across platforms. That is, computer vision technology is used to analyze layouts and detect any potential problems that may arise when content is displayed across platforms based on visibility thresholds.By leveraging computer vision algorithms, the system can identify inconsistencies such as misplaced elements, duplicate content, or improper scaling that may arise due to variations in screen size or resolution across different platforms or cross-platform. For example, video content may be scaled or mapped to cross-platform devices for display (e.g., by performing dumb scaling). Assets within video content are analyzed to determine whether they are the focus of the game context (i.e., whether they are important to gameplay based on the game context) and whether they meet the visibility threshold. When an asset is determined to be important within the game context, transformations are performed on the asset (e.g., a different asset, a different format of the asset, modifications to the asset, etc.) so that its display meets the visibility threshold. One example is field of view (FOV) adjustment (e.g., making the FOV surrounding the asset smaller and / or focusing on the FOV), such as when performing dumb zoom. In another embodiment, a dedicated toolset is provided to developers to test and evaluate various aspects of the layout to cross-platform devices, including functionality, compatibility, and visual presentation. In particular, computer vision with AI is used to identify layout issues when content is displayed on different platforms. These developer tools help verify that layouts behave as intended, function correctly across different platforms or devices, and adhere to design specifications. They may include, among other features, layout previews, code debugging, simulation in different environments, and performance metric collection.

[0014] The advantages of a method and system configured to perform cross-platform modification of content using artificial intelligence based on visibility requirement thresholds include the automatic identification of layout issues related to content displayed on cross-platform devices. Additional advantages include the automatic identification of critical assets within areas of the layout that have presentation issues on cross-platform devices. Further advantages include the automatic modification of content to ensure visibility of critical assets to the user. In this way, users can view content (e.g., from video games) on cross-platform devices even if the content was initially designed and generated to be displayed on another device, such as a display device.

[0015] Throughout this specification, references to “game,” “video game,” or “game application” mean any type of interactive application directed through the execution of input commands. For descriptive purposes only, interactive applications include applications for games, document processing, video processing, video game processing, and so on. Furthermore, the terms “virtual world,” “virtual environment,” or “metaverse” mean any type of environment generated by one or more corresponding applications for interaction between multiple users in a multiplayer session or multiplayer game session. Additionally, the terms introduced above are interchangeable.

[0016] Based on the above-mentioned general understanding of various embodiments, illustrative details of the embodiments will be described here with reference to various drawings.

[0017] Figure 1 shows a system 100 configured for cross-platform modification of content using artificial intelligence, according to one embodiment of the present disclosure. The system includes a platform transformation engine 120 which includes an AI model 170 that provides classification and / or identification of critical assets in one or more video frames and any problems that may arise when presenting those critical assets on cross-platform devices.

[0018] As illustrated, System 100 may provide a game to one or more client devices 110 via Network 150. In particular, System 100 may be configured to enable interaction with interaction applications, including providing a game to users who are participating in single-player or multiplayer game sessions via Cloud Game Network 190 (e.g., participating in a video game in single-player or multiplayer mode, or participating in a metaverse generated by other users and applications). The game may run locally (e.g., on a corresponding user's local client device) or remotely from a corresponding client device 110 (e.g., acting as a thin client) of a corresponding user playing a video game, according to one embodiment of the present disclosure. In at least one capability, Cloud Game Network 190 supports a multiplayer game session for a group of users, which includes distributing and receiving player game data for purposes such as coordinating and / or aligning the actions of objects and players within a scene in the game world or metaverse, and managing communication between users so that users in distributed locations participating in the multiplayer game session can interact with each other in real time within the game world or metaverse. In another capability, Cloud Gaming Network 190 supports multiple users participating in the metaverse.

[0019] In one embodiment, the cloud game network 190 may support artificial intelligence (AI)-based services, including a chatbot service (e.g., ChatGPT®) that provides one or more features such as conversational communication, document composition, music composition, answering questions, chat room simulation, and game play.

[0020] The user accesses the remote service using a client device 110 which includes at least a CPU, display, and input / output (I / O). For example, the user may access the cloud game network 190 via a communication network 150 using a corresponding client device 110 configured to provide input control, update the session controller (e.g., distribute and / or receive user game state data), receive streaming media, etc. The client device 110 may be a personal computer (PC), a mobile phone, a personal digital assistant (PAD), a handheld device, etc.

[0021] In one embodiment, as previously described, the client device 110 may be configured with a locally stored game title processing engine 111 and game logic 115 (e.g., executable code) for at least some local processing of the application, and may also be used to receive streaming content generated by an application running on the server, or for other content provided by backend server support.

[0022] In another embodiment, the client device 110 may be configured as a thin client that provides an interface with a backend server (e.g., the game server 160 of the cloud game network 190) configured to provide computing functionality (including, for example, the game logic 115 that implements the corresponding application, i.e., the game title processing engine 111 that executes executable code). In particular, the client device 110 of the corresponding user is configured to request access to the application via the communication network 150 such as the Internet, and is configured to render the display image generated by the video game executed by the game server 160, and the encoded image is delivered (i.e., streamed) to the client device 110 for display. For example, a user may be interacting with an instance of an application running on the game processor of the game server 160 through the client device 110 using input commands to drive game play. The client device 110 may receive input from various types of input devices such as a game controller, a tablet computer, a keyboard, gestures captured by a video camera, a mouse, a touch pad, and an audio input.

[0023] Furthermore, the system 100 includes a platform conversion engine 120 that incorporates an AI model 170. The AI model 170 is configured to classify and / or identify key assets presented in video frames generated through execution of an application (e.g., a video game), and determine whether a visibility issue exists when those assets are presented on a cross-platform device, for example, when those assets do not meet a visibility threshold or visibility requirement. If there is an identified issue in the presentation of the video frames, a video frame modification engine 360 is configured to modify content within the video frames such that the key assets meet the visibility threshold requirement. The platform conversion engine 120 may be implemented in a backend cloud gaming network, or in some embodiments, may be located on the client device 110. The classification and / or identification and / or prediction of key assets and any visibility issues of those assets may be performed using artificial intelligence (AI) via an AI layer. For example, the AI layer may be implemented via the AI model 170 executed by the deep / machine learning engine 195 of the platform conversion engine 120.

[0024] In the detailed description of the system 100 of Figure 1, the flowchart 200 of Figure 2 discloses a method for cross-platform modification of content using an AI model configured to classify and / or identify key assets in video content and ensure that the presentation of those key assets meets visibility requirements. The operations performed in the flowchart may be implemented by one or more of the components of the entities described above, and further by the system 100 described in Figure 1 including the platform conversion engine 120.

[0025] In 210, the method includes running a video game to generate video frames for presentation on a first device of a first platform, the video frames being generated for gameplay of the video game by a user. As previously stated, the video game may run on a local device (for example, in the case of a game in single-player or multiplayer mode) or run over a cloud gaming network for a single-player or multiplayer game session. More specifically, the video frames are generated based on a pre-selected platform device so that the video game is designed to run on a pre-selected platform (for example, a game console, personal computer, tablet, handheld device, etc.). For example, the video game may be designed to run on a first platform, such as running on a game console, displaying the video frames on a full-screen display to give the user the best experience. However, the video frames may be generated using a first platform (for example, a game console), but the video frames may be presented on different target display devices (i.e., cross-platform devices) on different platforms, such as a tablet or mobile phone, which are smaller than they would be displayed on a first platform (for example, a full display). In other embodiments, video frames may be generated and displayed on a target platform, such as a handheld device, or across platforms. Further other combinations of devices on one or more platforms for execution and display are supported. Although embodiments of this disclosure are described in the context of video games, it should be understood that cross-platform content modification may be performed on content generated from any type of application.

[0026] In 220, the method includes determining a target device to be used to display the generated video frame, wherein the target device is of a second platform or cross-platform. In particular, the video frame is generated for display on a first platform (i.e., objects are fully positioned and presented within the video frame assuming display on a first platform device), but the video frame is instead displayed on a device of a second platform. That is, even if a game is designed for display on a full-screen display (i.e., a first platform), it should be understood that a user may play or watch the video game on a different device or cross-platform device, such as a mobile phone, which has a different viewing area, which in this example is a smaller viewing area (i.e., a smaller screen space). Thus, a user is playing and watching a video game that is displayed on a target device of a platform different from the first platform on which the video game was designed to display the content.

[0027] In paragraph 230, the method includes mapping video frames from a first device to a target device (i.e., a cross-platform device), wherein the mapped video frames are presented on the target device. Generally, there may be some limitations when presenting video frames to a target device, which can be addressed or at least considered when mapping video frames to the target device. Therefore, tools may be used when performing cross-device mapping to avoid disadvantages for users on different platforms. For example, a game designed for a game console (i.e., controller input, console execution, and presentation on a full display) may require adjustments when performing mapping to a target device or a cross-platform device (e.g., PC, handheld device, etc.) to make the game easier and more intuitive to play on a cross-platform device (e.g., a PC platform using a keyboard to provide input commands, or a handheld device using a touchscreen to provide input commands). Thus, the content presented in the interface with the mapped video frames and / or their content presented on a cross-platform display may differ from the content presented on the platform for which it was designed or the first platform (e.g., a full display).

[0028] For example, when performing mapping, several considerations may need to be addressed, such as how input is performed for different platform devices (e.g., a toggle on a controller on the first platform may be performed by a motion across a touchscreen on the second platform). Furthermore, another consideration when mapping between devices is the amount of information (e.g., objects) that should be included on each platform device. For instance, under a first platform designed for a video game, all information is typically included, but cross-platform devices may have smaller displays, and therefore, by design, or because cross-platform devices cannot display all the information of a video frame at once, less information may be available. Additionally, a user playing on a first-platform device may typically use headphones (e.g., assuming execution on a game console and presentation on a full display). On the other hand, a user of a cross-platform device (e.g., a mobile phone) typically does not use headphones and instead relies on the built-in speaker of the cross-platform device, which may be inadequate in some way (e.g., not loud enough outdoors or in public). Therefore, when presented on a mobile device, the mapped video frame may include indicators on the screen, which provide visual information that would normally be heard using headphones.

[0029] In some embodiments, mapping may include simple scaling or dumb scaling. In this way, when mapping, the entire content or information within the corresponding video frame is included but scaled down to fit the viewing screen of the target display device or cross-platform display device. However, there are problems that can arise when performing dumb scaling. For example, one problem may be distortion of the game space, such as the correlated spacing between objects being changed beyond the user's expectations (i.e., objects appearing out of place) or the correlated size between objects appearing unnatural (i.e., objects appearing larger than usual compared to other objects in the scene). Furthermore, dumb scaling may present the content of the corresponding video frame at a size that makes it impossible for the user to play the game (i.e., objects are too small to see).

[0030] In 240, the method includes determining a focal region within a scene of gameplay presented in a video frame based on the game context of the scene in one or more video frames. When generating a video frame for the corresponding gameplay at a particular point in time, game state data generated during the execution of the video game is used to determine the game context of the gameplay. For example, the game state data may include the state of the running video game, and the game state data enables the generation of the game environment at the corresponding point in time of gameplay. The game state data may include the state of the devices used to render the gameplay (e.g., state of the CPU, GPU, memory, register values, etc.), identification of the executable code for running the video game at that point in time, game characters, game objects, object and / or game attributes, graphic overlays, and other information. User-saved data may also be analyzed to determine the game context, and user data personalizes the video game for the corresponding user, such as character information or attributes for personalizing characters, character loadouts that make characters unique to the user, and usage profile data. Therefore, game state data, and optionally user data, may be used to determine the game context of gameplay, such as the user's location within the game, the type of game, the atmosphere of the game, the game's rating (e.g., proficiency level), the number of other players in the game environment, the game dimension displayed, which players are playing a particular game session, descriptive information, game title, version of the game title, franchise, game title distribution format, downloaded content accessed, links, credits, achievements, rewards, trophies, and other information.

[0031] Based on the game context, the focal area within the scene of a corresponding video frame can be determined. For example, the focal area may define the video frame and / or the part of the scene presented in the video frame that the user's attention is directed to. The focal area may be aligned with the focus of user interaction within the scene. For example, when a user is fighting a boss in a video game, the focal area may include the surrounding area surrounding the boss and the user's character. That is, during combat, the user's attention is directed to the boss and / or the user's character. Other parts and / or areas of the scene (e.g., surrounding area, background, etc.) may be less important to the user, and therefore the user pays little to no attention to those areas outside the focal area.

[0032] In 250, the method includes classifying assets within a focal region using a computer vision model that implements artificial intelligence (AI). In particular, the computer vision model may be trained using known assets (e.g., objects, information, content, etc.) found in a video game, and the training is performed using AI techniques. For example, the computer vision model may be trained using a list of assets in a video game. Furthermore, the training may include being presented with lists of assets in different game environments. Thus, the computer vision model that implements artificial intelligence may analyze a scene or part of a scene (e.g., a focal region of a scene) generated for a corresponding video frame of a video game to identify and / or classify assets within the scene or part of a scene.

[0033] In 260, the method includes using a computer vision model to determine whether an asset is important to the gameplay of a video game. For example, the computer vision model may analyze one or more of the following: game state data, user data, and / or game context, to determine whether the asset has measurable importance that indicates it should be available to the user, such as how to provide an interaction (e.g., viewing, manipulating, etc.). That is, when an asset is not important, whether or not it is available to the user has little or no bearing on the outcome of the video game's gameplay. On the other hand, when an asset is important, whether or not it is available to the user affects the outcome of the gameplay. Furthermore, the outcome of interactions with assets deemed important also affects the outcome of the gameplay. In other words, if an asset is determined to be important, it must be available to the user for interaction. For example, a weapon available for pickup that is essential for defeating a boss during combat may be determined to be important and should be available for user interaction.

[0034] In 270, the method includes determining whether an asset in a video frame mapped to a target device does not meet or falls short of a visibility threshold. The asset is determined to be important and therefore must be made available to the user. In one embodiment, the visibility threshold defines whether the asset is available to the user. Furthermore, the visibility threshold may define whether the availability to the user is meaningful enough to be of value to the user. That is, the visibility threshold may define whether the asset is sufficiently visible to be available for meaningful interactions (e.g., viewing, manipulating, understanding, etc.) to the user. For example, the visibility threshold may not be met if the asset is visible to the user, but is so small that the user cannot understand what the asset is, how the asset moves within the game environment, how the asset interacts with other assets within the game environment, or how to manipulate the asset, etc.

[0035] In some embodiments, computer vision employing artificial intelligence (i.e., an AI model) is used to determine whether one or more assets within a video frame mapped to the display screen of a target platform meet a visibility threshold. In particular, computer vision techniques may be employed to identify any potential problems with displaying the content of corresponding video frames in a new layout (i.e., based on the mapping) on ​​devices of different platforms, and whether those problems lead to any deficiencies in meeting the visibility threshold. For example, by leveraging computer vision algorithms, inconsistencies when using a new layout can be identified, such as cropping resulting in missing elements or assets due to variations in screen size or resolution between different platforms, misaligned elements, duplicated content, improper scaling, and pixelation.

[0036] Computer vision can be used to identify missing or cropped assets when mapping to a new layout. For example, a computer vision algorithm can identify what is missing between the content presented in the original video frame and the content presented in the mapped video frame (e.g., by testing different screen layouts). By testing content presented on different screens and / or at different resolutions and / or in atypical or limited dimensions, a computer vision algorithm can detect whether any important part of a layout (e.g., assets containing essential characters) is missing or hidden (i.e., cropped). Computer vision can also be used to determine if there were any issues that led to the video frame not meeting visibility thresholds when scaling to a new layout. For example, scaling issues may include distortion of the game environment. Computer vision can also be used to identify any pixel issues that affect the overall visual quality and / or clarity of content, such as pixelation, blurring, and artifacts. For example, a computer vision algorithm can be employed to identify instances of pixelation and their locations on cross-platform screens by analyzing the visual characteristics of the content. Pixelation refers to the appearance of visible pixel blocks or distortions within an image or graphic.

[0037] In 280, the method includes modifying and / or transforming mapped video frames so that the asset meets a visibility threshold. Transformations may include one or more of the following: scaling or resizing the asset, dumb scaling of video frames, pixel modification, field of view (FOV) changes, changes to different content, or changes to different formats. Furthermore, the mapped and modified or transformed video frames are delivered to the target device.

[0038] For example, the transformation may include rescaling the asset to a larger size and then overlaying the rescaled asset within a video frame mapped to the display device of the target platform. In another embodiment, the transformation may include scaling or dumb scaling. For example, the entire video frame is scaled to fit the display device of the target platform so that the video frame is rescalable to a smaller size (e.g., dumb scaling).

[0039] Furthermore, the transformation may include corrective measures to ensure that assets identified as cropped are displayed on the target platform's screen (i.e., meet the visibility threshold). For example, the assets may be included as overlays within the mapped video frame. In another embodiment, the transformation may include pixel adjustment and / or movement to address pixel issues (e.g., pixelation, blurring, artifacts, etc.) to improve the overall visual quality and clarity of the content. For example, instances of pixelation identified by AI (i.e., the appearance of visible pixel blocks or distortions in an image or graphic) may be corrected by applying image processing techniques, such as resolution adjustment.

[0040] In yet another embodiment, the transformation may include changing the scene's FOV to a focal region and restricting the modified video frames to the modified FOV. The adjustment to the FOV may be based on the game context (i.e., identifying which assets are important or focal regions relevant to the game context) and / or the characteristics of the target platform's display device. In some embodiments, changing the scene's FOV may include dumb zooming to a focal region or assets determined to be important.

[0041] In one embodiment, a general process for modifying and / or transforming content may include, in no particular order, one or more of the following: scaling or mapping video frames to fit the display of the target platform (e.g., dumb scaling, scaling everything to a smaller size); analyzing the scaled video frames to determine whether assets are visible (i.e., whether assets meet the visibility threshold); transforming assets that are visually impaired (i.e., do not meet the visibility threshold) when the assets are the focus of the game context (i.e., important within the game context) (e.g., performing further scaling, resizing, changing the format of the assets, using different assets, etc.); and performing field of view (FOV) changes (e.g., dumb zoom to the focus area, FOV scaling / targeting / zooming, etc.) as needed to enhance the asset transformation (e.g., dumb zoom to the focus area surrounding the assets).

[0042] In one embodiment, the process for modifying and / or transforming content is performed in real time, such as while playing a video game, and the content is modified and / or transformed for the display of the detected target platform. This process is adaptable for use during video game development. Identification of issues with content display can be performed using AI trained to perform quality assurance (QA) and / or audit testing to quickly detect and highlight any problems or inconsistencies in newly created layouts or layouts targeted at specific cross-platforms. Testing and / or analyzing content display on target platforms can be performed against cloud streaming or against local execution in real time to dynamically adapt content display based in part on the platform on which the content is being viewed and visibility thresholds to ensure that the layout meets predefined design criteria, guidelines, or compatibility requirements, enabling efficient identification and resolution of potential issues. For example, if content is accessed on a mobile device on the target platform, the layout and formatting can be adjusted to fit smaller screen sizes to ensure an optimal user experience.

[0043] In another embodiment, the process for modifying and / or transforming content is performed during the development of a video game to be played under simulation. This process is adaptable for real-time execution use of the video game for gameplay. More specifically, the service may be provided as a dedicated toolset for developers to test and evaluate various aspects of the layout, including functionality, compatibility, and visual presentation. The tools may be used to help identify compliance of the visibility of assets within video frames generated during simulated gameplay of the video game and presented on one or more cross-platform devices. These developer tools help verify that the layout behaves as intended, functions properly across different platforms or devices, and conforms to design specifications. They may include, among other things, features for previewing the layout, debugging code, simulating different environments, and collecting performance metrics.

[0044] In particular, video frames may be generated using AI-driven simulations of running video games, such as using an AI persona to play the video game. That is, a computer-controlled virtual character or entity is used to simulate gameplay scenarios and interact with game content. By using an AI persona, developers can identify potential problems (e.g., gameplay imbalances, glitches, or graphical inconsistencies) when mapping to new layouts for cross-platform devices by observing how the AI ​​persona navigates the game world and interacts with various elements, mechanics, or visual aspects of the video game. This allows developers to identify and address any problems or inconsistencies in the layout design for cross-platform devices. For example, simulated gameplay by an AI persona provides a preview of the content layout on cross-platform devices before the video game ends. In particular, the content of video frames of the corresponding video game running under simulation may be mapped to new layouts for each platform device for display, and the content in the new layouts is analyzed and previewed to identify any problems for any of the platforms. For example, identifying issues with content display can be done using AI, employing computer vision trained to perform quality assurance (QA) and / or audit testing to quickly identify issues with new layouts or layouts generated for different platforms. By visualizing content layouts across cross-platform devices, developers can ensure that all elements are properly aligned, sized, and styled to achieve desired aesthetic and functional results.

[0045] Figure 3 is a data flow diagram 300 illustrating cross-platform modification of content using AI model 170 when streaming video content to client devices over a network and / or when delivering video content locally to client devices, according to an embodiment of the present disclosure. The operations performed in the flow diagram may be carried out by one or more of the components of the entities described above, and furthermore, by the system 100 described in Figure 1, which includes the platform translation engine 120. Cross-platform modification of content may be performed on the cloud game server when streaming the modified content to cross-platform devices, or it may be performed locally when delivering the modified content to cross-platform devices.

[0046] As illustrated, the game title processing engine 111 executes the game logic of the video game to generate multiple video frames 305 and game state data 303 in relation to the user's gameplay (e.g., a single-player or multiplayer game). As previously mentioned, the game title processing engine may be located remotely on a cloud game server for streaming video frames to the user over a network (e.g., Scenario 1), or it may be configured for local processing, such as when the user is participating in a multiplayer game session, which involves local execution of the video game in conjunction with a backend server that partially provides game updates from other players during the game session (e.g., Scenario 2).

[0047] A running video game is formatted for presentation on a display device of a first platform, but generates video frames that are delivered and presented on cross-platform devices. For example, a video frame may be formatted for a game platform (e.g., presented on a full-dimensional display), but the user is playing the video game on a cross-platform device such as a mobile phone. Because there may be visual difficulties in directly displaying video frames on cross-platform devices, the platform conversion engine 120 is configured to format the content within the video frame for presentation on cross-platform devices. In particular, the platform conversion engine receives and analyzes multiple video frames 305 and game state data 303 using artificial intelligence that implements computer vision to identify cross-platform presentation problems and modify the content so that the display of important assets meets visibility thresholds.

[0048] The platform transformation engine 120 includes a deep / machine learning engine 195 that implements an AI model 170 that identifies and / or classifies important assets within the content of video frames and determines if there are visibility issues with those assets when presenting them on cross-platform devices. The AI ​​model 170 may implement computer vision techniques to analyze visual input (e.g., video frames) to identify and / or classify important assets and to identify any problems when presenting those assets on cross-platform devices. Furthermore, the AI ​​model 170 may work in conjunction with one or more components configured to generate information related to video frames 305 and corresponding game state data 303, and their presentation on cross-platform devices. One or more components may generate corresponding information, partly based on the analysis of video frames 305 and / or game state data 303.

[0049] For example, the game asset library 320 provides a list of assets for a corresponding video game, which can be used to train the AI ​​model 170 to classify assets found in various scenes of the video game presented in the video frame, and / or for asset classification and / or validation. The game context analyzer 325 generates information corresponding to the game context of the scene presented in the video frame 305. The target device identifier 330 generates information identifying the target cross-platform device used to display the user's gameplay. The mapper 335 is configured to map the scene in the video frame 305 to the target cross-platform device. For example, the video game and / or execution platform may have a pre-designed mapping for displaying the video frame 305 on the target cross-platform device. Exemplarily, the mapping may include dumb scaling or content cropping to fit the entire content in the video frame 305 onto the display of the target cross-platform device. The focus area identifier 340 generates information corresponding to the location of the scene in the video frame that is the focus of the user's gameplay, based on the identified game context. This may include a defined region within each of the video frames 305 that is the focus of the gameplay. The scene asset identifier 345 is configured to generate information related to known or predicted assets within the scene in the video frame. Furthermore, the scene asset identifier 345 may be configured to identify assets within previously identified gameplay focus regions. The asset evaluator 350 also generates information about the importance of corresponding assets, such as assets found within the gameplay focus region, based on the identified game context. For example, an asset that is the focus of the gameplay based on the game context may be defined as highly important.

[0050] Therefore, given one or more pieces of information as input (e.g., a video frame 305, a game state 303, information from one or more components), the AI ​​model 170 performing computer vision is configured to provide an output 310 that includes an asset classification 312 of the assets in the scene of the video frame 305, and a visibility threshold evaluation 315 of the assets found in the scene of the video frame 305. Importantly, the AI ​​technique is used to identify important assets within the focus region against the game context and to determine whether those important assets meet the visibility threshold.

[0051] When one or more critical assets in a scene shown within a corresponding video frame do not meet the visibility threshold, the video frame modification engine 360 ​​is configured to modify and / or transform the content within the video frame so that at least one or more critical assets now meet the visibility threshold. The visibility threshold may define how well an asset is viewable by a user and / or, as described above, may partially correspond to identifying issues with the layout of the content as it appears on a target cross-platform device. For example, transformations may include resizing an asset, rescaling an asset, changing the format of an asset, or performing an FOV change or zoom. In this way, a modified video frame is generated that contains the modified and / or transformed content found within the original video frame. Repeating the process for at least all critical assets or assets found within the focal area for one or more video frames generates multiple modified video frames 307 that contain the modified and / or transformed content found within the original video frame 305. The multiple modified video frames 307 are configured for a target cross-platform device to present the user's gameplay.

[0052] As described above, the platform conversion engine 120 can modify and / or convert the content within video frames remotely at the backend cloud game server when streaming the user's gameplay to the target device 380, or locally at the client device along with the delivery of multiple modified video frames 307 to the target cross-platform device 380.

[0053] For example, Scenario 1 describes the streaming of content from a backend server to a target device. Specifically, multiple modified video frames 307 containing modified and / or transformed content are compressed by a compression engine 370 using any appropriate compression technique. The compressed data is delivered to a streamer 375, which coordinates the delivery of the compressed data to the target cross-platform device 380 over the network to present the multiple modified video frames 307. That is, the platform transformation engine 120 works closely with the game title processing engine in the backend cloud game server.

[0054] In the case of local processing, Scenario 2 shows that multiple modified video frames 307 are delivered directly to the target cross-platform device 380. In particular, compression is not performed because the devices (e.g., local game console or PC and the game title processing engine on the local target device 380) are locally close. In some embodiments, the target cross-platform device 380 (e.g., a mobile phone) is also configured to use the game title processing engine 111 to process the game logic of the video game and present the modified and / or transformed content. For example, a user may be using the cross-platform device to run the video game and participate in a multiplayer game session, coordinating with a game server that provides data related to the multiplayer game session and presents the modified and / or transformed content. That is, in embodiments, the platform transformation engine 120 may be located on the client device to deliver the modified and / or transformed content to the target cross-platform device. Alternatively, the platform transformation engine 120 may be located on the target cross-platform device configured for both running the video game and presenting the modified and / or transformed content (e.g., in a single-player or multiplayer game session).

[0055] Figure 4 shows a platform transformation engine 120 implementing an AI model 170 configured to classify and identify critical assets within video content and modifications to the display of those assets to meet visibility requirements in a cross-platform environment, according to one embodiment of the present disclosure. The cross-platform modifications of content performed by the platform transformation engine 120 may be performed on a cloud game server when streaming the modified content to cross-platform devices, or may be performed locally when displaying and / or delivering the modified content to cross-platform devices.

[0056] The capture engine 440 of the platform conversion engine 120 is configured to receive various data 405 related to classifying scene assets provided in one or more video frames generated for a user's gameplay of a video game. As mentioned above, the data 405 may partially include generated video frames, mapped video frames mapped to a target device, game state data, game assets, game context, target device identifier, focus area identifier, scene assets, identified assets within the focus area, asset ratings, user-saved data, metadata, and other information. The capture engine 440 is configured to provide input to the AI ​​model 170 for classifying the scene assets shown in one or more video frames. Thus, the capture engine 440 is configured to capture and / or receive as input any data that can be used to identify and / or classify those assets.

[0057] Selected portions of the captured data may be analyzed to identify and / or classify asset scenes shown within one or more video frames. In particular, the received data 405 is analyzed by a feature extractor 445A to extract prominent and / or relevant features that help classify and / or identify user emotions. The feature extractor may be configured to learn and / or define features associated with known video game assets or parts thereof. In some embodiments, feature definition and extraction are performed by a deep / machine learning engine 190, such that feature learning and extraction are performed internally, for example, within the feature extractor 445B. In any case, the feature extractor may be trained using a list of assets presented in different game environments of the video game.

[0058] As illustrated, the deep / machine learning engine 190 is configured in embodiments to classify and / or identify one or more assets of a scene shown in one or more video frames generated for a user's gameplay, and any problems that may arise when presenting these key assets to cross-platform devices. In one embodiment, the AI ​​model 170 is a machine learning model configured to apply machine learning to classify and / or identify assets of a scene shown in a video frame, and any problems when presenting those assets. In another embodiment, the AI ​​model is a deep learning model configured to apply deep learning to classify and / or identify one or more assets of a scene shown in a video frame, and any problems when presenting those assets to cross-platform devices, where machine learning is a subclass of artificial intelligence, and deep learning is a subclass of machine learning. In yet another embodiment, the AI ​​model 170 is configured to perform computer vision to classify assets of a scene in a video frame and identify problems with the visibility of those assets when displaying content on cross-platform devices.

[0059] For purely explanatory purposes, the deep / machine learning engine 190 may be configured as a neural network used to implement the AI ​​model 170 according to one embodiment of the present disclosure. Generally, a neural network represents a network of interconnected nodes that respond to inputs (e.g., extracted features) and produce outputs (e.g., classifying, identifying, or predicting the intent of an performed gesture). In one embodiment, the AI ​​neural network includes a hierarchy of nodes. For example, there may be an input layer of nodes, an output layer of nodes, and an intermediate or hidden layer of nodes. Input nodes are interconnected to hidden nodes in the hidden layer, and the hidden nodes are interconnected to output nodes. The interconnections between nodes may have numerical weights that can be used to link multiple nodes together between inputs and outputs, for example, when defining the rules of the AI ​​model 170.

[0060] In particular, the AI ​​model 170 is configured to apply rules that define relationships between features and outputs (e.g., biometric information corresponding to specific emotions), and features may be defined within one or more nodes located at one or more hierarchical levels of the AI ​​model 170. The rules link features (such as those defined by nodes) across layers of the hierarchy such that a given set of data inputs leads to a specific output (e.g., asset classification) 310 of the AI ​​model 170. For example, the rules may link one or more features or nodes between input and output (e.g., using relational parameters including weights) throughout the AI ​​model 170 (e.g., at the hierarchical level) such that one or more features make up a rule that is learned through training the AI ​​model 170. That is, each feature may be linked to one or more features in other layers, and one or more relational parameters (e.g., weights) define the interconnections between features in other layers of the AI ​​model 170. Thus, each rule or set of rules corresponds to a classified output. In this way, the resulting output 310 obtained according to the rules of AI model 170 can classify and / or label and / or identify the scene assets shown in the video frames generated for the user's gameplay, and any problems that may arise when presenting those key assets to cross-platform devices, based on the input data.

[0061] Furthermore, the output 310 from the AI ​​model 170 includes asset classification and / or visibility threshold evaluation for assets determined to be important to a given identified game context (e.g., assets that capture the user's attention). The output 310 is delivered to the video frame modification and / or transformation engine 260, which is configured to perform content modification and / or transformation on at least important assets that do not meet the visibility threshold of a scene shown in one or more video frames of a user's gameplay of a video game, which are classified and presented on a cross-platform device. In this way, the modified content presented on a cross-platform device meets the visibility threshold so that those assets are viewable by the user in a manner suitable for gameplay. The transformation and / or modification of content by the video frame modification engine 260 may be performed at least in part by the overlay engine 261, the FOV adjuster 263, and / or the transformation logic 265.

[0062] In particular, the transformation logic 265 is configured to modify and / or transform content containing one or more critical assets. Content transformation may include resizing or rescaling assets to meet visibility thresholds. Furthermore, content transformation may include formatting changes. For example, the original content may be text, and the text after mapping to a cross-platform device does not meet visibility thresholds (e.g., partially missing, too small). The text may be converted to emojis that replace the text, and the emojis meet visibility thresholds. In another example, the original content may be text provided within a window, and the text is enlarged. The text after mapping to a cross-platform device does not meet visibility thresholds. The text and window may be converted from a static text presentation to a rolling text bar that can be presented within a resized window (i.e., a bar). In this way, after transformation, the rolling text bar meets visibility thresholds.

[0063] In one embodiment, the modified and / or transformed content may be presented as an overlay. In particular, the overlay engine 261 is configured to generate an overlay containing the modified and / or transformed content, which is incorporated into one or more video frames generated by the video game to replace the original content. For example, the overlay containing the modified content may be overlaid on a portion of the initially rendered video frame for display on the intended platform device or for streaming to the intended platform device (e.g., full display). In another embodiment, the modified and / or transformed content is blended into the video frame to generate the modified video frame.

[0064] In one embodiment, a video frame containing modified and / or transformed content may be further transformed using an FOV adjuster 263. For example, when the modified and / or transformed content still does not meet the visibility threshold, or to further enhance that it meets the visibility threshold, the FOV adjuster 263 is configured to perform an FOV change or FOV zoom. For example, a video frame containing modified and / or transformed content may be further modified by zooming to a focal region identified in consideration of the game context in the scene shown in one or more video frames generated for gameplay of a user playing a video game.

[0065] In this way, after the video frame correction engine has run, the corrected video frame containing the corrected and / or transformed content is formatted for presentation on cross-platform devices so that the assets within the corrected video frame meet the visibility threshold. By repeating the process for multiple video frames, it is ensured that the important assets and / or assets found within the focal area of ​​the video frame depicting the scene meet the visibility threshold when presenting the video game user's gameplay on cross-platform devices.

[0066] Figures 5A and 5B illustrate the modification and / or transformation of content in video frames generated for user gameplay of a video game presented on a cross-platform device, according to embodiments of the present disclosure. Generally, the process for modifying and / or transforming content may be shown in Figures 5A and 5B and may include mapping and / or scaling the video frames to fit the display of the target cross-platform device; analyzing the scaled video frames to determine important assets and whether those assets are visible (i.e., whether the assets meet the visibility threshold); transforming visually impaired important assets (the focus of the game context or important within the game context) (i.e., not meeting the visibility threshold); and / or optionally performing FOV changes (e.g., dumb zoom relative to the focal area).

[0067] In particular, Figure 5A shows the mapping and / or scaling of a video frame 501 for display on a target cross-platform device according to one embodiment of the present disclosure. As illustrated, the video frame 501 is generated during user gameplay, and the video frame is initially rendered for presentation on an intended platform device, such as a display 510, which is given full dimensions to display the video frame for the game or to stream to the intended platform device. For example, the initially rendered video frame 501 contains content that depicts a scene, and the content may be represented by one or more assets, including asset A(511), asset B(512), asset C(513), and asset D(514). The assets are shown in a first format for presentation on the display 510 of the intended platform device, and include asset A(511a), asset B(512a), asset C(513a), and asset D(514a). The original video frame 501 is rendered without compromise to be presented on the intended platform device, so all assets are shown, and presumably each asset meets its corresponding visibility threshold when presented on the intended platform device's display.

[0068] Since the user's gameplay is presented on the display 520 of a cross-platform device (e.g., a mobile phone), the content within the initially rendered video frame 501 is mapped to the cross-platform device, which may include some scaling of assets, and the mapping may be pre-designed for the video game or for the execution processor when mapping to the target device. After mapping, the assets are presented in a second format for presentation on the display 520 of the target cross-platform device, including assets A(511b), B(512b), C(513b), and D(514b). For example, each asset may be resized and / or rescaled. The mapped video frame 502 presented on the display 520 may contain presentation issues. For example, assets B(511b) and C(511c) may be displayed with missing portions (i.e., half of each asset is outside the display 520 and not shown).

[0069] Due to the issues identified in the mapped video frame 502, additional scaling may be performed. For example, dumb scaling (the assets are scaled to a smaller size) may be performed on the mapped video frame 502 so that all of the assets are available for viewing on display 520. Furthermore, one or more assets may be identified as important to a given game context, at least for the initially rendered video frame 501. For example, asset B may be determined to be important and is highlighted (i.e., has a thick outline) in the dumb-scaled video frame 503. After rescaling, the assets are presented in a third format for presentation on the target cross-platform device's display 520, including asset A (511c), asset B (512c), asset C (513c), and asset D (514c). For example, each asset is further rescaled and / or resized so that each asset is displayed on display 520.

[0070] Figure 5B shows a transformation of the initially rendered video frame 501 introduced in Figure 5A for display on a target cross-platform device, according to one embodiment of the present disclosure. In particular, further analysis is performed on the scaled video frame 503 to determine whether there are any problems in presenting the scaled video frame 503 to the target device. For example, it may be determined whether each of assets A-D, or more specifically, each of the important assets (e.g., asset B), meets the visibility threshold. Asset B presented in the scaled video frame 503 may not meet the visibility threshold (e.g., too small to adequately view the game), and therefore, as described above, content transformation and / or modification is performed. For example, as shown, asset B (512d) may be rescaled and / or resized to be larger in the fourth format, and the rescaled video frame 504A is presented on the display 520 of the target cross-platform device. Furthermore, the scaled video frame 503 may also be modified and / or transformed so that only the rescaled and / or resized asset B(512d) is displayed on the display 520. Alternatively, the transformation and / or modification of the content may be performed by changing the format of asset B (e.g., converting text to emojis) so that it is represented by the reformatted asset B'(512e). In other words, a replacement asset may be used. The reformatted asset B'(513e) may also be automatically resized so that the resized and / or reformatted video frame 504B is displayed on the display 520 of the target cross-platform device.

[0071] Because the content of the first rendered video frame 501 has been converted to be presented in a scrolling manner, subsequent video frames show the remaining assets. In one embodiment, the remaining assets are shown in order of importance. For example, among the remaining assets, asset A may be determined to be more important than either asset C or D. Therefore, when scrolling to the next screen, asset A (511d) may be rescaled and / or resized to be larger in the sixth format, and the rescaled portion of video frame 505 is presented on the display 520 of the target cross-platform device, thereby showing only the rescaled and / or resized asset A (511d) on the display 520. Subsequently, the remaining assets C and D are equivalent but less important than asset B or A, and thereby assets C and D may be shown together on the screen. Therefore, when scrolling to the next screen, each of asset C(513d) and asset D(514d) may be rescaled and / or resized to the seventh format, and the reformatted portion of video frame 506 is presented on the display 520 of the target cross-platform device. For illustrative purposes, asset C may be shown above asset D on display 520, but may be shown in any suitable layout.

[0072] Although not illustrated, in another embodiment, FOV changes (e.g., dumb zoom to the focal region, FOV scaling / targeting / zoom, etc.) may be performed to enhance content presentation by further transforming assets and / or video frames. For example, a dumb zoom to a focal region surrounding a key asset may be performed.

[0073] Figure 6 shows components of an exemplary device 600 that may be used to perform various embodiments of the present disclosure. This block diagram shows a device 600 that may be incorporated into, or can be made into, a personal computer, video game console, personal digital assistant, server or other digital device, and includes a central processing unit (CPU) 602 for running software applications and, optionally, an operating system. The CPU 602 may consist of one or more homogeneous or heterogeneous processing cores. Further embodiments may be performed using one or more CPUs having a microprocessor architecture particularly adapted for highly parallel and computationally intensive applications.

[0074] In particular, CPU 602 may be configured to implement a platform prediction engine 120 that implements artificial intelligence configured to ensure that the display of those critical assets meets visibility requirements, including identifying critical assets within video content and modifying the video game content when delivering video frames for display on cross-platform devices. Identifying visibility issues when displaying content on cross-platform devices using artificial intelligence may be performed in real time during video game development or during video game gameplay.

[0075] Memory 604 stores applications and data for use by the CPU 602. Storage 606 provides non-volatile storage and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROMs, DVD-ROMs, Blu-ray®, HD-DVDs, or other optical storage devices, as well as signal transmission and storage media. User input devices 608 communicate user input from one or more users to device 600, and examples of user input devices 608 may include keyboards, mice, joysticks, touchpads, touchscreens, still recorders / cameras or video recorders / cameras, tracking devices for recognizing gestures, and / or microphones. Network interfaces 614 enable device 600 to communicate with other computer systems via electronic communication networks, and may include wired or wireless communication over local area networks and wide area networks such as the Internet. The audio processor 612 adapts instructions and / or data provided by the CPU 602, memory 604, and / or storage 606 to generate analog or digital audio output. The components of device 600 are connected via one or more data buses 622.

[0076] The graphics subsystem 620 is further connected to the data bus 622 and the components of device 600. The graphics subsystem 620 includes a graphics processing unit (GPU) 616 and graphics memory 618. The graphics memory 618 includes display memory (e.g., a frame buffer) used to store pixel data for each pixel of the output image. The pixel data may be provided directly to the graphics memory 618 from the CPU 602. Alternatively, the CPU 602 provides the GPU 616 with data and / or instructions defining a desired output image, from which the GPU 616 generates pixel data for one or more output images. The data and / or instructions defining a desired output image may be stored in memory 604 and / or graphics memory 618. In one embodiment, the GPU 616 includes a 3D rendering function for generating pixel data for an output image from instructions and data defining the scene geometry, lighting, shading, texturing, motion, and / or camera parameters. The GPU616 may further include one or more programmable execution units capable of executing shader programs. In one embodiment, the GPU616 may be implemented within an AI engine (e.g., a machine learning engine 190) to provide additional processing power to AI, machine learning functions, or deep learning functions, etc.

[0077] The graphics subsystem 720 periodically outputs pixel data of the image displayed on the display device 710 from the graphics memory 718. The display device 710 can be any device capable of displaying visual information in response to a signal from device 700.

[0078] In other embodiments, the graphics subsystem 720 includes multiple GPU devices that are coupled together to perform graphics processing for a single application running on the CPU. For example, the multiple GPUs can perform alternative forms of frame rendering, including different GPUs rendering different frames at different times, different GPUs performing different shader operations, having a master GPU perform the main rendering and compositing the output from slave GPUs performing selected shader functions (e.g., smoke, river, etc.), or different GPUs rendering different objects or parts of a scene. In the embodiments described above, these operations can be performed in the same frame duration (concurrently) or in different frame durations (sequentially in parallel).

[0079] Accordingly, in various embodiments, this disclosure describes systems and methods configured for cross-platform modification of content using artificial intelligence so that critical assets meet visibility requirements thresholds.

[0080] It should be noted that access services delivered across a wide geographical area, such as providing access to the game in this embodiment, often utilize cloud computing. Cloud computing is a computing mode in which dynamically scalable, often virtualized, resources are provided as a service over the internet. For example, cloud computing services often provide common applications (e.g., video games) accessed from a web browser online, while the software and data are stored on servers in the cloud.

[0081] In some embodiments, a game server may be used to perform operations for video game players playing video games over the internet. In a multiplayer game session, a dedicated server application collects data from players and distributes it to other players. The video game may also be run by a distributed game engine that includes multiple processing entities (PEs) acting as nodes, so that each PE performs a functional segment of a given game engine on which the video game is run. For example, the game engine implements game logic and performs game calculations, physical analysis, geometry transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. Additional services may include, for example, messaging, social utilities, voice communication, gameplay playback functionality, help functionality, etc. The PEs may be virtualized by a hypervisor on a specific server, or the PEs may reside on different server units in a data center. Each processing entity for performing operations may be a server unit, a virtual machine or container, a GPU, or a CPU, depending on the needs of each game engine segment. By distributing the game engine, the game engine is provided with flexible computational characteristics that are not limited by the capabilities of physical server units. Instead, the game engine is provisioned with more or fewer computing nodes as needed to meet the demands of the video game.

[0082] Users access remote services using client devices (e.g., PCs, mobile phones, etc.) that include at least a CPU, display, and I / O and are capable of communicating with the game server. It should be understood that a given video game may be developed for a specific platform and associated controller device. However, when such games become available via a game cloud system, users may access the video game using different controller devices, such as when accessing a game designed for a game console from a personal computer using a keyboard and mouse. In such scenarios, input parameter settings define the mapping from inputs that can be generated by the user's available controller devices to inputs that are acceptable for running the video game.

[0083] In another example, a user may access a cloud gaming system via a tablet computing device, a touchscreen smartphone, or other touchscreen-driven device. In this case, the client device and controller device are integrated, and input is provided by detected touchscreen inputs / gestures. For such devices, input parameter settings may define specific touchscreen inputs (e.g., buttons, directional pads, gestures or swipes, touch motions, etc.) corresponding to game inputs for video games.

[0084] In some embodiments, the client device functions as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection and transmits inputs from the controller device to the client device. The client device then processes these inputs and may send the input data to a cloud game server over the network. For example, these inputs may include video or audio captured from the game environment, which may be processed by the client device before being sent to the cloud game server. In addition, inputs from the controller's motion detection hardware may be processed by the client device in conjunction with captured video to detect the controller's position and motion before being sent to the cloud game server.

[0085] In other embodiments, the controller itself can be a network device, capable of directly communicating such inputs to a cloud game server over the network without first needing to communicate them through a client device. As a result, input latency can be reduced. For example, inputs that do not depend on detection by any additional hardware or processing separate from the controller itself can be sent directly from the controller to the cloud game server. Such inputs may include button inputs, joystick inputs, and built-in motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes).

[0086] Access to a cloud gaming network by a client device can be achieved through a network implementing one or more communication technologies. In some embodiments, the network may include fifth-generation (5G) radio network technology, including a cellular network serving small geographical cells. Analog signals representing sound and images are digitized at the client device and transmitted as a bitstream. 5G radio devices within a cell communicate radio waves with local antenna arrays and low-power automatic transceivers. The local antennas are connected to the telephone network and the Internet by high-bandwidth optical fiber or wireless backhaul connections. Mobile devices traversing between cells are automatically transferred to the new cell. A 5G network is just one communication network, and embodiments of this disclosure may utilize earlier generations of communication networks, as well as later generations of wired or wireless technologies that follow 5G.

[0087] In one embodiment, various technical examples can be implemented using a virtual environment via a head-mounted display (HMD), which may also be called a virtual reality (VR) headset. As used herein, this term generally refers to user interaction with a virtual space / virtual environment, including viewing the virtual space through the HMD in real time in response to the HMD's movements (controlled by the user) to give the user the feeling of being in a virtual space or metaverse. The HMD can be worn in the same way as glasses, goggles, or a helmet and is configured to display video games or other metaverse content to the user. The HMD can provide a highly immersive experience in a virtual environment with three-dimensional depth and perspective.

[0088] In one embodiment, the HMD may include an eye-tracking camera configured to capture images of the user's eyes while the user interacts with the VR scene. The eye-tracking information captured by the eye-tracking camera(s) may include the user's gaze direction and information related to specific virtual objects and content items in the VR scene that the user is looking at or is interested in interacting with.

[0089] In some embodiments, the HMD may include outward-facing cameras (or more) configured to capture images of the user's real-world space, such as the user's body movements, and images of any real-world objects that may be located in that real-world space. In some embodiments, the images captured by the outward-facing cameras may be analyzed to determine the location / orientation of the real-world objects relative to the HMD. Using the known location / orientation of the HMD, inertial sensor data from real-world objects and the user's gestures and movements can be continuously monitored and tracked while the user interacts with the VR scene. For example, while interacting with a scene in a game, the user may perform various gestures, such as commands, communication, pointing to a specific content item in the scene, and walking towards it. In one embodiment, the gestures can be tracked and processed by the system to generate predictions of interactions with specific content items in the game scene. In some embodiments, machine learning may be used to facilitate or assist in these predictions.

[0090] While using the HMD, various types of one-handed and two-handed controllers can be used. In some embodiments, the controller itself can be tracked by tracking lights contained within the controller, or by tracking shape, sensor, and inertial data associated with the controller. By using these various types of controllers, or even by using simple hand gestures created and captured by one or more cameras, it becomes possible to interface with, control, manipulate, interact with, and participate in virtual reality environments or metaverses rendered on the HMD. In some cases, the HMD can wirelessly connect to cloud computing and game systems via networks such as the internet or cellular. In one embodiment, the cloud computing and game system maintains and runs the video game being played by the user. In some embodiments, the cloud computing and game system is configured to receive input from the HMD and / or interface objects via the network. The cloud computing and game system is configured to process the input and affect the game state of the running video game. Output from the running video game, such as video data, audio data, and haptic feedback data, is sent to the HMD and interface objects.

[0091] In addition, while embodiments of this disclosure may be described with reference to HMDs, it will be understood that in other embodiments, non-HMDs may be substituted, such as the screen of a portable device (e.g., a tablet, smartphone, laptop, etc.), or any other type of display that can be configured to render video and / or provide the display of an interactive scene or virtual environment. It should be understood that the various embodiments defined herein may be combined or assembled into specific embodiments using the various features disclosed herein. Thus, the embodiments provided are not limited to the various embodiments that are possible by combining various elements and defining even more embodiments, but are merely some possible embodiments.

[0092] Embodiments of the present disclosure can be implemented using a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, and mainframe computers. Embodiments of the present disclosure can also be implemented in a distributed computing environment in which tasks are performed by remote processing devices linked via a wired or wireless network.

[0093] Although the operation of the method has been described in a specific order, it should be understood that other housekeeping operations may be performed between operations, or the operations may be coordinated to occur at slightly different times, or the operations may be distributed to a system that allows the operation to occur at various intervals related to the operation, as long as the processing of telemetry and game state data for generating the modified game state is performed in the desired manner.

[0094] With the embodiments described above in mind, it should be understood that embodiments of the present disclosure can employ a variety of computer operations that require data stored in a computer system. These operations require the physical manipulation of physical quantities. Any of the operations described herein in embodiments of the present disclosure are useful mechanical operations. Embodiments of the disclosure also relate to devices or apparatus for performing these operations. The apparatus may be built specifically for a required purpose, or the apparatus may be a general-purpose computer selectively started or configured by a computer program stored in the computer. Specifically, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to build an apparatus more specialized to perform the required operations.

[0095] One or more embodiments may also be fabricated as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device capable of storing data that can later be read by a computer system. Examples of computer-readable mediums include hard drives, network-attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. A computer-readable medium may include computer-readable tangible media distributed across network-connected computer systems so that the computer-readable code is stored and executed in a distributed manner.

[0096] In one embodiment, the video game runs locally on a game console, a personal computer, or on a server, or by one or more servers in a data center. When the video game is running, some instances of the video game may be simulations of the video game. For example, the video game may be run by an environment or server that generates a simulation of the video game. In some embodiments, the simulation is an instance of the video game. In other embodiments, the simulation may be produced by an emulator that emulates a processing system.

[0097] While the embodiments described above have been explained in some detail for clarity, it will be apparent that certain changes and modifications can be made within the scope of the appended claims. Therefore, these embodiments should be considered illustrative and not limiting, and the embodiments are not limited to the details described herein and can be modified within the scope of the appended claims and equivalents.

Claims

1. It is a method, Executing a video game to generate video frames for presentation on a first device of a first platform, wherein the video frames are generated in response to gameplay of the video game by a user, Determining a target device, wherein the target device is of the second platform, and the determination of the target device, Mapping the video frame from the first device to the target device, wherein the mapped video frame is presented on the target device, and the mapping is performed. The focus region within the gameplay scene presented in the video frame is determined based on the game context of the scene within the gameplay. Classifying assets within the focal region using a computer vision model that implements artificial intelligence, Using the aforementioned computer vision model, determine that the asset is important in the gameplay of the video game, Determining that the asset in the video frame mapped to the target device does not meet the visibility threshold, Modify the mapped video frame so that the asset satisfies the visibility threshold, The method, including the method described above.

2. The method according to claim 1, further comprising transmitting the mapped and modified video frames to the target device.

3. Modifying the mapped video frame is Transforming the aforementioned asset through rescaling, Overlaying the rescaled asset within the mapped video frame, The method according to claim 1, including the method described in claim 1.

4. Modifying the mapped video frame is Changing the field of view (FOV) of the aforementioned scene to the focal region, The video frames to be modified are limited to the modified FOV, The method according to claim 1, including the method described in claim 1.

5. Determining the aforementioned focal region means The method according to claim 1, comprising determining the focus of user interaction in the scene of the gameplay.

6. The method according to claim 1, further comprising training the computer vision model using the list of assets of the video game.

7. Mapping the aforementioned video frame is The method according to claim 1, comprising dam scaling the video frame to a smaller size for presentation on the target device.

8. A non-temporary computer-readable medium for storing a computer program for execution by a processor to perform a method, wherein the non-temporary computer-readable medium is A program instruction for executing a video game to generate video frames for presentation on a first device of a first platform, wherein the video frames are generated in response to gameplay of the video game by a user, and the program instruction for execution is: A program instruction for determining a target device, wherein the target device is of a second platform, and the program instruction for determining the target device, A program instruction for mapping the video frame from the first device to the target device, wherein the mapped video frame is presented on the target device, and the program instruction for mapping is provided. A program instruction for determining the focal region within the gameplay scene presented in the video frame based on the game context of the scene within the gameplay, Program instructions for classifying assets within the focal region using a computer vision model that implements artificial intelligence, A program instruction for determining, using the computer vision model, that the asset is important in the gameplay of the video game, A program instruction for determining that the asset in the video frame mapped to the target device does not meet the visibility threshold, Program instructions for modifying the mapped video frame so that the asset satisfies the visibility threshold, The non-temporary computer-readable medium, including the above.

9. The non-temporary computer-readable medium according to claim 8, further comprising program instructions for transmitting the mapped and modified video frames to the target device.

10. The program instructions for modifying the mapped video frames are: Program instructions for transforming the asset by rescaling, Program instructions for overlaying the rescaled asset within the mapped video frame, A non-temporary computer-readable medium according to claim 8, including the following:

11. The program instructions for modifying the mapped video frames are: A program instruction for changing the field of view (FOV) of the aforementioned scene to the focal region, A program instruction for restricting the video frame to be modified to the modified FOV, A non-temporary computer-readable medium according to claim 8, including the following:

12. The program instruction for determining the focal region is: The non-temporary computer-readable medium according to claim 8, comprising program instructions for determining the focus of user interaction in the scene of the gameplay.

13. The non-temporary computer-readable medium according to claim 8, further comprising program instructions for training the computer vision model using the list of assets of the video game.

14. The program instruction for mapping the video frame is: A non-temporary computer-readable medium according to claim 8, comprising program instructions for dumb scaling the video frames to a smaller size for presentation on the target device.

15. A computer system, wherein the computer system is Processor and The memory is coupled to the processor and stores instructions, the instructions, when executed by the computer system, cause the computer system to execute a method for implementing a graphics pipeline, and the method is Executing a video game to generate video frames for presentation on a first device of a first platform, wherein the video frames are generated in response to gameplay of the video game by a user, Determining a target device, wherein the target device is of the second platform, and the determination of the target device, Mapping the video frame from the first device to the target device, wherein the mapped video frame is presented on the target device, and the mapping is performed. The focus region within the gameplay scene presented in the video frame is determined based on the game context of the scene within the gameplay. Using a computer vision model that implements artificial intelligence, it is determined that the assets within the focal region are important to the gameplay of the video game, Using the aforementioned computer vision model, classify the assets, Determining that the asset in the video frame mapped to the target device does not meet the visibility threshold, Modify the mapped video frame so that the asset satisfies the visibility threshold, The computer system including the computer system.

16. The aforementioned method, The computer system according to claim 15, further comprising transmitting the mapped and modified video frames to the target device.

17. In the above method, modifying the mapped video frame is Changing the field of view (FOV) of the aforementioned scene to the focal region, The video frames to be modified are limited to the modified FOV, The computer system according to claim 15, including the above.

18. In the above method, determining the focal region is The computer system according to claim 15, comprising determining the focus of user interaction in the scene of the gameplay.

19. The aforementioned method, The computer system according to claim 15, further comprising training the computer vision model using the list of assets of the video game.

20. In the above method, the mapping of the video frame is The computer system according to claim 15, comprising dumb scaling the video frame to a smaller size for presentation on the target device.

Images