Implementing accessibility features during gameplay by leveraging machine learning models

Abstract

Aspects of the present disclosure provide systems and methods that utilize machine learning techniques to provide enhanced accessibility features to games. An accessibility service is provided that can instantiate one or more machine learning models capable of processing the current state of game play and generating commands to assist the user during game play. Accessibility commands are provided to the game and can be used to supplement or correct input provided by the user to compensate for specific user needs. In a further aspect, an accessibility user interface is provided that enables the user to dynamically enable or disable the accessibility features during game play. This user interface is operable to receive an accessibility selection during game play and provide the selection data to the accessibility service.

Description

Technical Field

[0001] Cross - Reference to Related Applications

[0001] This application claims the priority of U.S. Provisional Patent Application No. 63 / 345,216, filed on May 24, 2022, entitled "Creation and Personalization of Virtual Gamers", and claims the priority of U.S. Patent Application No. 18 / 199,693, filed on May 19, 2023, entitled "Leveraging Machine Learning Models to Implement Accessibility Features during Gameplay", the entire disclosure of which is incorporated herein by reference.

Background Art

[0002] Background

[0002] The gaming industry occupies a large part of the technology industry sector. In a future envisioned by technology as becoming increasingly interconnected, games are a central element of the interconnected future. Artificial intelligence has always been a major element throughout the gaming industry, but as AI development continues to progress, it becomes more difficult for the gaming industry to incorporate that progress into games. Furthermore, as the gaming market continues to grow, accessibility features become more important to accommodate users with various needs. However, the identification and / or implementation of accessibility features often require specialized knowledge that standard game developers may not have.

[0003]

[0003] The aspects disclosed herein are provided in view of these and other general considerations. Additionally, although relatively specific problems may be discussed, it should be understood that the examples are not to be limited to solving the specific problems made apparent in the background or elsewhere of this disclosure.

Summary of the Invention

[0004] Summary

[0004] Aspects of the present disclosure provide systems and methods that utilize machine learning techniques to provide enhanced accessibility features to games. An accessibility service can be provided that processes the current state of game play and instantiates one or more machine learning models capable of generating commands to assist the user during game play. Accessibility commands are provided to the game and can be used to supplement or correct the input provided by the user to compensate for specific user needs. By doing so, an accessible and customizable functionality is provided to a robust system that allows users with various game play needs (e.g., visually impaired users, single-handed users, etc.) to enjoy the benefits of the game even if the game does not natively support specific accessibility features.

[0005]

[0005] In a further aspect, an accessibility user interface is provided that allows the user to dynamically enable or disable accessibility features during game play. This user interface is operable to receive an accessibility selection during game play and provide the selection data to the accessibility service. The selected accessibility feature can be used by the service to instantiate one or more machine learning models operable to generate accessibility commands for incorporating the selected accessibility feature into the game during game play.

[0006]

[0006] This summary is provided to introduce a simplified form of a series of concepts that will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be described in part in the following description, become apparent in part from the description, or may be learned by practice of the present disclosure.

[0007] Brief Description of the Drawings

[0007] Non-limiting and non-exhaustive examples are described with respect to the following drawings.

Brief Description of the Drawings

[0008]

Fig. 1A

[0008] An overview of an example of a system for generating and using a user anthropomorphic agent in a game system is shown.

Fig. 1B

[0009] An overview of an example of a system for generating and utilizing responses of an anthropomorphic agent in a game system is shown.

Fig. 2

[0010] An example of a method for generating an anthropomorphic agent is shown.

Fig. 3

[0011] An exemplary system 300 for providing an accessibility service 302 using one or more machine learning models is shown.

Fig. 4

[0012] An exemplary method for instantiating an accessibility machine learning model to assist a user during a game play session is shown.

Fig. 5A

[0013] An exemplary method for generating an accessibility command using one or more machine learning models is shown.

Fig. 5B

[0014] An exemplary method 520 for implementing an accessibility function during game play is shown.

Fig. 6

[0015] An exemplary method 600 for modifying an accessibility function based on changes in a user's game play is shown.

Fig. 7

[0016] A block diagram showing an example of the physical components of a computing device in which aspects of the present disclosure may be practiced.

Fig. 8

[0017] A simplified block diagram of another mobile computing device in which aspects of the present disclosure may be practiced.

Best Mode for Carrying Out the Invention

[0009] Detailed Description

[0018] This specification forms part of it and various aspects of the present disclosure will be more fully described below with respect to the accompanying drawings that illustrate specific example embodiments. However, the various aspects of the present disclosure can be implemented in many different ways and should not be construed as limited to the embodiments described herein. Rather, those embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the aspects to those skilled in the art. The embodiments to be practiced can be in the form of a method, system, or apparatus. Thus, the aspects can take the form of a hardware implementation, a completely software implementation, or an implementation combining software and hardware aspects. Therefore, the following detailed description should not be taken in a limiting sense.

[0010]

[0019] Aspects of the present disclosure provide systems and methods that utilize machine learning techniques to provide enhanced accessibility features to games. As described herein, an accessibility service is provided that can utilize one or more machine learning models to generate accessibility commands that provide accessibility features to users executable by a game. The degree of accessibility features can be customized for each user. Further, since machine learning models are utilized, game play control commands can be generated to assist a user during game play, for example, to assist a one-handed user, and game state commands can be generated to modify game graphics, for example, to assist a user with color blindness or visual impairment.

[0011]

[0020] As described above, the accessibility features can be generated through the use of a machine learning model that is executed during gameplay. Thereby, the accessibility service can provide accessibility features to a variety of different games through instructions generated by the machine learning model, without requiring individual games to natively implement the accessibility features. By doing so, the accessibility system provided herein can be utilized to greatly expand the use of accessibility features across the entire game market, making games more inclusive for all users regardless of their specific needs or abilities.

[0012]

[0021] In an example, generative multimodal machine learning model processing can be employed to generate multimodal output. For example, a conversational agent according to the aspects described herein can receive user input and thereby use a generative multimodal machine learning model to process the user input to generate multimodal output. The multimodal output can include, among other examples, natural language output and / or program output. The multimodal output can be processed and used to affect the state of the relevant application. For example, at least a portion of the multimodal output can be executed, or it can be used to call an application programming interface (API) of the application. The generative multimodal machine learning model (also generally referred to herein as the multimodal machine learning model) used according to the aspects described herein can be, in some examples, a generative transformer model. In some examples, explicit and / or implicit feedback can be processed to improve the performance of the multimodal machine learning model.

[0013]

[0022] FIG. 1 shows an overview of an example of a system 100 for generating and using user anthropomorphic agents in a game system. As shown in FIG. 1, a user device 102 interacts with a cloud service 104 that hosts an instantiation of a game service 106 (or other types of applications) and an agent 108 that can interact with the game. The game device can be a console game system, a mobile device, a smartphone, a personal computer, or any other type of device capable of locally executing a game or accessing a game hosted on a server. In one example, a game related to the game service 106 can be directly hosted by the cloud service 104. In an alternative example, the user device may host and execute the game locally, in which case the game service 106 can serve as an interface that facilitates communication between one or more instantiated agents 108 and the game. The anthropomorphic agent library 107 can store and execute components for one or more agents related to the user of the user device 102. The components of the anthropomorphic agent library 107 can be used to control the instantiated agent 108.

[0014]

[0023] In an example, one or more agents from the anthropomorphic agent library 107 interact with the game via the instantiated agent 108 based on text communications, voice commands, and / or player actions received from the user device 102. That is, the system 100 supports interaction with the agents as if the agents were other human players, or as a player would typically interact with an NPC within a conventional game. By doing so, one or more agents hosted by the agent library 108 can operate to interact with various games that a user plays without requiring changes to the games. That is, the system 100 can operate to cooperate with games without requiring the games to be specially developed to support the agents (e.g., the agents do not require API access, and the games do not need to be developed with specific code to interact with the agents). By doing so, the system 100 provides an extensible solution that enables a user to play with customized agents across a wide variety of different games. That is, the game state is a game by the game service 106 that communicates between one or more instantiated agents 108 using audio and visual interactions and / or a public API. By doing so, the instantiated agent 108 can interact with the game in the same way as the user (e.g., by interpreting video, audio, and / or tactile feedback from the game) and / or in the same way as an NPC within the game (e.g., by the public API). Similarly, one or more agents can interact with a user playing a game using the game device 102 in the same way as another player or as an NPC that interacts with the user. That is, one or more agents can receive visual, text, or voice input from the user playing the game by the game service 106 and / or game information (e.g., the current game state, the inventory of the NPC, the abilities of the NPC, etc.) by the public API.In this way, system 100 enables a user playing a game on user device 102 to interact with one or more agents so that the user can interact with any other player on the NPC. However, the instantiated agent 108 can be anthropomorphized according to the user based on past interactions with the user that occurred in both the current game and other games. To facilitate this type of user interaction, one or more instantiated agents 108 can employ computer vision, speech recognition, or other known techniques when processing the interaction with the user to interpret the commands received from the user and / or generate response actions based on the user's actions.

[0015]

[0024] For example, consider a user who plays a first-person shooting game with an anthropomorphic agent. The user might say "Cover the right side." A speech recognition model, which could be one of models 118, can be utilized to interpret the voice received from the user in a manner that can be understood by the agent. Accordingly, agent 108 can take a position on the user's right side or perform an action to cover the user's right side in response to the user's voice command. As yet another example, consider an action role-playing game where two objectives must be defended simultaneously. The agent playing with the user can use computer vision to analyze the current view of the game and determine that there are two objectives (e.g., identify the two objectives on a map, interpret a display quest log indicating that the two objectives must be defended, etc.). Similarly, computer vision can be used to identify the user's player character and determine that the player character is headed towards a first objective. Based on feedback from the computer vision model, the agent can move its proxy character towards the second objective and command it to defend it.

[0016]

[0025] From these examples, those skilled in the art will understand that by adopting speech recognition, computer vision techniques, etc., one or more agents from the agent library can interact with the user in the same way as other users do during cooperative gameplay. By doing so, the instantiated agent 108 may not have API access or program access to the game to interact with the user, so one or more agents from the agent library can be generated separately from a specific game. Alternatively, in the case where the instantiated agent "depends" and for instantiated agents such as NPCs in the game, the instantiated agent 108 may have API access or program access to the game. In such a situation, the agent can interact with the game state and the user based on the API access. Since the system 100 provides a solution that can implement agents separately from individual games, one or more agents in the agent library can be anthropomorphized to interact with a specific user. This anthropomorphization can be maintained across various games. That is, the agent can learn details about the user over time, such as the user's strengths and weaknesses, play style, communication pattern, preferred strategies, etc., and can appropriately respond to the user across various games.

[0017]

[0026] The anthropomorphization of the agent can be generated by the feedback collection engine 110 and updated over time. The feedback collection engine 110 receives feedback occurring within the game from the user and / or from the instantiated agent 108. The feedback collected may include the user's play style, the user's communication, the user's interaction with the game, the user's interaction with other players, the user's interaction with other agents, the results of actions performed by the instantiated agent 108 within the game, information related to the interaction between the actions of the players within the game and the actions of the instantiated agent 108, or any type of information generated by the user device 102 when the user plays the game. To comply with considerations regarding the user's privacy, the information can be collected by the feedback collection engine 110 only when permission is obtained from the user. The user can choose at any time to participate or not participate in the collection. The data collected can be implicit data, such as data based on the user's normal interaction with the game, or explicit data such as specific commands provided by the user to the system. An example of a specific command can be instructing the user to address the agent by a specific character name. The data collected by the feedback collection engine 110 can be provided to the prompt generator 112.

[0018]

[0027] The prompt generator 112 can generate prompts that use the data collected by the feedback collection engine 110 to anthropomorphize one or more agents in the agent library 108. That is, the prompt generator 112 interprets the collected feedback data to generate instructions that one or more agents can execute to perform actions by the agents. The prompt generator is operable to generate new prompts or instructions based on the collected feedback, or to modify existing prompts based on newly collected feedback. For example, if a user first plays the game as a frontline attacker, the prompt generator 112 can generate instructions to implement a supportive playstyle, such as a ranged attacker or a support character, for the instantiated agent 108. If the user shifts their playstyle to one of the ranged damage dealers, the prompt generator 112 can identify that change from the feedback data and adjust the instantiated agent to adapt to the player's new style (e.g., switch to a frontline attacker to remove confrontation or adversarial situations from the player's character). The instructions generated by the prompt generator are provided to the cloud service 104 to be stored by the cloud service 104 as part of the agent library 108, thereby storing meta-classifications (e.g., sentiment analysis, intent analysis, etc.) related to a particular user. By doing so, the instructions generated based on the user's playstyle or preferences in the first game can be incorporated by the agents not only in the first game but also in other games the user plays. That is, using the instructions generated by the prompt generator 112, the cloud service 104 can instantiate agents across a variety of different games that have already been anthropomorphized for a particular user based on the user's past interactions with the instantiated agents 108, regardless of whether the user is playing the same game or different games.While the aspects described herein are presented as a separate prompt generator 112 generating commands to control the instantiated agent 108, in alternative aspects, the commands may be generated directly by one or more machine learning models employed by the agent library 107, or may be generated directly by combinations of the various different components disclosed herein.

[0019]

[0028] However, in some scenarios, there can be problems encountered during the training process of the various machine learning models that can be utilized to generate anthropomorphic agents. For example, the training session may fail due to data errors or processing errors. In yet another example, the training process may fail due to user errors. For instance, a user starting a new game may initially play the game incorrectly. The user's incorrect actions or experiences may train the anthropomorphic agent to play in a way that negatively impacts the gameplay. Thus, the system 100 can include a process for rolling back or resetting one or more machine learning models (or any of the other components disclosed herein) to correct errors that may occur during the training of one or more machine learning models, or other errors that generally may occur when anthropomorphic agents are developed. For example, the system 100 can periodically maintain snapshots of the various machine learning models (or other components) that save the state of the components at the time of the snapshot. Thereby, the system 100 can roll back all components, a subset of components, or a particular component in response to detecting a training error in the future. As the anthropomorphic agent evolves over time, multiple different snapshots representing its state can be stored as a store, thereby providing the system 100 (or the user) with options for determining a state suitable for rollback when an error is encountered.

[0020]

[0029] The anthropomorphic agent library 107 may also include a fine-tuning engine 114 and one or more models 116. The fine-tuning engine 114 and the models 116 are operable to interact with user actions by the user device 102, the instantiated agent 108, and the game training model 124 to process feedback data received from various sources. For use with the examples disclosed herein, any number of different models can be employed individually or in combination. For example, a base model, a language model, a computer vision model, a speech model, a video model, and / or an audio model may be employed by the system 100. As used herein, a base model is a model trained based on a wide range of data adaptable to a wide range of tasks (e.g., a model capable of processing various different tasks or styles).

[0021]

[0030] One or more models 116 can process video, audio, and / or text data received from the user or generated by the game during gameplay to interpret the user's commands and / or derive the user's intent based on the user's communication or actions within the game. The output from one or more models 116 is provided to the fine-tuning model 114, and the fine-tuning model 114 can use the output to modify the prompt generated by the prompt generator 112 to further anthropomorphize the commands generated for the user based on the agent's past interactions with the user. The anthropomorphic agent library 107 may also include a game data (heritage) 122 component operable to store information about various different games. The game data (heritage) 122 component stores information about games, game heritages, story elements, available abilities, and / or items, etc., that can be utilized by the agent by other components (e.g., model 118, fine-tuning engine 114, prompt generator 112, etc.) to generate instructions for controlling the instantiated agent 108 according to the theme, story, requirements, etc. of the game.

[0022]

[0031] The various components described herein can be utilized to interpret the current game state based on data received from the game (e.g., visual data, audio data, tactile feedback data, data exposed by the game via an API, etc.). Further, the components disclosed herein are operable to generate instructions for controlling the actions of anthropomorphic agents within the game based on the current game state. For example, the components disclosed herein can be operable to generate instructions for controlling the interaction of an anthropomorphic agent with the game in a manner similar to how a human interacts with the game (e.g., by providing specific controller instructions, keyboard instructions, or any other type of interface having supported game play controls). Alternatively or in addition, the various components may generate code that controls how an anthropomorphic agent interacts within the game. For example, the code may be executed by the anthropomorphic agent, and such execution causes the anthropomorphic agent to perform specific actions within the game.

[0023]

[0032] The anthropomorphic agent library 107 can also include an agent memory component 120. The agent memory component can be used to store anthropomorphic data generated by the various other components described herein, as well as play styles, techniques, and interactions learned by the agent through past interactions with the user. The agent memory 120 can provide additional inputs to the prompt generator 112 that can be used to determine the actions of an instantiated agent during game play.

[0024]

[0033] So far, the described components of system 100 have focused on the creation of anthropomorphic agents, the continuous evolution of anthropomorphic agents through continuous interaction with the user, and the instantiation of anthropomorphic agents in the user's game session. The anthropomorphization of agents for a particular user provides many in-game advantages, but the instantiated agents also require an understanding of how to interact with the game and how to play the game. In one aspect, the components of the anthropomorphic agent library 107 are operable to learn and refine the game play of the agent based on the session with the user. However, when a user plays a new game, the time required for the agent to learn the mechanics of the game play to be a useful companion may not be achievable with just the user's game play. To address this issue, system 100 also includes a game play training service 124 that includes a game library 126 and a game play machine learning model 128. In various aspects, the game library includes any number of games supported by the cloud service 104 and / or the user device 102. The game play training service 124 is operable to execute sessions of various games stored within the game library 126 and to instantiate agents within the executed games. The game play machine learning model 128 is operable to receive, as input, data from the executed games and the actions of the agents taken within the games, and in response, generate control signals that direct the game play of the agents within the games. Through the use of reinforcement learning, the game play machine learning model is operable to deepen its understanding of the mechanics of game play for both a particular game and a particular genre of games. By doing so, the game play training service 124 provides a mechanism by which agents can be trained to play a particular game or a particular type of game without the need for user interaction.The anthropomorphic agent library 107 is operable to receive (or interact with) a trained model that can be stored as part of the agent memory from the gameplay training service 124, and control the agent 108 instantiated using those models along with other anthropomorphic components. This significantly improves the user experience of interacting with the agent in the game, as it eliminates the need for the user to spend time training the agent in a particular mechanism of the game. Additionally, by importing or interacting with the trained gameplay machine learning model 128 provided by the gameplay training service 124, the anthropomorphic agent library 107 can employ the trained instantiated agent 108 to play with the user when the user first launches a new game.

[0025]

[0034] Those skilled in the art will understand that the system 100 provides an anthropomorphic agent or artificial intelligence that is operable to learn the player's uniqueness, learn the player's communication style or tendencies, learn the strategies employed and used by the player in various different games and scenarios, and learn the gameplay mechanisms of specific games and game genres. Further, one or more agents generated by the system 100 can be stored as part of a cloud service that enables the system to maintain a "memory" of the user's past interactions, thereby enabling the system to generate an agent that functions as a consistent user companion across various games without the game having to be specially designed to support such an agent.

[0026]

[0035] Figure 1B shows an overview of an example of a system 150 for generating and utilizing responses of anthropomorphic agents in a game system. As shown in system 150, two players, player 1 152 and player 2 154, are interacting with one or more agents 156. Although two players are shown, one of ordinary skill in the art will understand that any number of players can participate in a game session using system 150. A helper service 158 is provided to assist in anthropomorphizing the interaction of one or more agents 158 with an individual player or simultaneously with multiple players. As discussed in Figure 1A, one or more models 166 can be used to generate or modify a prompt 168. The prompt 168 is provided to the helper service 158, which applies several engines (e.g., agent persona engine 160, user goal or intent engine 162, and game lore or constraint engine 164) to modify the prompt and provide a more anthropomorphic interaction with an individual player or group of players.

[0027]

[0036] For example, the agent persona engine 160 can modify or adjust a prompt (or an action determined by the prompt) according to anthropomorphic information related to the agent. For example, the user can adopt an agent with a preferred personality. The agent persona engine 160 can modify the prompt or a response generated by the prompt according to the personality of the agent. The user intention or goal engine can modify a prompt (or an action determined by the prompt) based on the user's current goal or the intention behind the user's action or request. The user's goal or intention may change over time, may be based on a specified user goal, or may be determined based on the user's action. The game lore or constraint engine 164 can modify or adjust a prompt (or an action determined by the prompt) according to the characteristics of the game. For example, the agent may be "possessed" by a non-player character in the game (as discussed in more detail below). The game lore or constraint engine 164 can modify the prompt based on the personality or limitations of the NPC. The various engines of the helper service can be adopted individually or in combination when modifying or adjusting the prompt. Thereafter, the adjusted prompt is provided to one or more related agents 156 for execution.

[0028]

[0037] Figure 2 shows an example of a method 200 for generating a humanoid agent. For example, method 200 may be employed by system 100. The flow starts with an operation 202 of instantiating a game session or receiving an instruction that a game session is being instantiated. As described above, the game session may be hosted by the system executing method 200 or by a user device such as a game console. When the game is instantiated or when an instruction that a game session has been established is received, the flow proceeds to an operation 204 of instantiating an agent as part of the game session. In one example, the agent can be instantiated in response to receiving a request to add an agent to the game session. For example, a request to instantiate an agent within a multiplayer game or an agent controlling an NPC or AI companion within a single-player game may be received. Instantiating the agent may include identifying an agent related to the user playing the game from an agent library. As described above, aspects of the present disclosure enable the generation of agents playable across various games. Thus, the agent instantiated in operation 204 can be instantiated using various components stored as part of a humanoid agent library. For example, the agent can be instantiated using a machine learning model trained to perform game-specific mechanisms and actions, anthropomorphic characteristics learned over time through interactions with play, game data, or lore stored for a particular game or genre, and the agent is instantiated or trained based on a game genre similar to the game in which the agent is instantiated. As discussed previously, the selected agent can be anthropomorphized to suit the user playing the game based on past interactions with the user in the same or a different game as the one started in operation 202.Alternatively, rather than dynamically instantiating an agent using various components stored within an agent library, a specific agent can be selected in operation 204. That is, the agent library can include specific “builds” for various types of agents that are designed by the user or derived by a specific gameplay with the user. These agents are saved and can be instantiated by the user in future game sessions of the same game in which they were first created or in a different game. When an agent is instantiated in operation 204, the agent participates in the game session with the user.

[0029]

[0038] In operation 206, the current game state is interpreted based on audio and visual data and / or API access granted to the agent by the game. As previously described, certain aspects of the present disclosure enable the generation of agents that can interact with games without the need for API access or program access to the game. Thus, the instantiated agent interacts with the game in the same way as the user does, i.e., through game-related audio and visual data. Alternatively, for example, if the agent relies on an NPC to interact with the game, API access to the game can be granted to the agent. In operation 206, various speech recognition, computer vision, object detection, OCR processes, etc. can be employed to process the game state for interpreting communications received from the player (e.g., spoken commands, text-based commands) and the current display view (e.g., using computer vision) or an API for the game state. The current game state is used in operation 208 to generate agent actions. For example, agent actions can be executed based on spoken commands received from the user. Alternatively, agent commands can be generated based on the current view. For example, when an enemy appears on the screen, the enemy can be identified using computer vision and / or object detection, and in operation 208, the agent can generate a command to attack the enemy. Although not shown, operations 206 and 208 can be continuously executed while the game session is active.

[0030]

[0039] The flow proceeds to operation 210 of receiving user feedback. The user feedback received can be explicit. For example, the user may issue a specific command to the agent to perform an action or to change the action currently being performed. Alternatively or in addition, the user feedback may be implicit. Implicit user feedback can be feedback data generated based on the user's interaction with the game. For example, the user may not explicitly provide a command to the agent and instead can adjust their actions or play style based on the current game state and / or in response to the actions performed by the agent. In the example, the user feedback can be continuously collected during the game session. The collected feedback can be related to the concurrent game state or agent actions.

[0031]

[0040] When user feedback is collected, the flow proceeds to operation 212 of generating a prompt for one or more agents based on the user feedback. In the example, the generated prompt is an instruction for executing an agent action in response to the state of the game or a specific user interaction. The prompt can be generated using one or more machine learning models that receive user feedback and / or actions performed by one or more agents and / or existing prompts and / or state data. The output of the machine learning model can be used to generate one or more prompts. In the example, the machine learning model can be trained using information related to the user so that the output from the machine learning model is anthropomorphized for the user. Alternatively or in addition, the machine learning model can be trained with respect to a specific game or application, with respect to a specific group of users (e.g., an e-sports team), etc. Multiple machine learning models can be employed in operation 212 to generate the prompt. In yet other examples, in addition to or instead of using the machine learning model in operation 212, other processes such as a rule-based process may be employed. Further, in operation 212, a new prompt may be generated or an existing prompt may be modified.

[0032]

[0041] When one or more prompts are generated in operation 212, the flow proceeds to operation 214 where one or more prompts are stored for future use by one or more agents. For example, one or more prompts can be stored in an agent library. By storing the prompts generated in 212 using the agent library, the agents can utilize the prompts to interact with the user across various games, thereby providing an anthropomorphic agent that the user can play across various games.

[0033]

[0042] The aspects of the present disclosure described so far have related to importing anthropomorphic agents for playing with a user, but game developers can utilize the systems described herein to provide an accessibility system for assisting players. That is, the anthropomorphic agents can be used to implement accessibility features during a game session. For example, the aspects disclosed herein can be used to assist disabled persons in playing video games. For example, the aspects of the present disclosure can be utilized to assist visually impaired persons, colorblind persons, persons with limb loss who may be affected in game play, or to assist novice gamers in learning how to play a new game.

[0034]

[0043] Figure 3 shows an exemplary system 300 for providing an accessibility service 302 using one or more machine learning models. As shown in system 300, the accessibility service 302 can communicate with a game 304 that is executed on or controlled by a user device. In an example, the game 304 can be executed locally on a user device (such as a game console, a PC, a smartphone, etc.) or on a cloud service network. When the game 304 is executed on a cloud service, the user can still control the game play by a local device 301 (such as a controller, a game console, a PC, etc.). In some examples, the user device 301, the game 304, and the accessibility service 302 can all be executed locally on a single device (such as a game console or a PC). However, as shown in Figure 3, the user device 301, the game 304, and the accessibility service may be executed on different devices connected by a network 303. The network 303 can be a local area network, a wide area network, the Internet, or any other type of network. In some aspects, the accessibility service 302 may be hosted using a cloud network or a distributed network. For example, the accessibility service may utilize one or more machine learning models that may require computing resources not available on the user device.

[0035]

[0044] The accessibility service 302 may include various data repositories, machine learning models, and / or executable code that enable the accessibility service to provide accessibility features to games. That is, the accessibility service 302 can be a service that can be utilized by game 304 without requiring each individual game to implement the functionality of the service. For example, the accessibility service 302 can provide an accessibility SDK that can be utilized by games such as game 304 to incorporate accessibility features without the game developers having to implement the accessibility features in the game. In this way, the accessibility service 302 is a service that provides accessibility features and can be utilized by various different games to create a more inclusive game ecosystem for users with disabilities.

[0036]

[0045] In one aspect, the accessibility service 302 may include user profile data and / or data related to play history. For example, when the user grants permission to the accessibility service 302, the accessibility service 302 can collect and store user profile / play history information 306 related to the user's accessibility preferences (e.g., enabling color blindness mode, enabling one-handed mode, etc.), game preferences (e.g., difficulty level, play style preferences, etc.), and the like. In a further aspect, the user profile / play history information 306 can track the user's play style, abilities, and performance related to various games. The user profile / play history information 306 can be tracked across the various games the user plays, thereby providing a baseline of information and preferences that the accessibility service 302 can utilize when implementing accessibility features for the user when the user plays various and new games.

[0037]

[0046] The accessibility service 302 may also include one or more game models 308. The one or more game models 308 can be any type of machine learning model that is generated, trained, or fine-tuned for a particular game. That is, the accessibility service 302 may have one or more specific machine learning models that are specific to an individual game or an individual game genre (e.g., first-person shooting, role-playing game, puzzle game, massively multiplayer online role-playing game (MMORPG), etc.). The one or more game models can be any type of machine learning model (e.g., a base model, a language model, a computer vision model, a speech model, a video model, an audio model, a multimodal machine learning model, etc.). In various aspects, one or more game models 308 can be generated, trained, or fine-tuned to execute the gameplay of a particular game (or game genre) based on data collected from general play settings, based on bot simulations, or a combination of both. In an example, the game model 308 represents the standard gameplay of a particular game (or game genre). Additionally, different game models 308 may be generated to reflect various gameplay capabilities (e.g., beginner gameplay, advanced gameplay, expert gameplay, etc.). The game model 308 can be used by the accessibility service 302 as a baseline gameplay generator to generate controls for playing a particular game (or game genre) at a desired skill level.

[0038]

[0047] The game model 308 can be utilized by the accessibility service 302 to generate gameplay for a specific game, but the game model 308 is essentially general-purpose and is not trained to assist users with specific disabilities. To provide assistance to players with specific disabilities (such as visual impairments, limb deficiencies, etc.), the accessibility service includes one or more cohort models 310. The game model 308 is generated, trained, or fine-tuned to generate gameplay commands based on a general user base, while the cohort model 310 is one or more machine learning models (such as a base model, language model, computer vision model, speech model, video model, audio model, multimodal machine learning model, etc.) that are generated, trained, or fine-tuned to generate gameplay commands that can assist users with specific disabilities (such as a cohort of disabled users). For example, there may be a specific cohort model 308 for generating specific gameplay commands for assisting users with specific disabilities (such as color vision impairment, visual impairment, playing with one hand, etc.). The cohort model 310 can be generated or trained using a dataset of gameplay by users with specific disabilities, thereby generating gameplay commands that are specially adjusted to assist users with specific disabilities. Although the embodiments have been described as including different models for specific disabilities, those skilled in the art will understand that a single model for generating gameplay commands that assist with multiple disabilities may be generated, trained, or fine-tuned.

[0039]

[0048] The accessibility service 302 may also include a user-specific model 312. For example, assume that an individual user provides an accessibility service for collecting data regarding the individual user's gameplay, and one or more machine learning models specific to the individual user's gameplay (e.g., a base model, a language model, a computer vision model, a speech model, a video model, an audio model, a multimodal machine learning model, etc.). These user-specific models 312 can be generated using individual data from other game sessions (e.g., data from the user profile / play history 306) or individual data from past game sessions of the currently-played game (e.g., game 304). Further, the user-specific model 312 is continuously updated based on the individual user's gameplay, such that as the individual user continues to play the game and improve their skills in a particular game (or genre of games), the model can adapt to the improvement of that user. Similar to other machine learning models (e.g., game models, cohort models, etc.), the user-specific model 312 can receive game data (e.g., game state, user input during gameplay, etc.) and generate user-specific game play assistance commands.

[0040]

[0049] The exchange of gameplay data and accessibility commands can be facilitated by the Accessibility SDK 314. As described above, the Accessibility SDK 314 exposes the accessibility service 302 to various games such as game 304 so that the game can provide accessibility features using one or more machine learning models without the need to individually implement the accessibility features in the game itself. Thus, developers can easily add accessibility features to the game, thereby creating a more inclusive game environment. The Accessibility SDK 314 enables the transmission and reception of gameplay data, user input, and assistive controls generated by the accessibility service during a gameplay session.

[0041]

[0050] In an exemplary operation, various machine learning models of the accessibility service (e.g., game model 308, cohort model 310, and user-specific model 314) receive the current gameplay data of game 304 and can generate assistance commands to help a user with a disability play the game when the user with a disability plays the game. In one example, one or more of various types of machine learning models can receive gameplay data including, for example, the current game state, the current user display (e.g., the current view of the user), commands input by the user, etc. as input, and the input is processed using a machine learning model to generate commands that can assist the user when controlling the game. In one aspect, these models can make individual decisions, and the decisions can be input to other models (e.g., along with the gameplay data described above) to generate commands specific to assisting the user based on the user's disability. For example, game model 308 can receive gameplay data to generate general commands (e.g., commands based on the set of all users). The general commands can be provided to the cohort model with or without the gameplay data, and the cohort model can generate commands adjusted to assist users with disabilities similar to the user playing game 304 by the user device 301. Subsequently, the adjusted commands can be provided as input to the user-specific model with or without the gameplay data, and the user-specific model can generate user-specific commands (e.g., adjusted based on the specific needs or preferences of the user). Subsequently, to assist the user playing the game, the user-specific commands can be provided to the game, for example, via network 303, to be implemented along with the user's input. Those skilled in the art will understand that the type of assistance can vary depending on the needs of individual users, the current gameplay, the limitations of the gameplay, and / or the current game state.By doing so, the accessibility service 302 is operable to provide any number of different assistive functions (e.g., aiming assistance, object detection, game play hints, assisted steering or driving control, additional button inputs) based on the user's needs and preferences. Further, while the aspects described herein teach the use of multiple machine learning models, those skilled in the art will understand that a single machine learning model may be implemented to perform the functions of the various types of models described above. Thus, the distinction between different models is made to facilitate the description and explanation of the various functions provided by the accessibility service, and those skilled in the art will understand that the present disclosure is not limited to implementations using multiple different machine learning models.

[0042]

[0051] Next, move to game 304, which may include various components that can be used to interface with the accessibility service 320 and / or implement the assistive controls received from the accessibility service 302. For example, game 304 may include an accessibility request component 316, a game constraint analyzer 318, an accessibility service interface 320, and an input execution engine 322. In one aspect, these various components may be implemented as part of the game, and the game developer may be able to control the extent to which the accessibility commands provided by the accessibility service 302 can control the game play. For example, in some situations such as competitive multiplayer games, the game developer may want to limit the amount of control that the accessibility service 302 can exercise preferentially over the game play to ensure a fair competitive environment. However, the functions described as being performed by the game 304 within the system 300 may also be performed by the accessibility service 302 in other implementations. That is, while specific functions are described as being performed by a particular actor within the system 300, such functions can be performed using different actors without departing from the scope of the present disclosure.

[0043]

[0052] The accessibility requirement component 316 can request a list of accessibility features that can be provided to the game, for example, by using calls to the accessibility SDK 314. For example, the game 304 can query the accessibility service 302 at startup to receive a list of available accessibility features. The list of features can be incorporated into the game menu. The game 304 can display the accessibility features within the game to the user. Through this menu user interface, the game 304 can receive a selection from the user of the accessibility features to be employed (such as color blindness mode, tutorial hints, driving assistance, aiming assistance, etc.). The selected accessibility features can then be implemented during game play. For example, the game can send the selected accessibility features to the accessibility service 302, thereby instructing the accessibility service to generate controls for implementing the selected features during game play. If the user has selected an accessibility feature in the past, the selected feature is stored and can be loaded during subsequent game sessions, thereby instructing the accessibility service 302 to provide that feature without the user having to reselect the desired feature each time the user launches the game 304. In various aspects, the commands received from the accessibility service 302 can be of various different types. For example, the command can be an input command to assist the user in controlling game play. The input command can be related to, for example, aiming assistance (such as adjusting aim during game play), driving assistance, additional button inputs (such as for one-handed game play), etc. That is, the command received from the accessibility service 302 can be input as if it were a command generated using an input device such as a controller, mouse, or keyboard connected to the user device 301. Alternatively or in addition, the command can be a command to adjust the game environment or a command to provide additional information to the user.Examples of such commands can be commands that highlight or visually mark objects (e.g., to assist users with visual impairments), commands for displaying hints or advice, and the like. Commands received by the accessibility service interface 320 can be implemented by the game 304.

[0044]

[0053] However, as described above, the game developer can impose restrictions on the types of commands, the degree of commands, or the types of commands that can be implemented by the accessibility service 302. Therefore, a game constraint analyzer 318 may be provided that can evaluate the commands received from the accessibility service and determine whether to implement the commands, modify the commands, or block the execution of the commands. As described above, there may be reasons why the game developer wishes to block assistance. For example, during a competitive multiplayer game, in order to ensure fair gameplay among all players, the game developer may wish to restrict or block aiming assistance. Or alternatively, in addition, the game developer may wish to block certain hints or advice, because providing such advice (such as providing the solution to a puzzle, providing information that can ruin future game events, etc.) can ruin the user's gameplay. Therefore, the game constraint analyzer 318 can analyze the received commands in light of the restrictions set by the game developer or based on the restrictions set by the user to ensure that the accessibility service 302 does not have an adverse impact on the gameplay. As described above, in some examples, restricted commands can be blocked or modified by the game constraint analyzer 318 to conform to the game constraints. For example, if the accessibility command is a control command, the game constraint analyzer 318 may modify the control command (such as by adjusting the degree of aiming assistance or steering assistance) to adjust the degree of control. If the command is to highlight an item in the game, the game constraint analyzer 318 may block the command or reduce the degree of highlighting the item, so that it is not so obvious that the user playing the game can see it without difficulty, but there is still some degree of assistance. The above-mentioned modifications can take the form of actually modifying the input or graphical commands generated by the accessibility service 302.

[0045]

[0054] Aspects of the present disclosure are directed to assisting a user's gameplay. However, if the assistance overrides the player's own actions and thereby deprives the player of their agency, such assistance is not welcome by many players. That is, the accessibility service 302 is not intended to hijack the user's gameplay, but rather is intended to adjust the user's gameplay to assist users who may have disabilities (or may desire additional assistance). That is, the commands received by the game from the accessibility service 302 are intended to be used with, rather than instead of, the commands received from the player, i.e., the commands received from the user device 301. Therefore, an input execution engine 322 is provided for implementing gameplay actions based on commands received from both the user device 301 and the accessibility service 302. For example, if the user is steering a vehicle to the left and the accessibility command received from the accessibility service is to steer the vehicle to the right, rather than overriding the user command, the input execution engine can essentially mix the commands by, for example, reducing the degree to which the user command turns the vehicle to the left instead of completely overriding the command. Further, the input execution engine can continuously monitor the user commands and compare them to the commands generated by the accessibility service 302. If the user commands continue to conflict with the commands generated by the accessibility service 302, the input execution engine 322 can completely ignore the commands generated by the accessibility service 302, thereby maintaining the player's agency over the control of the game. In the foregoing situation, the input execution engine can also provide the user with a notification that certain accessibility features that conflict with the user's input are enabled, along with an option to disable the specific assistance features via the user interface. By doing so, the system 300 provides complementary accessibility services that can be used to assist gameplay without overriding the user's actions, thereby maintaining the player's agency during the game session.

[0046]

[0055] Figure 4 shows an exemplary method 400 for instantiating an accessibility machine learning model to assist a user during a game play session. The flow begins with an operation 402 of receiving a request from the user or the game for providing an accessibility function. For example, a user interacting via the user interface of a game such as game 304 (Figure 3) can select a set of accessibility functions for enabling with respect to the game play. Upon receiving the selected accessibility function, the game can transmit a request to an accessibility service such as accessibility service 302 (Figure 3) to provide the selected accessibility function. The request is received at operation 402. In one aspect, the request can be received with a unique identifier of the user, such as the user's gamer tag or another unique identifier of the user. Alternatively or in addition, the request can also include an identifier of the game that is requesting the accessibility function. Further, the selected accessibility function can be identified using a specific function identifier (e.g., an identifier corresponding to aiming assistance, an identifier corresponding to color blindness mode, etc.). Alternatively, the selected accessibility function can be identified using natural language (e.g., "enable aiming assistance", "enable one-handed control", etc.).

[0047]

[0056] The flow continues to operation 404 which receives or obtains user profile data. For example, based on a user identifier (such as a game tag) related to the request received in operation 402, the accessibility service can obtain for the user the related user profile data stored by it. The user profile data can include data regarding the user's past play history with respect to a particular game, related game genres, etc. The profiled data can be stored locally by the accessibility service as needed or can be requested from the user device. The user profile data can be used to select a particular machine learning model that generates an accessibility function, and / or to establish the context of the user's game play session, and / or to generate prompts for one or more selected machine learning models to assist in the adjustment of the one or more selected models. Similarly, in operation 406, the user's play history related to the game for which the accessibility function is requested can be received or obtained. The user's play history can similarly be used to select a particular machine learning model that generates control of the accessibility function, or to prompt one or more selected machine learning models to generate a particular level of accessibility commands. For example, if the user's game play history indicates a tendency for the user's aiming to improve with each game play session, the game play history received or obtained in operation 406 can be used to select a machine learning model that provides a lesser degree of aiming assistance, or to prompt the selected machine learning models to generate aiming accessibility commands that are less intrusive. Thus, the profile data and play history data received in operations 404 and 406 enable the accessibility service to automatically adjust the degree of assistance (more assistance or less assistance) based on the user's game play history.

[0048]

[0057] The flow proceeds to operation 408, where one or more machine learning models are identified to provide an accessibility function based on the received requests and / or received user profile data and play history data. For example, one or more game models, one or more cohort models, and one or more user-specific models may be identified by the accessibility service to provide the requested accessibility function. The flow then continues to operation 410, as further described in FIG. 5A, where the selected model is instantiated to provide the accessibility function during the game play session.

[0049]

[0058] The operations of method 400 can be performed using a single device or multiple devices. For example, the operations can be performed using a single device (such as a PC or game console) that generates the accessibility function. Alternatively, the user device may interact with a cloud service to perform the operations of method 400. Those skilled in the art will understand that the operations can be performed using a single device or multiple devices and that different operations can be performed on different devices without departing from the scope of the present disclosure.

[0050]

[0059] FIG. 5A shows an exemplary method 500 for generating accessibility commands using one or more machine learning models. The flow begins with an operation 502 of instantiating one or more machine learning models to provide an accessibility function. For example, the process detailed with respect to method 400 can be employed to generate one or more models. Alternatively or in addition, specific game constraints can be used to select one or more machine learning models. The flow then continues to an operation 504 of receiving current game play data. In one example, the game play data can be received through integration with the game by an API such as the API exposed by the accessibility SDK described in FIG. 3. The game play data can include the current game state, current received input by one or more users, environmental information, actions of NPCs or other players, or any other kind of data related to the current game play. Alternatively or in addition, the current game play data can be received using computer vision. For example, the current view of the game from the user's perspective can be provided to a computer vision tool to evaluate the state of the game play based only on the display. By doing so, the game play data can be analyzed based on the view of the game display, thereby generating game play data without the need for API access to the underlying game data. Thus, aspects of the present disclosure can be provided without the need for access to the game's code or internal game data.

[0051]

[0060] The flow proceeds to operation 506, where one or more machine learning models are used to analyze the current gameplay data to determine modifications to gameplay actions and / or game states based on the selected accessibility function. For example, as described above, various machine learning models of the accessibility service (e.g., game model, cohort model, and user-specific model) receive the current gameplay data during gameplay and can generate assistive commands to assist the gameplay of a user with a disability when the user with a disability plays the game. In one example, one or more of various types of machine learning models can receive, as input, gameplay data including, for example, the current game state, the current user display (e.g., the current view of the user), commands input by the user, etc., and the input is processed using the machine learning model to generate commands that can assist the user in controlling the game. In one aspect, these models can make individual decisions, and the decisions can be input to other models (e.g., along with the gameplay data described above) to generate commands specific to assisting the user based on the user's disability. For example, the game model can receive gameplay data to generate general commands (e.g., commands based on the set of all users). The general commands can be provided to the cohort model with or without the gameplay data, and the cohort model can generate commands adjusted to assist users having disabilities similar to the user playing the game on the user device. Subsequently, the adjusted commands can be provided as input to the user-specific model with or without the gameplay data, and the user-specific model can generate user-specific commands (e.g., adjusted based on the specific needs or preferences of the user). Subsequently, the user-specific commands can be provided to the game to be implemented along with the user's input to assist the user playing the game. Alternatively, in some aspects, a single machine learning model can be instantiated to provide the assistive function.

[0052]

[0061] The accessibility features represented by accessibility commands that take the form of game play commands, game state commands, or a combination of both are generated using one or more machine learning models and then provided to the game for execution at operation 508. For example, commands generated by one or more machine learning models are provided to the game for execution along with received input by the user. The flow continues to operation 510 which optionally tracks user input made in response to the accessibility feature. As described above, aspects of the present disclosure relate to providing a complementary system that assists a user during game play in a way that does not take away the user's agency. If the user provides an input that conflicts with the accessibility command, such as moving the vehicle or adjusting the aim contrary to the command provided by one or more machine learning models, the assistance model can be adjusted to generate commands that do not go against the user. Thus, the user's response to the command can be tracked at operation 510 and stored as potential feedback data. Along with the commands generated by one or more models and / or game play data, the tracked user input can be saved at operation 510 and used in a fine-tuning process or a training process to update one or more models at operation 512. By doing so, one or more models can be continuously updated using feedback or reinforcement learning, thereby adapting to the user's play style, preferences, and improvement of play as the user continues to play the game. In this way, method 500 provides a mechanism for continuously updating machine learning models used to provide player assistance to maximize the user's game play.

[0053]

[0062] The operations of method 500 can be performed using a single device or multiple devices. For example, the operations can be performed using a single device (such as a PC or a game console) for generating accessibility features. Alternatively, a user device may interact with a cloud service to perform the operations of method 500. It will be understood by those skilled in the art that, without departing from the scope of the present disclosure, the operations can be performed using a single device or multiple devices, and different operations can be performed on different devices.

[0054]

[0063] FIG. 5B shows an exemplary method 520 for implementing an accessibility feature during game play. The flow begins with operation 522 of displaying a user interface showing the accessibility options available in the game. In one example, the available accessibility options can be determined based on the accessibility features that can be provided by an accessibility service such as the accessibility service 302 of FIG. 3. That is, the accessibility features do not necessarily have to be specifically provided by the game. In operation 524, one or more accessibility options can be selected from the accessibility user interface. Next, in operation 506, the selected option is sent to the accessibility service. The selected accessibility option can be used to instantiate one or more machine learning models for providing the accessibility feature by the accessibility service as described herein.

[0055]

[0064] It will be understood by those skilled in the art that operations 522 through 526 may be performed dynamically. That is, the accessibility user interface can be invoked at any time during game play, and different accessibility features can be selected at that time. Further, the selected accessibility feature can be saved, so that the user does not have to select the accessibility feature each time the game is played. Rather, the accessibility feature selected in the past can be loaded during subsequent game sessions.

[0056]

[0065] The flow proceeds to operation 520 of receiving an accessibility command from the accessibility service. The accessibility command can take the form of a gameplay command (e.g., a command for controlling a player's action during the game), a game state command (e.g., a command for changing the state of the game such as the difficulty level of the game, or the visual state of the game such as highlighting an item or adjustment for color vision deficiency, or a command for providing advice or hints for gameplay), or a combination of both. The flow proceeds to operation 530 of optionally modifying the accessibility command based on game constraints. For example, as described above, in order to ensure that the accessibility service does not adversely affect gameplay, the received accessibility command can be analyzed in light of the restrictions set by the game developer or based on the restrictions set by the user. As mentioned earlier, in some examples, restricted accessibility commands can be blocked or modified to conform to the game constraints. For example, if the accessibility command is a gameplay control command, the degree of control can be adjusted by modifying the accessibility command (e.g., by adjusting the degree of aiming assistance or steering assistance). If the command is one that highlights an item in the game, the accessibility command can be blocked or the degree of highlighting the item can be reduced so that it is not so obvious that it is difficult for the user playing the game to see, but there is still some degree of assistance. The above modifications can take the form of actually modifying the input or graphical commands generated by the accessibility service.

[0057]

[0066] In operation 508, the game executes an accessibility command, which may or may not have been modified. In some examples, the execution of the accessibility command may occur regardless of user interaction. For example, an accessibility command that changes the game display (such as highlighting an object or providing advice) may be executed regardless of the corresponding user input. However, the control of game play commands can be executed in conjunction with the received user input in a way that adjusts or complements the received user input, as opposed to completely overriding the received user input. By doing so, the user maintains their agency over their game play.

[0058]

[0067] The process of receiving, modifying, and executing commands can continue during a game play session. However, in some examples, an instruction to change the assistive command may be automatically received. For example, if the user continues to adjust contrary to the game play function, such as correcting a movement made by aim assist, or if it is determined that some accessibility features are no longer necessary based on the improvement of the user's game play, an instruction to change the accessibility feature can be received in operation 534. This change can cause the game to display user interface elements in operation 536. The displayed user interface elements can provide information regarding the change of the assistive function, along with activatable elements that allow the user to modify or change the accessibility feature, such as by removing aim assist or adjusting it to be less intrusive. Upon receiving a selection of a user interface element, the flow proceeds to operation 538, which updates the accessibility feature, saves the corresponding changes, and transmits them to the accessibility service.

[0059]

[0068] Figure 6 shows an exemplary method 600 for modifying accessibility features based on changes in a user's gameplay. The flow begins at operation 602, where gameplay actions determined by one or more machine learning models, such as accessibility commands, are tracked during gameplay, along with actions taken by the user in response to the accessibility commands. At operation 604, the user's actions are compared to the gameplay actions (e.g., accessibility commands) to reveal differences between the accessibility commands and the user input. For example, the accessibility commands can be compared to the user input to reveal the degree of difference between the actions generated by the machine learning model and the user input. If the degree of difference is below a threshold, it can be determined that the user no longer requires a particular type of assistance. Alternatively or in addition, if the degree of difference between the actions generated by the machine learning model and the user input is relatively high, or if subsequent received input by the user indicates that the user is compensating against the actions generated by the model, it can be determined that the user has not consented to the proposed actions. In the foregoing example, a determination can be made to modify the level of assistance provided or to completely disable the accessibility feature.

[0060]

[0069] At operation 606, based on the revealed differences, one or more proposals for modifying the accessibility options can be generated. The proposed adjustments can be displayed along with interactive user interface elements that allow the user to select the modifications (as described in FIG. 5B). At operation 608, a request to modify the accessibility feature according to the proposed actions can be received by the user interface elements. In some examples, the updated selections can be transmitted from the game to the accessibility service. At operation 610, future accessibility commands can be generated based on the updated accessibility feature.

[0061]

[0070] Figure 7 is a block diagram showing the physical components (e.g., hardware) of a computing device 700 in which aspects of the present disclosure may be practiced. The components of the computing device described below may be suitable for the computing device described above. In a basic configuration, the computing device 700 may include at least one processing unit 702 and a system memory 704. Depending on the configuration and type of the computing device, the system memory 704 may include, but is not limited to, a volatile storage area (e.g., random access memory), a non-volatile storage area (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 704 can include an operating system 705 and one or more program tools 706 suitable for executing the various aspects disclosed herein. The operating system 705 may be suitable for controlling the operation of the computing device 700, for example. Further, aspects of the present disclosure may be practiced with a graphics library, other operating systems, or any other application programs, and are not limited to a particular application or system. This basic configuration is shown in Figure 7 by the components within the dashed line 708. The computing device 700 can have additional features or functionality. For example, the computing device 700 may also include additional (removable and / or non-removable) data storage devices such as, for example, magnetic disks, optical disks, or tapes. Such additional storage areas are shown in Figure 7 by a removable storage device 709 and a non-removable storage device 710.

[0062]

[0071] As described above, several program tools and data files can be stored in the system memory 704. While being executed on at least one processing unit 702, the program tool 706 (e.g., the application 720) can execute a process including, but not limited to, the aspects described herein. The application 720 includes the accessibility model 730, the accessibility user interface 732, the accessibility instructions 734, and the instructions for executing various processes disclosed herein. Other program tools that can be used in accordance with aspects of the present disclosure can include, for example, email and contact applications, document processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided application programs, and the like.

[0063]

[0072] Aspects of the present disclosure can be practiced within an electrical circuit that includes discrete electronic components, in a packaged or integrated electronic chip that includes logic gates, within a circuit that utilizes a microprocessor, or on a single chip that includes an electronic component or a microprocessor. For example, aspects of the present disclosure can be practiced by a system-on-a-chip (SOC), in which each or many of the components shown in FIG. 7 can be integrated on a single integrated circuit. Such an SOC device can include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned in") onto a chip substrate as a single integrated circuit. When operating by an SOC, the functions described herein with respect to the ability of a client to switch protocols can operate by application-specific logic integrated with other components of a computing device 700 on a single integrated circuit (chip). Aspects of the present disclosure can be practiced using other techniques, including but not limited to mechanical, optical, fluidic, and quantum techniques, that are capable of performing logical operations such as AND, OR, and NOT. Additionally, aspects of the present disclosure can be practiced within a general-purpose computer or within any other circuit or system.

[0064]

[0073] The computing device 700 can also include one or more input devices 712 such as a keyboard, a mouse, a pen, a voice or voice input device, a touch or swipe input device, etc. It can also include an output device 714 such as a display, a speaker, a printer, etc. The foregoing devices are examples, and other devices can be used. The computing device 700 can include one or more communication connections 716 that enable communication with other computing devices 750. Examples of communication connections 716 include, but are not limited to, radio frequency (RF) transmitter, receiver, and / or transceiver circuits, universal serial bus (USB), parallel and / or serial ports.

[0065]

[0074] As used herein, the term computer-readable medium may include computer storage media. Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, or program tools. System memory 704, removable storage device 709, and non-removable storage device 710 are all examples of computer storage media (e.g., memory storage areas). Computer storage media can include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage area, magnetic cassette, magnetic tape, magnetic disk storage area or other magnetic storage device, or any other manufacture that can be used to store information and is accessible by computing device 700. Such any computer storage media can be part of computing device 700. Computer storage media does not include carrier waves or other propagated data signals or modulated data signals.

[0066]

[0075] Communication media may be embodied by computer-readable instructions, data structures, program tools, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may represent a signal having one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, communication media can include wired media such as a wired network or direct wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0067]

[0076] FIG. 8 shows a computing device or mobile computing device 800 in which aspects of the present disclosure may be practiced, such as a cellular phone, smartphone, wearable computer (such as a smartwatch), tablet computer, laptop computer, and the like. In some aspects, the client utilized by a user (e.g., the client device 102 shown within the system 100 of FIG. 1) may be a mobile computing device. FIG. 8 is a block diagram showing an architecture of one aspect of a computing device, server, mobile computing device, and the like. That is, the mobile computing device 800 can incorporate a system 802 (e.g., a system architecture) for implementing some aspects. The system 802 can be implemented as a "smartphone" capable of executing one or more applications (e.g., a browser, email, calendar, contact management, messaging client, game, and media client / player). In some aspects, the system 802 is integrated as a computing device such as a personal digital assistant (PDA) and a wireless phone.

[0068]

[0077] One or more application programs 866 are loaded into the memory 862 and can be executed on or in relation to the operating system 864. Examples of application programs include a telephone dialer program, an email program, a personal information management (PIM) program, a document processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 802 also includes a non-volatile memory area 868 within the memory 862. The non-volatile memory area 868 can be used to store persistent information that should not be lost when the power of the system 802 is turned off. The application program 866 can use and store information within the non-volatile memory area 868, such as emails or other messages used by an email application. A synchronization application (not shown) is also on the system 802 and is programmed to interact with a corresponding synchronization application on a host computer to continuously synchronize the information stored in the non-volatile memory area 868 with the corresponding information stored on the host computer. As should be understood, other applications may be loaded into the memory 862 and executed on the mobile computing device 800 described herein.

[0069]

[0078] The system 802 has a power source 870 that can be implemented as one or more batteries. The power source 870 can further include an external power source such as an AC adapter or a powered docking station that supplements or charges the battery.

[0070]

[0079] System 802 can also include a wireless interface layer 872 that performs the function of transmitting and receiving radio frequency communications. The wireless interface layer 872 facilitates a wireless connection between System 802 and the "outside world" via a communication carrier or service provider. Transmissions to and from the wireless interface layer 872 are performed under the control of the operating system 864. In other words, communications received by the wireless interface layer 872 can be spread to the application program 866 by the operating system 864, and vice versa.

[0071]

[0080] The visual indicator 820 (e.g., an LED) can be used to provide visual notifications, and / or the voice interface 874 can be used to create audible notifications by the voice converter 825. In the illustrated configuration, the visual indicator 820 is a light-emitting diode (LED), and the voice converter 825 is a speaker. These devices can be directly coupled to the power supply 870 so as to remain on for a period directed by the notification mechanism at startup, even if the processor 860 and other components may shut down to conserve battery power. The LED can be programmed to remain on indefinitely to indicate the powered-on state of the device until the user takes an action. The voice interface 874 is used to provide an audible signal to the user and to receive an audible signal from the user. For example, in addition to being coupled to the voice converter 825, the voice interface 874 may also be coupled to a microphone to receive audible input to facilitate a telephone conversation. According to an aspect of the present disclosure, the microphone may also function as an audio sensor to facilitate control of notifications as described below. System 802 may further include a video interface 876 that enables the operation of a device connected to the peripheral device port 830 for recording still images, video streams, etc.

[0072]

[0081] The mobile computing device 800 implementing the system 802 can have additional features or functions. For example, the mobile computing device 800 can also include additional (removable and / or non-removable) data storage devices such as magnetic disks, optical disks, or tapes. Such additional storage areas are shown by the non-volatile storage area 868 in FIG. 8B.

[0073]

[0082] The data / information generated or captured by the mobile computing device 800 and stored by the system 802 can be stored locally on the mobile computing device 800 as described above, or the data can be stored on any number of storage media, and those storage media can be accessed by the device via the wireless interface layer 872 or via a wired connection between the mobile computing device 800 and another computing device associated with the mobile computing device 800, such as a server computer in a distributed computing network such as the Internet. As should be understood, such data / information can be accessed by the mobile computing device 800 via the wireless interface layer 872 or via the distributed computing network. Similarly, such data / information can be easily transferred between computing devices for storage and use according to known data / information transfer and storage means including email and collaborative data / information sharing systems.

[0074]

[0083] As understood from the above disclosure, one aspect of this technology relates to a computer-implemented method for providing accessibility features to a game using accessibility services, the method comprising receiving a selection of an accessibility feature for the game; instantiating one or more machine learning models based on the accessibility feature, the one or more machine learning models being operable to generate commands for implementing the accessibility feature; receiving current game play data; generating an accessibility command using the one or more machine learning models, the current game play data being provided as input to the one or more machine learning models; and providing the accessibility command to the game, providing the accessibility command causing the accessibility feature to be implemented during game play.

[0075]

[0084] In one example, the game play data includes one or more of current user input, game state information, information about other player characters, or non-player character information.

[0076]

[0085] In another example, receiving the current game play data includes receiving the current view of the game, the current view of the game being the view depicted to the player during game play.

[0077]

[0086] In yet another example, the current game play data further includes processing the current view of the game using computer vision to generate the game play data.

[0078]

[0087] In still another example, instantiating the one or more machine learning models includes instantiating at least one of a game machine learning model, a cohort machine learning model, or a user-specific machine learning model.

[0079]

[0088] In a further example, a cohort machine learning model trained to generate accessibility commands for a particular impairment.

[0080]

[0089] In yet a further example, a computer-implemented method further includes receiving user input responsive to the provision of an accessibility command and corresponding to an adjustment made by the accessibility command.

[0081]

[0090] In another example, a computer-implemented method further includes updating one or more machine learning models based on user input corresponding to an adjustment made by an accessibility command.

[0082]

[0091] In another aspect, the technology relates to a computer-implemented method for providing an accessibility user interface for a game based on accessibility features provided by an accessibility service, the method including generating a user interface depicting a plurality of accessibility features, the plurality of accessibility features including accessibility features provided by the accessibility service, the accessibility service being a service separate from the game; receiving a selection of a first accessibility feature provided by the accessibility service; transmitting the first accessibility feature to the accessibility service; in response to transmitting the accessibility feature to the accessibility service, receiving a plurality of accessibility commands from the accessibility service during game play; and executing the plurality of accessibility commands to implement the first accessibility feature.

[0083]

[0092] In one example, transmitting the first accessibility feature to the accessibility service further includes transmitting a unique identifier of the player along with the first accessibility feature.

[0084]

[0093] In another example, the first accessibility command includes a gameplay control command, and executing the gameplay control command includes generating a gameplay action based on the gameplay control command and user input.

[0085]

[0094] In yet another example, generating a gameplay action further includes modifying user input based on a gameplay control command.

[0086]

[0095] In a further example, generating a gameplay action further includes supplementing user input with a gameplay control command.

[0087]

[0096] In another example, executing a plurality of accessibility commands includes comparing a first accessibility command to game constraints, determining a modification to the first accessibility command based on the comparison of the first accessibility command, executing the modified first accessibility command, comparing a second accessibility command to game constraints, and executing the second accessibility command without modification based on the comparison of the second accessibility command.

[0088]

[0097] In yet another example, a computer-implemented method further includes receiving a proposal to change an accessibility feature, generating a user interface element based on the proposal, displaying a user interface for changing the accessibility feature during gameplay, receiving a selection of the user interface element, and in response to receiving the selection, sending a second accessibility feature to an accessibility service.

[0089]

[0098] In yet another aspect, when executed by at least one processor, this technique involves receiving a selection of an accessibility function for a game, instantiating one or more machine learning models based on the accessibility function, where the one or more machine learning models are operable to generate commands for implementing the accessibility function, receiving current game play data, using the one or more machine learning models to generate accessibility commands, where the current game play data is provided as input to the one or more machine learning models, generating the accessibility commands, and providing the accessibility commands to the game, where providing the accessibility commands causes the accessibility function during game play to be implemented. The computer storage medium encodes computer-executable instructions for causing at least one processor to execute a method including the above.

[0090]

[0099] In one example, instantiating one or more machine learning models includes instantiating at least one of a game machine learning model, a cohort machine learning model, or a user-specific machine learning model.

[0091]

[0100] In yet another example, a cohort machine learning model trained to generate accessibility commands for a particular impairment.

[0092]

[0101] In yet another example, a user-specific machine learning model is a machine learning model trained to generate accessibility commands for a particular user.

[0093]

[0102] The description and explanatory drawings of one or more aspects shown in this application are not intended to limit or restrict the scope of the present disclosure as set forth in the claims. The disclosure set forth in the claims should not be construed as limited to any aspect or detail provided in this application, for example. Various features (structural and methodological of the agent) may be selectively included or omitted in order to create an aspect having a particular set of features, whether illustrated and described in combination or separately. Having provided the description and explanatory drawings of this application, those skilled in the art can devise modifications, corrections, and alternative aspects that are encompassed by the broader aspects of the general inventive concept embodied in this application without departing from the broader scope of the disclosure set forth in the claims.

Claims

1. A computer-implemented method for providing an accessibility function to a game using an accessibility service, comprising: Receiving a selection of an accessibility function for the game (402); Instantiating one or more machine learning models based on the accessibility function (410), wherein the one or more machine learning models are operable to generate commands for implementing the accessibility function; Receiving current game play data (504); Generating an accessibility command using the one or more machine learning models (506), wherein the current game play data is provided as input to the one or more machine learning models; Providing the accessibility command to the game (508), wherein providing the accessibility command causes the accessibility function to be implemented during game play; A computer-implemented method comprising the above steps.

2. The computer-implemented method according to claim 1, wherein the game play data includes one or more of: Current user input; Game state information; Information about other player characters; or Non-player character information.

3. The computer-implemented method according to claim 1, wherein receiving current game play data includes receiving the current view of the game, and the current view of the game is the view depicted to the player during game play.

4. The computer-implemented method according to claim 1, wherein receiving current game play data further includes using computer vision to process the current view of the game to generate game play data.

5. The computer-implemented method according to claim 1, wherein instantiating one or more machine learning models includes instantiating at least one of: A game machine learning model; A cohort machine learning model; or A user-specific machine learning model.

6. The computer-implemented method according to claim 5, wherein the cohort machine learning model is trained to generate accessibility commands for a specific disability.

7. ​ ​ The computer-implemented method according to claim 5, wherein the user-specific machine learning model is a machine learning model trained to generate accessibility commands for a specific user.

8. The computer-implemented method according to claim 1, further comprising receiving user input corresponding to an adjustment made by the accessibility command in response to providing the accessibility command.

9. The computer-implemented method according to claim 8, further comprising updating the one or more machine learning models based on the user input corresponding to the adjustment made by the accessibility command.

10. A computer-implemented method for providing an accessibility user interface for a game based on accessibility features provided by an accessibility service, comprising: generating a user interface depicting a plurality of accessibility features (522), wherein the plurality of accessibility features includes accessibility features provided by the accessibility service, and the accessibility service is a service separate from the game; receiving a selection of a first accessibility feature provided by the accessibility service (524); transmitting the first accessibility feature to the accessibility service (526); receiving, during gameplay, a plurality of accessibility commands from the accessibility service in response to transmitting the accessibility feature to the accessibility service (528); and executing the plurality of accessibility commands to implement the first accessibility feature (532). A computer-implemented method.

11. The computer-implemented method according to claim 10, wherein transmitting the first accessibility feature to the accessibility service further comprises transmitting a unique identifier of the player along with the first accessibility feature.

12. The computer-implemented method according to claim 10, wherein the first accessibility command includes a gameplay control command, and executing the gameplay control command includes generating a gameplay action based on the gameplay control command and user input.

13. The computer-implemented method according to claim 12, wherein generating the game play action further comprises modifying the user input based on the game play control command.

14. The computer-implemented method according to claim 12, wherein generating the game play action further comprises supplementing the user input with the game play control command.

15. Executing a plurality of accessibility commands includes comparing a first accessibility command with game constraints, determining a modification to the first accessibility command based on the comparison of the first accessibility command, executing the modified first accessibility command, comparing a second accessibility command with the game constraints, and executing the second accessibility command without modification based on the comparison of the second accessibility command The computer-implemented method according to claim 10, further comprising.

16. Receiving a proposal to change an accessibility function, Generating user interface elements based on the proposal, Displaying the user interface for changing the accessibility function during game play, Receiving a selection of the user interface element, and In response to receiving the selection, transmitting a second accessibility function to the accessibility service The computer-implemented method according to claim 10, further comprising.

17. When executed by at least one processor, Receiving a selection of an accessibility function for a game (402), Instantiating one or more machine learning models based on the accessibility function (410), wherein the one or more machine learning models are operable to generate commands for implementing the accessibility function, Receiving current game play data (504), and Generating an accessibility command using the one or more machine learning models (506), wherein the current game play data is provided as input to the one or more machine learning models Providing the accessibility commands to the game (508), wherein providing the accessibility commands implements the accessibility function during gameplay, and A computer storage medium encoding computer-executable instructions for causing the at least one processor to execute a method comprising the same. **Claim 18** Instantiating one or more machine learning models, A game machine learning model, A cohort machine learning model, or A user-specific machine learning model The computer storage medium according to claim 17, comprising instantiating at least one of. **Claim 19** The computer storage medium according to claim 18, wherein the cohort machine learning model is trained to generate accessibility commands for a specific disability. **Claim 20** The computer storage medium according to claim 18, wherein the user-specific machine learning model is a machine learning model trained to generate accessibility commands for a specific user.

Images