Learning model by means of machine to achieve barrier-free functions during play of game

Abstract

Aspects of the present disclosure provide systems and methods that utilize machine learning techniques to provide enhanced barrier-free functionality to games. A barrier-free service is provided that can instantiate one or more machine learning models that can process a current game play state and generate commands to assist a user during game play. The barrier-free commands may be provided to the game and used to supplement or modify user-provided inputs in order to compensate for particular user requirements. In further aspects, a barrier-free user interface is provided that allows a user to dynamically enable or disable barrier-free functionality during play of a game. The user interface is operable to receive a barrier-free selection and provide selection data to the barrier-free service during play of the game.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. patent application No. 18 / 199,693, filed on May 19, 2023, entitled “Leveraging Machine Learning Models to Implement Accessibility Features during Gameplay,” which claims priority to U.S. Provisional Application No. 63 / 345,216, filed on May 24, 2022, entitled “Creation and Personalization of Virtual Gamers,” and the entire disclosure of the above application is incorporated by reference into this application. Background Art

[0002] The gaming industry constitutes a significant portion of the technology industry. As technology moves towards an increasingly connected future, gaming will be a central component of that connected future. Artificial intelligence has always been a major component of the gaming industry, however, as AI developments continue to advance, it becomes more difficult for the gaming industry to incorporate such advancements into gaming. Additionally, as the gaming market continues to grow, accessibility features become more important in order to accommodate users with different needs. However, specialized expertise is often required to identify and / or implement accessibility features, which standard game developers may not have.

[0003] It is based on these and other general considerations that various aspects of the present disclosure are proposed.In addition, although relatively specific problems are discussed, it should be understood that these examples should not be limited to solving specific problems identified in the background or elsewhere in this disclosure. Summary of the invention

[0004] Various aspects of the present disclosure provide systems and methods for providing enhanced accessibility to games using machine learning techniques. An accessibility service is provided that can instantiate one or more machine learning models that can process the current game play state and generate commands to assist users during game play. Accessibility commands can be provided to the game and used to supplement or modify the input provided by the user to compensate for specific user needs. In doing so, a robust system is provided for accessibility that can be customized to allow users with different game play needs (e.g., users with visual impairments, users with one hand, etc.) to enjoy the benefits of the game, even if the game does not natively support specific accessibility features.

[0005] In a further aspect, an accessibility user interface is provided that allows a user to dynamically enable or disable accessibility features during game play. The user interface is operable to receive accessibility selections and provide the selection data to an accessibility service during game play. The selected accessibility features can be used by the service to instantiate one or more machine learning models that are operable to generate accessibility commands to incorporate the selected accessibility features into the game during game play.

[0006] This summary is provided to introduce a selection of concepts in simplified form, which will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the following description, and will be partially apparent from the description, or may be learned through practice of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0008] Figure 1A An overview of an example system for generating and using a user-personalized agent in a gaming system is presented.

[0009] Figure 1B An overview of an example system for generating and utilizing personalized agent responses in a gaming system is presented.

[0010] Figure 2 An example of a method for generating a personalized agent is shown.

[0011] Figure 3 An exemplary system 300 for providing accessibility services 302 using one or more machine learning models is depicted.

[0012] Figure 4 An exemplary method for instantiating an accessibility machine learning model to assist a user during a game play session is shown.

[0013] Figure 5A An exemplary method for generating accessibility commands using one or more machine learning models is shown.

[0014] Figure 5B An exemplary method 520 for implementing accessibility functionality during game play is shown.

[0015] Figure 6 An exemplary method 600 for modifying accessibility functionality based on changes in a user's gameplay is depicted.

[0016] Figure 7 is a block diagram illustrating example physical components of a computing device that may be used to implement various aspects of the present disclosure.

[0017] Figure 8 is another simplified block diagram of a mobile computing device in which various aspects of the present disclosure may be implemented. DETAILED DESCRIPTION

[0018] Various aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which are from a portion of this article and illustrate specific example aspects. However, different aspects of the present disclosure can be implemented in many different ways and should not be construed as limited to the aspects set forth herein; on the contrary, these aspects are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of various aspects to those skilled in the art. Practical aspects can be as methods, systems or devices. Therefore, various aspects can take the form of hardware implementation, complete software implementation or combination of software and hardware aspects. Therefore, the following detailed description should not be understood as limiting.

[0019] Various aspects of the present disclosure provide systems and methods for utilizing machine learning techniques to provide enhanced accessibility to games. As described herein, an accessibility service is provided that can utilize one or more machine learning models to generate accessibility commands that can be executed by the game to provide accessibility to the user. The degree of accessibility can be customized on a per-user basis. In addition, when aided by a machine learning model, game play control commands can be generated to assist the user during game play, for example, to assist one-handed users, game state commands can be generated to modify game graphics, for example, to assist color blind or visually impaired users, and the like.

[0020] As mentioned, accessibility features can be generated through the use of machine learning models executed during game play. This allows accessibility services to provide accessibility features to a variety of different games through instructions generated by machine learning models, without requiring individual games to natively implement accessibility features. In doing so, the accessibility system provided herein can be used to greatly expand the use of accessibility features across the gaming market, making games more inclusive to all users regardless of the specific needs or abilities of individual users.

[0021] In an example, a generative multimodal machine learning model process may be employed to generate a multimodal output. For example, a conversational agent according to aspects described herein may receive user input such that the user input may be processed using a generative multimodal machine learning model to generate a multimodal output. The multimodal output may include natural language output and / or programmatic output, among other examples. The multimodal output may be processed and used to affect the state of an associated application. For example, at least a portion of the multimodal output may be executed or may be used to call an application programming interface (API) of an application. In some instances, the generative multimodal machine learning model (also generally referred to herein as a multimodal machine learning model) used according to aspects described herein may be a generative transformer model. In some instances, explicit and / or implicit feedback may be processed to improve the performance of the multimodal machine learning model.

[0022] Fig. 1 shows an overview of an example system 100 for generating and using a user personalized agent in a game system. As depicted in Fig. 1, user equipment 102 interacts with the cloud service 104 operating a game service 106 (or other types of applications) and the instantiation of an agent 108 that can interact with a game. The game device can be a host game system, a mobile device, a smart phone, a personal computer or any other type of device that can locally execute a game or can access a game operated on a server. In one example, the game associated with game service 106 can be directly operated by cloud service 104. In an alternative example, user equipment can locally operate and execute a game, in which case game service 106 can be used as an interface to promote communication between one or more instantiation agents 108 and the game. Personalized agent library 107 can store and execute components for one or more agents associated with the user of user equipment 102. The components of personalized agent library 107 can be used to control instantiation agent 108.

[0023] In the example, one or more agents from personalized agent library 107 interact with the game via (multiple) instantiated agents 108 based on text communication, voice commands and / or player actions received from user device 102. That is, system 100 supports interaction with agents, as if they are other human players, or alternatively, usually interact with conventional in-game NPCs according to players. In doing so, one or more agents hosted by agent library 108 are operable to interact with different games played by users, without requiring changes to the game. In other words, system 100 is operable to work with games, without requiring that the game be specifically developed to support agents (e.g., agents do not require API access, and games do not have to be developed to interact with agents using specific codes, etc.). In doing so, system 100 provides an extensible solution, which allows users to play games with customized agents across various games. In other words, using audio and video interaction and / or disclosed API, game state is communicated between one or more instantiated agents 108 and games via game services 106. In doing so, (multiple) instantiated agents 108 can interact with the game in the same way as the user (e.g., by interpreting the video, audio and / or tactile feedback from the game) and / or in the same way as the NPC in the game (e.g., via a public API). Similarly, one or more agents can use the game device 102 to interact with the user playing the game in a manner similar to the NPC interacting with the user. That is, one or more agents can receive visual, text or voice input from the user playing the game via the game service 106 and / or receive game information (e.g., current game state, NPC inventory, NPC capabilities, etc.) via a public API. In this way, the system 100 allows users to play games via user devices 102 to interact with one or more agents, in the manner in which these users would interact with any other player or NPC. However, based on past interactions with the user in both the current game or other games, (multiple) instantiated agents 108 can be personalized to the user. In order to promote this type of user interaction, one or more instantiated agents 108 can use computer vision, speech recognition or other known technologies when processing interactions with the user, so as to interpret the commands received from the user and / or generate response actions based on the user's actions.

[0024] For example, consider a user playing a first-person shooter game using a personalized agent. The user can say "cover the right side". A speech recognition model (which can be one of the models 118) can be utilized to interpret the audio received from the user as a modality understood by the agent. In response, the agent 108 can take a position on the right side of the user or otherwise perform an action to cover the right side of the user in response to the user's audio instructions. As another example, consider an action role-playing game in which two targets must be defended at the same time. The agent that plays the game in conjunction with the user can use computer vision to analyze the current view of the game to determine that the two targets exist (e.g., identify the two targets on the map, explain the displayed task log indicating that the two targets must be defended, etc.). Similarly, computer vision can be used to identify the user's player character and determine that the player character is heading for the first target. Based on feedback from the computer vision model, the agent can guide its representative character to move toward the second target and defend the second target.

[0025] From these examples, it will be understood by those skilled in the art that, by adopting speech recognition, computer vision technology, etc., one or more agents from the agent library will be able to interact with the user in a manner similar to how other users would interact during the cooperative game process. In doing so, one or more agents from the agent library can be generated separately from a specific game, because (multiple) instantiated agents 108 may not have to have API or programmatic access to the game in order to interact with the user. Alternatively, (multiple) instantiated agents 108 may have API access or programmatic access to the game, such as when the instantiated agent "dominates" the NPC in the game. In the case described, the agent can be based on interacting with the game state and the user via API access. Because the system 100 provides a solution in which the agent can be implemented separately from a single game, one or more agents in the agent library can be personalized to interact with a specific user. This personalization can be implemented across different games. That is, over time, the agent learns details about the user, such as the user's likes and dislikes, the user's game style, the user's communication mode, the user's preference strategy, etc., and can accommodate the user across different games accordingly.

[0026] Agent personalization can be generated and updated over time via feedback collection engine 110. Feedback collection engine 110 receives feedback from users and / or (multiple) instantiated agents 108 executed in the game. The collected feedback can include information associated with the following: the user's game style, user communication, user interaction with the game, user interaction with other players, user interaction with other agents, (multiple) instantiated agents 108 The results of the actions performed in the game, the interaction between the player and (multiple) instantiated agents 108 in the game, or any type of information generated by the user device 102 when the user plays the game. In order to comply with user privacy considerations, information can only be collected by the feedback collection engine 110 when receiving permission to do so from the user. The user can choose to join or exit the collection at any time. The collected data can be implicit data, for example, data based on the normal interaction of the user with the game, or explicit data, such as a specific command provided by the user to the system. An example of a specific command can be that the user instructs the agent to call the user by a specific character name. The data collected by the feedback collection engine 110 can be provided to the prompt generator 112.

[0027] The prompt generator 112 can use the data collected by the feedback collection engine 110 to generate prompts for personalizing one or more agents of the agent library 108. That is, the prompt generator 112 interprets the collected feedback data to generate instructions that can be executed by one or more agents to perform actions by the agents. The prompt generator is operable to generate new prompts or instructions based on the collected feedback, or to change existing prompts based on newly collected feedback. For example, if the user initially plays the game as a frontline attacker, the prompt generator 112 can generate instructions that enable (multiple) instantiated agents 108 to implement a supportive game style, such as a ranged attacker or a support character. If the user switches the game style to one of the ranged damage dealers, the prompt generator 112 can identify the change via the feedback data and adjust the (multiple) instantiated agents to adjust to the player's new style (e.g., switch to a frontline attacker to attract hatred or hostility from the player character). The instructions generated by the prompt generator are provided to the cloud service 104 to be stored by the cloud service 104 as part of the agent library 108, thereby storing meta-classifications (e.g., sentiment analysis, intent analysis, etc.) associated with a particular user. In doing so, instructions generated based on the user's gaming style or preferences in the first game can be incorporated by the agent not only into the first game, but also into other games played by the user. That is, using the instructions generated by the prompt generator 112, the cloud service 104 is able to instantiate an agent across a variety of different games, which has been personalized for a particular user based on the user's previous interactions with (multiple) instantiated agents 108, regardless of whether the user is playing the same game or a different game. Although the aspects described herein describe the separate prompt generator 112 as generating commands to control (multiple) instantiated agents 108, in alternative aspects, the commands can be directly generated by one or more machine learning models employed by the agent library 107, or directly generated via a combination of various different components disclosed herein.

[0028] However, in some scenarios, there are problems that may be encountered during the training process for various machine learning models that can be used to generate personalized agents. For example, there may be failed training sessions due to data or processing errors. In another example, the training process may fail due to user errors. For example, a user trying to get in touch with a new game may not play the game correctly at first. The user's incorrect actions or experiences may train the personalized agent to play the game in a way that negatively affects the progress of the game. In this way, the system 100 may include a process to roll back or reset one or more machine learning models (or any of the other components disclosed herein) to correct errors that may occur when training one or more machine learning models, or to correct any other errors that may usually occur when the personalized agent is developed. For example, the system 100 may periodically maintain snapshots of different machine learning models (or other components), which save the state of the components at the time of the snapshot. This allows the system 100 to respond to detecting training errors in the future, roll back all components, subsets of components, or specific components. Multiple different snapshots can be stored, representing the state of the personalized agent as the personalized agent develops over time, thereby providing the system 100 (or user) with options to determine the appropriate state for rollback when an error is encountered.

[0029] The personalized agent library 107 may also include a fine-tuning engine 114 and one or more models 116. The fine-tuning engine 114 and the model 116 are operable to interact with user actions via the user device 102, (multiple) instantiated agents 108, and (multiple) game training models 124 to process feedback data received from various sources. Any number of different models may be adopted individually or in combination for use with the examples disclosed herein. For example, a base model, a language model, a computer vision model, a speech model, a video model, and / or an audio model may be adopted by the system 100. As used herein, a base model is a model trained on a wide range of data that is adaptable to a wide range of tasks (e.g., a model capable of handling a variety of different tasks or modalities).

[0030] One or more models 116 can process video, audio and / or text data received from a user or generated by the game during game play to interpret user commands and / or derive user intent based on user communications or in-game actions. Output from one or more models 116 is provided to a fine-tuning model 114, which can use the output to modify the prompts generated by the prompt generator 112 to further personalize the instructions generated for the user based on the agent's past interactions with the user. The personalized agent library 107 can also include a game data (background story) 122 component that is operable to store information about a variety of different games. The game data (background story) 122 component stores information about the game, the game's background story, story elements, available abilities and / or items, etc., which can be used by other components (e.g., models 118, fine-tuning engine 114, prompt generator 112, etc.) to generate instructions to control (multiple) instantiated agents 108 according to the theme, story, requirements, etc. of the game.

[0031] The various components described herein can be utilized to interpret the current game state based on data received from the game (e.g., visual data, audio data, tactile feedback data, data exposed by the game through an API, etc.). In addition, the components disclosed herein are operable to generate instructions based on the current game state to control the actions of the personalized agent in the game. For example, the components disclosed herein can be operable to generate instructions to control the interaction of the personalized agent with the game in a manner similar to how a human would interact with the game (e.g., by providing specific controller instructions, keyboard instructions, or any other type of interface with a supported game for control). Alternatively or additionally, the various components can generate code that controls how the personalized agent interacts in the game. For example, the code can be executed by the personalized agent, which causes the personalized agent to perform specific actions within the game.

[0032] The personalized agent library 107 may also include an agent memory component 120. The agent memory component may be used to store personalized data generated by various other components described herein, as well as game styles, techniques, and interactions learned by the agent via past interactions with the user. The agent memory 120 may provide additional input to the prompt generator 112, which may be used to determine (multiple) instantiated agent actions during game play.

[0033] So far, the components of the described system 100 have focused on the creation of personalized agents, the continuous evolution of personalized agents via continuous interaction with users, and the instantiation of personalized agents in user's game sessions. Although personalization of agents for specific users provides many game play benefits, instantiated agents also require understanding how to interact with games and how to play games. In one aspect, components of the personalized agent library 107 are operable to learn and refine the game play of agents based on sessions with users. However, when users play new games, the time required for agents to learn game play mechanisms in order to become useful companions may not be feasible just by playing games with users. To address this problem, the system 100 also includes a game play training service 124, which includes a game library 126 and a game play machine learning model 128. In various aspects, the game library includes any number of games supported by the cloud service 104 and / or the user device 102. The game play training service 124 is operable to perform sessions for various games stored in the game library 126, and instantiate agents within the executed games. The game play machine learning model 128 is operable to receive data from an executed game and agent actions performed in the game as input, and in response, generate control signals to direct the game play of the agent within the game. Through the use of reinforcement learning, the game play machine learning model is operable to develop an understanding of the game play mechanics for both specific games and game genres. In doing so, the game play training service 124 provides a mechanism where an agent can be trained to play a specific game or a specific type of game without requiring user interaction. The personalized agent library 107 is operable to receive (or interact with) trained models from the game play training service 124, which can be stored as part of the agent memory, and adopt those models with other personalized components to control (multiple) instantiated agents 108. In doing so, the user experience of interacting with the agent in the game is greatly enhanced because the user is not required to invest time to train the agent in the specific mechanism of the game. Additionally, by importing or interacting with a trained game play machine learning model 128 provided by a game play training service 124, the personalized agent library 107 is able to employ the trained (multiple) instantiated agents 108 to play with the user when the user first launches a new game.

[0034] Those skilled in the art will appreciate that the system 100 provides a personalized agent or artificial intelligence that is operable to learn the identity of a player, learn a player's communication style or tendencies, learn strategies employed and used by players in a variety of different games and scenarios, learn game play mechanics for specific games and game genres, etc. In addition, one or more agents generated by the system 100 may be stored as part of a cloud service, which allows the system to retain a "memory" of a user's past interactions, thereby allowing the system to generate an agent that acts as a consistent user companion across different games without requiring the game to be specifically designed to support such an agent.

[0035] Figure 1B An overview of an example system 150 for generating and utilizing personalized agent responses in a gaming system is shown. As depicted in the system 150, two players, player 1 152 and player 2 154, are interacting with one or more agents 156. Although two players are shown, one skilled in the art will appreciate that any number of players may participate in a gaming session using the system 150. An assistant service 158 is provided that assists in personalizing the one or more agents 158 to interact with a single player or with multiple players simultaneously. Figure 1A As discussed in , one or more models 166 may be used to generate or modify prompts 168. The prompts 168 are provided to the assistant service 158, which applies multiple engines (e.g., agent role engine 160, user goal or intent engine 162, and game background story or constraint engine 164) to modify the prompts to provide more personalized interactions with a single or group of players.

[0036] For example, the agent role engine 160 can modify or adjust the prompt (or the action determined by the prompt) according to the personalized information associated with the agent. For example, the user can adopt an agent with a preferred personality. The agent role engine 160 can modify the prompt or the response generated by the prompt according to the agent's personality. The user intention or goal engine can modify the prompt (or the action determined by the prompt) based on the user's current goal or the intention behind the user's action or request. The user's goal or intention can change over time, can be based on a specified user goal, or can be determined based on the user's action. The game background story or constraint engine 164 can modify or adjust the prompt (or the action determined by the prompt) according to the characteristics of the game. For example, the agent can be a non-player controlled character in a "dominated" game (as discussed in further detail below). The game background story or constraint engine 164 can modify the prompt based on the NPC personality or restrictions. When modifying or adjusting the prompt, various engines of the assistant service can be adopted individually or in combination. The adjusted prompt is then provided to one or more associated agents 156 for execution.

[0037] Figure 2An example of a method 200 for generating a personalized agent is shown. For example, the method 200 can be adopted by the system 100. The process begins at operation 202, where a game session is instantiated, or an indication of a game session being instantiated is received. As described above, the game session can be hosted by a system that executes the method 200, or can be hosted by a user device, such as on a game console. When the game is instantiated or an indication that a game session is established is received, the process continues to operation 204, where the agent is instantiated as part of the game session. In one example, the agent can be instantiated in response to receiving a request to add an agent to the game session. For example, a request can be received to instantiate an agent in a multiplayer game or an agent that controls an NPC or AI companion in a stand-alone game. Instantiating the agent can include identifying an agent from an agent library associated with a user playing the game. As described above, aspects of the present disclosure provide for generating an agent that can be played across different games. Therefore, the agent instantiated at operation 204 can be instantiated using various components stored as part of a personalized agent library. For example, the agent can be instantiated using personalized features learned over time through interaction with players, saved game data or background stories about a specific game or genre, a machine learning model trained to perform mechanics and actions specific to the game in which the agent is to be instantiated, or a machine learning model trained based on a game genre similar to the game in which the agent is to be instantiated, etc. As previously discussed, the selected agent can be personalized to the user playing the game based on past interactions with the user in the same game or a different game as the game initiated at operation 202. Alternatively, instead of dynamically instantiating an agent using different components stored in an agent library, a specific agent can be selected at operation 204. That is, the agent library can contain specific "builds" for different types of agents, which are designed by the user or derived through specific game play with the user. These agents can be saved and instantiated by the user in future game sessions for the same game or a different game from the game in which they were originally created. When the agent is instantiated at operation 204, the agent joins the game session with the user.

[0038] At operation 206, the current game state is interpreted through audio and visual data and / or through API access granted by the game to the agent. As previously mentioned, certain aspects of the present disclosure provide the generation of agents that can interact with the game without requiring API or programmatic access to the game. In this way, the instantiated agent interacts with the game in the same way as the user, that is, through audio and visual data associated with the game. Alternatively, for example, when the agent is dominating the NPC, the agent can be granted API access to the game to interact with the game. At operation 206, various speech recognition, computer vision, object detection, OCR processes, etc. can be adopted to process communications (e.g., verbal commands, text-based commands) and game states received from players through the currently displayed view (e.g., using computer vision) or via API to interpret the game state. Then the current game state is used to generate agent actions at operation 208. For example, agent actions can be performed based on verbal commands received from users. Alternatively, agent commands can be generated based on the current view. For example, if an enemy appears on the screen, computer vision and / or object detection may be used to identify the enemy, and a command for the agent to attack the enemy may be generated at operation 208. Although not shown, operations 206 and 208 may be performed continuously while the game session is active.

[0039] The process continues to operation 210, where user feedback is received. The received user feedback can be explicit. For example, a user can issue a specific command to the agent to perform an action or change the action they are currently performing. Alternatively or additionally, the user feedback can be implicit. Implicit user feedback can be feedback data generated based on user interaction with the game. For example, the user may not explicitly provide commands to the agent, instead, the user can adjust their actions or game style based on the current game state and / or in response to the actions performed by the agent. In an example, user feedback can be collected continuously during the game session. The collected feedback can be associated with concurrent game states or agent actions.

[0040] When collecting user feedback, the process continues to operation 212, where prompts are generated for one or more agents based on the user feedback. In an example, the generated prompt is an instruction to perform an agent action in response to the state of the game or a specific user interaction. Using one or more machine learning models that receive user feedback and / or actions performed by one or more agents and / or existing prompts and / or state data, prompts can be generated. The output of the machine learning model can be used to generate one or more prompts. In an example, the machine learning model can be trained using information related to the user so that the output from the machine learning model is personalized for the user. Alternatively or additionally, the machine learning model can be trained for a specific game or application, for a specific group of users (e.g., an e-sports team), etc. Multiple machine learning models can be adopted at operation 212 to generate prompts. In other examples, other processes (such as rule-based processes) can be adopted as an addition or alternative to the use of the machine learning model at operation 212. In addition, new prompts can be generated at operation 212, or existing prompts can be modified.

[0041] Once one or more prompts are generated at operation 212, flow continues to operation 214, where the one or more prompts are stored for future use by one or more agents. For example, the one or more prompts may be stored in an agent library. By storing the prompts generated at 212 with the agent library, the agent will be able to utilize the prompts to interact with the user across different games, thereby providing a personalized agent that the user can play with across different games.

[0042] Although aspects of the disclosure described so far relate to importing personalized agents to play with users, game developers can use the system described herein to provide barrier-free systems to help players. That is, personalized agents can be used to implement barrier-free functions during game sessions. For example, aspects disclosed herein can be used to help people with disabilities play video games. For example, aspects of the disclosure can be used to help those with visual impairments, color blindness, limb loss that may affect game play, or to help novice gamers learn to play new games.

[0043] Figure 3An exemplary system 300 for providing an accessibility service 302 using one or more machine learning models is depicted. As depicted in system 300, accessibility service 302 may communicate with a game 304 executing on a user device or being controlled via a user device. In an example, game 304 may be executed locally on a user device (such as a game console, PC, smart phone, etc.) or executed over a cloud service network. When game 304 is executed on the cloud service, the user may still control the game play via a local device 301 (e.g., a controller, game console, PC, etc.). In some instances, user device 301, game 304, and accessibility service 302 may all be executed on a single device (e.g., a game console or PC). However, as Figure 3 As shown, user device 301, game 304, and accessibility service can be executed on different devices connected via network 303. Network 303 can be a local area network, a wide area network, the Internet, or any other type of network. In some aspects, accessibility service 302 can be hosted using a cloud network or a distributed network. For example, the accessibility service can utilize one or more machine learning models that may require computing resources that are not available to the user device.

[0044] Accessibility service 302 may include various data repositories, machine learning models, and / or executable code that allow the accessibility service to provide accessibility functionality to games. That is, accessibility service 302 may be a service that games 304 may leverage without requiring a single game to implement the functionality of the service. For example, accessibility service 302 may provide an accessibility SDK that games (such as game 304) may leverage to incorporate accessibility functionality without requiring game developers to implement the accessibility functionality into the game. Thus, accessibility service 302 is a service that may be leveraged by a variety of different games to provide accessibility functionality, thereby creating a gaming ecosystem that is more inclusive for users with disabilities.

[0045] In one aspect, the accessibility service 302 may include user profile data and / or data related to gaming history. For example, if the user provides the accessibility service 302 with permission to do so, the accessibility service 302 may collect and store user profile / gaming history information 306 related to the user's accessibility preferences (e.g., enabling color blind mode, enabling one-handed mode, etc.), gaming preferences (e.g., difficulty level, gaming style preferences, etc.), etc. In another aspect, the user profile / gaming history information 306 may track the user's gaming style, abilities, and performance associated with different games. The user profile / gaming history information 306 may be tracked across different games played by the user, thereby providing a baseline of information and preferences that the accessibility service 302 may leverage when implementing accessibility features for the user as the user plays different games and new games.

[0046] The accessibility service 302 may also include one or more (multiple) game models 308. One or more (multiple) game models 308 may be any type of machine learning model generated, trained, or fine-tuned for a specific game. That is, the accessibility service 302 may have one or more specific machine learning models specific to a single game or a single game genre (e.g., first-person shooter, role-playing game, puzzle game, massively multiplayer online role-playing game (MMORPG), etc.). One or more (multiple) game models may be any type of machine learning model (e.g., base model, language model, computer vision model, speech model, video model, audio model, multimodal machine learning model, etc.). In aspect, one or more game models 308 may be generated, trained, or fine-tuned to perform game play for a specific game (or game genre) based on data collected from a general game set, based on robot simulation, or a combination of both. In the example, (multiple) game models 308 represent standard game play for a specific game (or game style). Additionally, different game models 308 may be generated to reflect different game play abilities (e.g., novice game play, advanced game play, expert game play, etc.) Game model(s) 308 may be used by accessibility service 302 as a baseline game play generator to generate controls to play a particular game (or game genre) at a desired skill level.

[0047] Although the game model(s) 308 may be used by the accessibility service 302 to generate game play for a specific game, the game model(s) 308 are generic in nature, that is, they are not trained to assist users with specific impairments. In order to provide assistance to players with specific impairments (e.g., visual impairment, limb loss, etc.), the accessibility service includes one or more group models 310. When the game model 308 is generated, trained, or fine-tuned to generate game play commands based on a general user base, the group model 310 may be one or more machine learning models (e.g., base models, language models, computer vision models, speech models, video models, audio models, multimodal machine learning models, etc.), which are generated, trained, or fine-tuned to generate game play commands that can assist users with specific impairments (e.g., groups of users with impairments). For example, specific group models 308 may exist to generate game play commands specific to assisting users with specific impairments (e.g., color blindness, visual impairment, one-handed gaming, etc.). The group model(s) 310 can be generated or trained using a game play dataset from users with a specific impairment, and can therefore generate game play commands that are specifically tailored to assist users with the specific impairment. Although aspects have been described as including different models for specific impairments, those skilled in the art will appreciate that a single model can be generated, trained, or fine-tuned to generate game play commands to assist with multiple impairments.

[0048] The accessibility service 302 may also include (multiple) user-specific models 312. For example, one or more machine learning models (e.g., base models, language models, computer vision models, speech models, video models, audio models, multimodal machine learning models, etc.) specific to a single user's game play provide accessibility services provided by a single user to collect data about the single user's game play. These (multiple) user-specific models 312 can be generated using separate data from other game sessions (e.g., data from user profile / game history 306) or separate data from past game sessions of the currently played game (e.g., game 304). In addition, (multiple) user-specific models 312 can be continuously updated based on the game play of a single user, thereby reflecting the improvements made by a single user as they continue to play the game and their skills in a specific game (or game genre) increase. Similar to other machine learning models (e.g., game models, group models, etc.), (multiple) user-specific models 312 can receive game data (e.g., game state, user input during game play, etc.) and generate user-specific game play assistance commands.

[0049] The exchange of game play data and accessibility commands may be facilitated by the accessible SDK 314. As previously described, the accessibility SDK 314 exposes the accessibility service 302 to different games, such as the game 304, so as to allow the game to use one or more machine learning models to provide accessibility functionality without the need to implement the accessibility functionality separately within the game itself. Thus, developers may easily add accessibility functionality to games, thereby creating a more inclusive gaming environment. Via the accessibility SDK 314, game play data, user input, and auxiliary controls generated by the accessibility service may be transmitted and received during a game play session.

[0050] In an exemplary operation, various machine learning models of the accessibility service (e.g., game model(s) 308, group model(s) 310, and user-specific model(s) 314) may receive current game play data for a game 304 and generate assistive commands to help the impaired user when the impaired user plays the game. In one example, one or more of the various types of machine learning models may receive game play data, including, for example, the current game state, the current user display (e.g., the user's current view), commands entered by the user, etc., as inputs processed using the machine learning model(s) to generate commands that can assist the user in controlling the game. In one aspect, these models may make separate determinations that may be input to other models (e.g., together with the game play data described above) to generate commands specific to assisting the user based on the user's impairment. For example, the game model 308 may receive game play data to generate generic commands (e.g., commands based on the entire set of users). Generic commands may be provided to a group model with or without game play data, which may generate commands that are customized to assist a user who has an impairment similar to that of a user playing the game 304 via the user device 301. The customized commands are then provided as input (with or without game play data) to a user-specific model, which can then generate user-specific commands (e.g., customized based on user-specific needs or preferences). The user-specific commands can then be provided to the game, for example, via network 303, to be implemented together with the user input to assist the user in playing the game. Those skilled in the art will appreciate that the type of assistance may vary depending on the needs of a single user, the current game play, game play restrictions, and / or the current state of the game. In doing so, the accessibility service 302 is operable to provide any number of different auxiliary functions (e.g., aiming assistance, object detection, game play guidance, assisted steering or driving control, additional button input) based on the needs and preferences of the user. In addition, although the aspects described herein teach the use of multiple machine learning models, those skilled in the art will understand that a single machine learning model can be implemented to perform the functions of the different types of models described above. Therefore, those skilled in the art will understand that the differences between different models are provided to facilitate the illustration and description of the various functions provided by the accessibility service, and the present disclosure is not limited to the implementation using multiple different machine learning models.

[0051] Turning now to the game 304, the game 304 may include various components that can be used to interface with the accessibility service 320 and / or implement auxiliary controls received from the accessibility service 302. For example, the game 304 may include an accessibility request component 316, a game constraint analyzer 318, an accessibility service interface 320, and an input execution engine 322. In one aspect, these various components may be implemented as part of the game to give the game developer control over the extent to which accessibility commands provided by the accessibility service 302 can control the game play. For example, in some cases, such as competitive multiplayer games, the game developer may want to limit the amount of control that the accessibility service 302 can assert over the game play in order to ensure that a fair playing field exists. However, functions described as being performed by the game 304 in the system 300 may also be performed by the accessibility service 302 in other embodiments. That is, while particular functionality is described as being performed by a particular executor in the system 300, the functionality may be performed using a different executor without departing from the scope of the present disclosure.

[0052] The accessibility request component 316 can, for example, use a call to the accessibility SDK 314 to request a list of accessibility functions that can be provided to the game. For example, at startup, the game 304 can query the accessibility service 302 to receive a list of available accessibility functions. The function list can be incorporated into the game menu. The game 304 can display the accessibility functions in the game to the user. Via the menu user interface, the game 304 can receive a selection of accessibility functions to be adopted from the user (e.g., color blind mode, tutorial guidance, driving assistance, aiming assistance, etc.). The selected accessibility function can then be implemented during game play. For example, the game can send the selected accessibility function to the accessibility service 302, thereby instructing the accessibility service to generate controls to implement the selected function during game play. When the user has previously selected an accessibility function, the selected function can be stored and loaded on a subsequent game session, thereby instructing the accessibility service 302 to provide the function without requiring the user to reselect the desired function each time the user starts the game 304. In various aspects, the commands received from the accessibility service 302 can be of various different types. For example, the command may be an input command that helps the user control the game. The input command may, for example, relate to aiming assistance (e.g., causing the aiming to be adjusted during the game play), driving assistance, additional button input (e.g., for one-handed game play), etc. That is, the commands received from the accessibility service 302 may be input commands (e.g., as if they were commands generated using an input device such as a controller, mouse, or keyboard connected to the user device 301). Alternatively or additionally, the command may be a command to adjust the game environment or provide additional information to the user. Examples of such commands may be commands that cause an object to be highlighted or visually marked (e.g., to help visually impaired users), commands that display instructions or tips, etc. The commands received via the accessibility service interface 320 may be implemented by the game 304.

[0053] However, as described above, game developers can limit the type of command, the degree of command, or the type of command that can be implemented by the barrier-free service 302. Therefore, a game constraint analyzer 318 can be provided, which can evaluate the command received from the barrier-free service and determine whether to implement the command, modify the command, or prevent the command from being executed. As described above, there may be reasons why game developers expect to block assistance. For example, during competitive multiplayer games, game developers may wish to limit or prevent aiming assistance to ensure fair game play between all players. Alternatively or additionally, game developers may wish to prevent certain guidance or tips because providing such tips may destroy the game for users (e.g., by providing solutions to puzzles, providing information that may destroy upcoming game events, etc.). In this way, the game constraint analyzer 318 can analyze the received commands against the restrictions set by the game developer or based on the restrictions set by the user to ensure that the barrier-free service 302 will not negatively affect the game. As described, in some instances, the restricted commands can be blocked by the game constraint analyzer 318 or can be modified to meet the game constraints. For example, if the accessibility command is a control command, the game constraint analyzer 318 can modify the control command to adjust the degree of control (e.g., by adjusting the degree of aiming assistance or steering assistance). If the command is to highlight an item in the game, the game constraint analyzer 318 can block the command, or reduce the degree to which the item is highlighted, so that there is still some degree of assistance, but it is not so obvious that the user playing the game will not make any effort to see it. The modification can take the form of actually modifying the input or modifying the graphical commands generated by the accessibility service 302.

[0054] Aspects of the present disclosure are directed to assisting users in playing games. That is, when the assistance overwrites the player's own actions, such assistance is unwelcome, thereby removing the player agent. That is, the barrier-free service 302 is not intended to take over the user's game play, but to adjust the user's game method to assist users who may have obstacles (or may expect additional help). That is, the commands received by the game from the barrier-free service 302 are intended to be used together with the commands received from the player (that is, the commands received from the user device 301), rather than replacing the commands received from the player. In this way, the input execution engine 322 is provided to implement the game play action based on the commands received from both the user device 301 and the barrier-free service 302. For example, if the user is turning the vehicle to the left, and the barrier-free command received from the barrier-free service is to turn the vehicle to the right, the input execution engine can essentially mix the commands instead of overwriting the user commands, such as by reducing the degree to which the user command is turning the vehicle to the left instead of completely overwriting the write command. In addition, the input execution engine can continuously monitor the user commands and compare those commands with the commands generated by the barrier-free service 302. If the user commands continue to contradict the commands generated by the accessibility service 302, the input execution engine 322 may eventually completely ignore the commands generated by the accessibility service 302, thereby maintaining the player's agent's control of the game. In such a case, the input execution engine may also provide the user with a notification that a certain accessibility feature is enabled that contradicts the user's input, and an option to disable the particular accessibility feature via the user interface. In doing so, the system 300 provides a complementary accessibility service that can be used to assist in game play without overwriting user actions, thereby maintaining player agency during a game session.

[0055] Figure 4 An exemplary method 400 for instantiating an accessibility machine learning model to assist a user during a game play session is shown. The process begins at operation 402, where a request is received from a user or a game to provide accessibility functionality. For example, via a user interface of a game (such as game 304 ( Figure 3 )) can select a set of accessibility features to enable for the game. Upon receiving the selected accessibility features, the game can send a request to an accessibility service (such as accessibility service 302 ( Figure 3)) transmits a request to provide the selected accessibility functionality. The request is received at operation 402. In one aspect, the request can be received along with a unique identifier for the user (such as a gamer tag of the user or another unique identifier for the user). Alternatively or additionally, the request can also include an identifier for the game for which the accessibility functionality is requested. In addition, the selected accessibility functionality can be identified using a specific functionality identifier (e.g., an identifier corresponding to aiming assist, an identifier corresponding to color blindness mode, etc.). Alternatively, the selected accessibility functionality can be identified using natural language (e.g., "enable aiming assist", "enable one-handed control", etc.).

[0056] The process continues to operation 404, where user profile data is received or retrieved. For example, relevant user profile data stored by the accessibility service can be retrieved for the user based on a user identifier (e.g., a gamer tag) associated with the request received at operation 402. The user profile data may include data related to the user's past gaming history, data related to a specific game, data related to a gaming genre, and the like. The profile data may be stored locally by the accessibility service, or may be requested from the user device if necessary. The user profile data may be used to select a specific machine learning model to generate accessibility functionality, and / or to generate prompts for one or more selected machine learning models to establish context and / or to help tune one or more selected models for the user's game session. Similarly, at operation 406, the user's gaming history associated with the game requesting accessibility functionality may be received or retrieved. The user's gaming history may similarly be used to select a specific machine learning model to generate accessibility functionality controls, or to prompt one or more selected machine learning models to generate accessibility commands for a specific level. For example, if the user's gameplay history shows a trend in which the user has improved their aiming over gameplay sessions, the gameplay history received or retrieved at operation 406 may be used to select a machine learning model that may provide a lesser degree of aiming assistance, or may prompt the selected machine learning model to generate less intrusive aiming accessibility commands. In this way, the profile data and gameplay history data received in operations 404 and 406 allow the accessibility service to automatically adjust the degree of assistance (provide greater or lesser assistance) based on the user's gameplay history.

[0057] Flow continues to operation 408, where, based on the received request and / or the received user profile data and game history data, one or more machine learning models are identified to provide accessibility functionality. For example, one or more game models, one or more population models, and one or more user-specific models may be identified by the accessibility service to provide the requested accessibility functionality. Flow then continues to operation 410, where the selected models are instantiated to provide accessibility functionality during the game play session, as will be described in Figure 5A Further described in .

[0058] The operations of method 400 may be performed using a single device or multiple devices. For example, the operations may be performed using a single device (such as a PC or a game console) to generate accessibility functionality. Alternatively, a user device may interact with a cloud service to perform the operations of method 400. Those skilled in the art will appreciate that the operations may be performed using a single device or multiple devices, and different operations may be performed on different devices without departing from the scope of the present invention.

[0059] Figure 5A An exemplary method 500 for generating accessibility commands using one or more machine learning models is shown. The process begins at operation 502, where one or more machine learning models are instantiated to provide accessibility functionality. For example, the process detailed with respect to method 400 can be used to generate one or more models. Alternatively or additionally, one or more machine learning models can also be selected using specific game constraints. The process then continues to operation 504, where current game play data is received. In one example, the game play data can be generated via an API (such as via Figure 3 The Accessibility SDK described in Receive Game Play Data via integration with the game (API disclosed by the Accessibility SDK described in ). Game data may include the current game state, current input received by one or more users, environmental information, NPC or other player actions, or any other type of data related to the current game play. Alternatively or additionally, computer vision can be used to receive the current game play data. For example, the current view of the game from the user can be provided to the computer vision tool to evaluate the state of the game play based on a separate display. In doing so, the game play data can be analyzed based on the view displayed by the game, thereby generating the game play data without the need for API access to the underlying game data. Therefore, various aspects of the present disclosure can be provided without accessing the game code or internal game data.

[0060] The process continues to operation 506, where one or more machine learning models are used to analyze the current game play data to determine game play actions and / or game state modifications based on the selected accessibility function. For example, as previously described, various machine learning models of the accessibility service (e.g., (one or more) game models, (one or more) queue models, and (one or more) user-specific models) can receive the current game play data during game play and generate auxiliary commands to help users with disabilities play the game while they play. In one example, one or more of the various types of machine learning models can receive game play data, including, for example, the current game state, the current user display (e.g., the user's current view), commands entered by the user, etc. as inputs processed using machine learning models to generate commands that can assist the user in controlling the game. In one aspect, these models can make separate determinations that can be input to other models (e.g., together with the above-mentioned game play data) to generate commands specific to assisting the user based on the user's disability. For example, a game model can receive game play data to generate general commands (e.g., commands based on the entire set of users). Generic commands can be provided to a group model with or without game play data, which can generate commands customized to help users with similar impairments when playing games via user devices. The customized commands can then be provided as input to a user-specific model, which can then generate user-specific commands (e.g., customized based on user-specific needs or preferences). The user-specific commands can then be provided to the game along with the user input to help the user play the game. Alternatively, in some aspects, a single machine learning model can be instantiated to provide auxiliary features.

[0061] Accessibility functions represented by accessibility commands in the form of game play commands, game state commands, or a combination of both are generated using one or more machine learning models and then provided to the game for execution at operation 508. For example, commands generated by one or more machine learning models are provided to the game for execution together with input received by the user. The process continues to operation 510, where, optionally, user input performed in response to the accessibility function is tracked. As noted, aspects of the present disclosure relate to providing a supplementary system to assist users during game play in a manner that does not remove the user agent. If the user provides input to contradict the accessibility command, such as moving a vehicle or adjusting the target for the command provided by one or more machine learning models, the auxiliary model can be adjusted to generate a command that does not contradict the user. In this way, the response to the command can be tracked at operation 510 and stored as potential feedback data. The tracked user input and the commands and / or game play data generated by one or more models can be saved at operation 510, and the tracked user input can be used to update one or more models using a fine-tuning or training process at operation 512. In doing so, feedback or reinforcement learning may be used to continuously update one or more models to adjust to the user's play style, preferences, and improvements in the game as the user continues to play the game. In doing so, method 500 provides a mechanism for continuously updating a machine learning model that is used to provide player assistance to optimally complement the user's game play.

[0062] The operations of method 500 may be performed using a single device or multiple devices. For example, a single device (such as a PC or a game console) may be used to perform the operations to generate accessibility functionality. Alternatively, a user device may interact with a cloud service to perform the operations of method 500. Those skilled in the art will appreciate that operations may be performed using a single or multiple devices, with different operations being performed on different devices, without departing from the scope of the present invention.

[0063] Figure 5B An exemplary method 520 for implementing accessibility features during game play is shown. The process begins at operation 522, where a user interface indicating accessibility options available for the game is displayed. In one example, the available accessibility options may be based on the accessibility options available to the game (e.g., Figure 3 The accessibility function provided by the accessibility service 302 of the game can be determined. That is, the accessibility function does not need to be provided exclusively by the game. In operation 524, one or more accessibility options can be selected from the accessibility user interface. The selected options are then sent to the accessibility service in operation 506. The selected accessibility options can be used to generate and instantiate one or more machine learning models to provide accessibility functions through the accessibility service, as described herein.

[0064] Those skilled in the art will appreciate that operations 522 to 526 may be performed dynamically. That is, the accessibility user interface may be invoked at any time during game play, at which time different accessibility features may be selected. Furthermore, the selected accessibility features may be saved so that the user is not required to select an accessibility feature each time the user plays the game. Instead, previously selected accessibility features may be loaded during subsequent game sessions.

[0065] The process continues to operation 520, where accessibility commands are received from an accessibility service. Accessibility commands can take the form of game play commands (e.g., commands for controlling player actions during a game), game state commands (e.g., commands for changing the state of a game, such as game difficulty, the visual state of a game, such as highlighting items or adjusting for color blindness, or providing game tips or tricks), or a combination of the two. The process continues to operation 530, where accessibility commands are optionally modified based on game constraints. For example, as previously described, the received accessibility commands can be analyzed for restrictions set by a game developer or based on restrictions set in place by a user to ensure that the accessibility service does not negatively impact the game. As noted, in some cases, restricted accessibility commands can be blocked or can be modified to comply with game constraints. For example, if the accessibility command is a game play control command, the accessibility command is modified to adjust the degree of control (e.g., by adjusting the degree of aiming assistance or steering assistance). If the command is to highlight an item in the game, the accessibility command may be blocked, or modified to reduce the degree to which the item is highlighted so that there is still some level of assistance, but it is less noticeable because the user playing the game will not be making any effort. The modification may take the form of actually modifying the input or graphical commands generated by the accessibility service.

[0066] At operation 508, the modified or unmodified accessibility commands are executed by the game. In some cases, the execution of accessibility commands may be performed regardless of user interaction. For example, accessibility commands that change the game display (e.g., highlight objects, provide prompts) may be executed regardless of the corresponding user input. However, in contrast to completely overwriting the received user input, the game command control may be executed in conjunction with the corresponding user input in a manner that adjusts or supplements the received user input. In doing so, the user maintains the mechanism for their game play.

[0067] The process of receiving, modifying, and executing commands can continue during the game play session. However, in some cases, an indication of a change to an auxiliary command can be automatically received. For example, if the user continues to make adjustments to the game play feature, for example, to correct the movement performed by the aiming assist, or if it is determined that some accessibility features are no longer needed, for example, based on the improvement of the user's game play, an indication of a change in the accessibility feature can be received at operation 534. The change can cause the game to display a user interface element at operation 536. The displayed user interface element can provide information about the change in the auxiliary feature and an activatable element that will allow the user to modify or change the accessibility feature, such as, for example, by removing the aiming assist or adjusting the aiming assist to become less invasive. Upon receiving a selection of the user interface element, the process continues to operation 538, where the accessibility feature is updated and the corresponding changes are saved and sent to the accessibility service.

[0068] Figure 6 An exemplary method 600 for modifying accessibility functionality based on changes in user game play is depicted. The process begins at operation 602, where game play actions (e.g., accessibility commands) determined by one or more machine learning models and actions taken in response to accessibility commands are tracked during game play. At operation 604, the user actions are compared to the game actions (e.g., accessibility commands) to determine the difference between the accessibility commands and the user input. For example, the accessibility commands can be compared to the user input to determine the degree of difference between the actions generated by the machine learning model and the user input. If the degree of difference is below a threshold, it can be determined that the user no longer requires a particular type of assistance. Alternatively or additionally, if the degree of difference between the actions generated by the machine learning model and the user input is relatively high, or if subsequent input received by the user indicates that the user is compensating for the actions generated by the model, it can be determined that the user disagrees with the suggested action. In an example, a determination can be made to modify the level of assistance provided or to completely disable the accessibility functionality.

[0069] At operation 606, one or more suggestions for modifying accessibility options may be generated based on the determined differences. The suggested adjustments may be provided along with interactive user interface elements (e.g., Figure 5B ), which will allow the user to select modifications. At operation 608, a request to modify the accessibility function according to the suggested action can be received via the user interface element. In some cases, the updated selection can be sent from the game to the accessibility service. At operation 610, future accessibility commands can be generated based on the updated accessibility function.

[0070] Figure 7is a block diagram illustrating the physical components (e.g., hardware) of a computing device 700 with which aspects of the present disclosure may be practiced. The computing device components described below may be applicable to the computing devices described above. In a basic configuration, the computing device 700 may include at least one processing unit 702 and a system memory 704. Depending on the configuration and type of the computing device, the system memory 704 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 704 may include an operating system 705 and one or more program tools 706 suitable for performing various aspects disclosed herein. The operating system 705 may, for example, be suitable for controlling the operation of the computing device 700. In addition, aspects of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application, and is not limited to any particular application or system. The basic configuration in Figure 7 708. The computing device 700 may have additional features or functionality. For example, the computing device 700 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tapes. Such additional storage Figure 7 , which is illustrated by a removable storage device 709 and a non-removable storage device 710.

[0071] As described above, a plurality of program tools and data files may be stored in system memory 704. When executed on at least one processing unit 702, program tools 706 (e.g., application programs 720) may perform processes including, but not limited to, various aspects as described herein. Application programs 720 include accessibility models 730, accessibility user interfaces 732, accessibility instructions 734 disclosed herein, and instructions for performing various processes disclosed herein. Other program tools that may be used according to aspects of the present disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided applications, and the like.

[0072] In addition, various aspects of the present disclosure may be implemented in circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing a microprocessor, or circuits on a single chip containing electronic components or a microprocessor. For example, various aspects of the present disclosure may be practiced via a system on a chip (SOC), wherein Figure 7Each or more of the components explained in the can be integrated into a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned") into the chip substrate as a single integrated circuit. When operated via the SOC, the functions described herein regarding the capabilities of the client-to-switch protocol can be operated via dedicated logic integrated with other components of the computing device 700 on a single integrated circuit (chip). Various aspects of the present disclosure may also be practiced using other technologies capable of performing logical operations, such as AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, various aspects of the present disclosure may be practiced within a general-purpose computer or any other circuit or system.

[0073] The computing device 700 may also have one or more input devices 712, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. Output devices 714 may also be included, such as a display, a speaker, a printer, etc. The aforementioned devices are examples, and other devices may be used. The computing device 700 may include one or more communication connections 716 that allow communication with other computing devices 750. Examples of communication connections 716 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuits; universal serial bus (USB), parallel and / or serial ports.

[0074] The term computer-readable medium as used herein may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures or program tools). System memory 704, removable storage device 709 and non-removable storage device 710 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassette, magnetic tape, disk storage or other magnetic storage device, or any other product that can be used to store information and can be accessed by computing device 700. Any such computer storage medium may be part of computing device 700. Computer storage media does not include carrier waves or other propagation or modulated data signals.

[0075] Communication media may be embodied by computer readable instructions, data structures, program tools, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode the information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0076] Figure 8 A computing device or mobile computing device 800 is shown in which aspects of the present disclosure may be practiced, such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc. In some aspects, a client used by a user (e.g., the client device 102 as shown in the system 100 in FIG. 1 ) may be a mobile computing device. Figure 8 802 is a block diagram illustrating an architecture of one aspect of a computing device, server, mobile computing device, etc. That is, a mobile computing device 800 can be incorporated into a system 802 (e.g., a system architecture) to implement some aspects. The system 802 can be implemented as a "smart phone" capable of running one or more applications (e.g., a browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 802 is integrated into a computing device, such as an integrated digital assistant (PDA) and a wireless phone.

[0077] One or more application programs 866 may be loaded into the memory 862 and run on or in association with the operating system 864. Examples of application programs include a phone dialer program, an email program, a information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 802 also includes a non-volatile storage area 868 within the memory 862. The non-volatile storage area 868 may be used to store persistent information that should not be lost if the system 802 loses power. The application programs 866 may use and store information in the non-volatile storage area 868, such as emails or other messages used by the email application, and the like. A synchronization application (not shown) also resides on the system 802 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep information stored in the non-volatile storage area 868 synchronized with corresponding information stored at the host computer. It should be understood that other applications may be loaded into the memory 862 and run on the mobile computing device 800 described herein.

[0078] System 802 has a power source 870, which can be implemented as one or more batteries. Power source 870 can also include an external power source, such as an AC adapter or a powered docking cradle that replenishes or recharges the batteries.

[0079] The system 802 may also include a radio interface layer 872 that performs the functions of sending and receiving radio frequency communications. The radio interface layer 872 facilitates wireless connectivity between the system 802 and the "outside world" via a communications carrier or service provider. Transmissions to and from the radio interface layer 872 are performed under the control of the operating system 864. In other words, communications received by the radio interface layer 872 may be propagated to the application 866 via the operating system 864, and vice versa.

[0080] Visual indicator 820 (e.g., LED) can be used to provide visual notification, and / or audio interface 874 can be used to generate audible notification via audio transducer 825. In the configuration shown, visual indicator 820 is a light emitting diode (LED), and audio transducer 825 is a speaker. These devices can be directly coupled to power supply 870 so that when activated, they remain turned on for the duration indicated by the notification mechanism, even if processor 860 and other components may be turned off to save battery power. LED can be programmed to remain on indefinitely until the user takes action to indicate the power-on state of the device. Audio interface 874 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to audio transducer 825, audio interface 874 can also be coupled to a microphone to receive audible input, such as promoting a telephone conversation. According to aspects of the present disclosure, the microphone can also be used as an audio sensor to facilitate the control of notifications, as described below. System 802 can also include a video interface 876, which enables the operation of the device connected to peripheral device port 830 to record still images, video streams, etc.

[0081] The mobile computing device 800 implementing the system 802 may have additional features or functionality. For example, the mobile computing device 800 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 8 This is shown in B by the non-volatile storage area 868.

[0082] Data / information generated or captured by the mobile computing device 800 and stored via the system 802 may be stored locally on the mobile computing device 800, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the radio interface layer 872, or accessed via a wired connection between the mobile computing device 800 and a separate computing device associated with the mobile computing device 800 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information may be accessed via the mobile computing device 800 via the radio interface layer 872 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use according to well-known data / information transfer and storage means, including electronic mail and collaborative data / information sharing systems.

[0083] It will be understood from the foregoing disclosure that one aspect of the present technology relates to a computer-implemented method for providing accessibility functionality for a game with the aid of an accessibility service, the method comprising: receiving a selection of an accessibility functionality for the game; based on the accessibility functionality, instantiating one or more machine learning models, wherein the one or more machine learning models are operable to generate commands to implement the accessibility functionality; receiving current game play data; generating accessibility commands using one or more machine learning models, wherein the current game play data is provided as input to the one or more machine learning models; and providing accessibility commands to the game, wherein providing the accessibility commands causes implementation of the accessibility functionality during game play.

[0084] In an example, the game play data includes one or more of: current user input; game state information; information about other player characters; or non-player character information.

[0085] In another example, receiving current game play data includes receiving a current view of the game, where the current view of the game is the view depicted to the player during game play.

[0086] In yet another example, the current game play data further includes processing a current view of the game using computer vision to generate the game play data.

[0087] In yet another example, instantiating one or more machine learning models includes instantiating at least one of: a game machine learning model; a population machine learning model; or a user-specific machine learning model.

[0088] In another example, a swarm machine learning model is trained to generate accessibility commands for specific obstacles.

[0089] In yet another example, the computer-implemented method further includes, in response to providing the accessibility command, receiving a user input, the user input being responsive to an adjustment made by the accessibility command.

[0090] In another example, the computer-implemented method further includes updating one or more machine learning models based on user input responsive to adjustments made by the accessibility commands.

[0091] In another aspect, the present technology relates to a computer-implemented method for providing an accessible user interface for a game based on accessibility functions provided by an accessibility service, the method comprising: generating a user interface depicting multiple accessibility functions, wherein the multiple accessibility functions include accessibility functions provided by an accessibility service, and wherein the accessibility service is a service separate from the game; receiving a selection of a first accessibility function provided by the accessibility service; sending the first accessibility function to the accessibility service; and in response to sending the accessibility function to the accessibility service, receiving multiple accessibility commands from the accessibility service during game play; and executing the multiple accessibility commands to implement the first accessibility function.

[0092] In an example, sending the first accessibility function to the accessibility service further includes sending a unique identifier for the player along with the first accessibility function.

[0093] In another example, the first accessibility command includes a game play control command, and wherein executing the game play control command includes generating a game play action based on the game play control command and the user input.

[0094] In yet another example, generating the game play action further includes modifying the user input based on the game play control command.

[0095] In another example, generating the game play action further includes supplementing the user input with a game play control command.

[0096] In another example, executing multiple accessibility commands also includes: comparing a first accessibility command to a game constraint; based on the comparison of the first accessibility command, determining a modification to the first accessibility command; executing the modified first accessibility command; comparing a second accessibility command to the game constraint; and based on the comparison of the second accessibility command, executing the second accessibility command without modification.

[0097] In yet another example, the computer-implemented method further includes receiving a suggestion to change an accessibility feature; generating a user interface element based on the suggestion; displaying the user interface during game play to change the accessibility feature; receiving a selection of the user interface element, and in response to receiving the selection, sending a second accessibility feature to an accessibility service.

[0098] On the other hand, the present technology relates to a computer storage medium encoding computer executable instructions, which when executed by at least one processor causes the at least one processor to perform a method, the method comprising: receiving a selection of an accessibility feature for a game; based on the accessibility feature, instantiating one or more machine learning models, wherein the one or more machine learning models are operable to generate commands to implement the accessibility feature; receiving current game play data; generating accessibility commands using one or more machine learning models, wherein the current game play data is provided as input to the one or more machine learning models; and providing accessibility commands to the game, wherein providing the accessibility commands causes implementation of the accessibility feature during game play.

[0099] In an example, instantiating one or more machine learning models includes instantiating at least one of: a game machine learning model; a population machine learning model; or a user-specific machine learning model.

[0100] In yet another example, a swarm machine learning model is trained to generate accessibility commands for specific obstacles.

[0101] In yet another example, a user-specific machine learning model is a machine learning model trained to generate accessibility commands for a specific user.

[0102] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the present disclosure claimed in any way. The disclosure claimed for protection should not be interpreted as being limited to any aspect or detail provided in the application, for example. Regardless of being shown and described in combination or separately, various features (agent structures and methods) are intended to be selectively included or omitted to produce aspects with a specific feature set. Having provided the description and illustration of the present application, those skilled in the art can envision the various aspects of changes, modifications and substitutions that fall within the spirit of the broader aspects of the general inventive concept embodied in the present application, without departing from the broader scope of the disclosure claimed for protection.

Claims

1. A computer-implemented method for providing accessibility functionality for a game using an accessibility service, the method comprising: Receiving (402) a selection of an accessibility feature for the game; Based on the accessibility functionality, instantiating (410) one or more machine learning models, wherein the one or more machine learning models are operable to generate commands to implement the accessibility functionality; Receiving (504) current game play data; generating (506) accessibility commands using the one or more machine learning models, wherein the current game play data is provided as an input to the one or more machine learning models; as well as The accessibility commands are provided (508) to the game, wherein providing the accessibility commands causes implementation of the accessibility functionality during game play.

2. The computer-implemented method of claim 1, wherein the game play data comprises one or more of: Current user input; Game status information; Information about other player characters; or Non-player character information.

3. The computer-implemented method of claim 1 , wherein receiving current game play data comprises receiving a current view of the game, wherein the current view of the game is the view depicted to a player during game play. 4 . The computer-implemented method of claim 1 , wherein receiving current game play data further comprises processing the current view of the game using computer vision to generate the game play data.

5. The computer-implemented method of claim 1 , wherein instantiating one or more machine learning models comprises instantiating at least one of: Game machine learning models; Swarm machine learning models; or User-specific machine learning models.

6. The computer-implemented method of claim 5, wherein the swarm machine learning model is trained to generate the accessibility commands for specific obstacles.

7. A computer-implemented method according to claim 5, wherein the user-specific machine learning model is a machine learning model trained to generate accessibility commands for a specific user.

8. The computer-implemented method of claim 1, further comprising, in response to providing the accessibility command, receiving user input responsive to an adjustment made by the accessibility command.

9. The computer-implemented method of claim 8, further comprising updating the one or more machine learning models based on the user input in response to the adjustments made by the accessibility commands.

10. A computer-implemented method for providing an accessible user interface for a game based on accessibility functions provided by an accessibility service, the method comprising: generating (522) a user interface depicting a plurality of accessibility functions, wherein the plurality of accessibility functions includes an accessibility function provided by the accessibility service, and wherein the accessibility service is a service separate from the game; receiving (524) a selection of a first accessibility function provided by the accessibility service; sending (526) the first accessibility function to the accessibility service; as well as In response to sending the accessibility function to the accessibility service, receiving (528) a plurality of accessibility commands from the accessibility service during game play; as well as The plurality of accessibility commands are executed (532) to implement the first accessibility function.

11. The computer-implemented method of claim 10, wherein sending the first accessibility function to the accessibility service further comprises: A unique identifier for the player is sent along with the first accessibility function.

12. The computer-implemented method of claim 10, wherein the first accessibility command comprises a game play control command, and wherein executing the game play control command comprises generating a game play action based on the game play control command and a user input.

13. The computer-implemented method of claim 12, wherein generating the game play action further comprises modifying the user input based on the game play control command.

14. The computer-implemented method of claim 12, wherein generating the game play action further comprises supplementing the user input with the game play control command.

15. The computer-implemented method of claim 10, wherein executing a plurality of accessibility commands further comprises: Compare the first accessibility command with the game constraints; determining, based on the comparing of the first accessibility commands, a modification to the first accessibility command; executing the modified first accessibility command; comparing a second accessibility command to the game constraint; as well as Based on the comparing of the second accessibility command, the second accessibility command is executed without modification.

16. The computer-implemented method of claim 10, further comprising: Receive suggestions for accessibility changes; generating a user interface element based on the suggestion; displaying the user interface during game play to change the accessibility features; receiving a selection of the user interface element, and In response to receiving the selection, a second accessibility function is sent to the accessibility service.

17. A computer storage medium encoding computer executable instructions which, when executed by at least one processor, cause the at least one processor to perform a method comprising: Receiving (402) a selection of accessibility features for the game; Based on the accessibility functionality, instantiating (410) one or more machine learning models, wherein the one or more machine learning models are operable to generate commands to implement the accessibility functionality; Receiving (504) current game play data; generating (506) accessibility commands using the one or more machine learning models, wherein the current game play data is provided as an input to the one or more machine learning models; as well as The accessibility commands are provided (508) to the game, wherein providing the accessibility commands causes implementation of the accessibility functionality during game play.

18. The computer storage medium of claim 17, wherein instantiating one or more machine learning models comprises instantiating at least one of: Game machine learning models; Swarm machine learning models; or User-specific machine learning models.

19. The computer storage medium of claim 18, wherein the swarm machine learning model is trained to generate the accessibility commands for specific disabilities.

20. The computer storage medium of claim 18, wherein the user-specific machine learning model is a machine learning model trained to generate the accessibility commands for a specific user.

Images