Systems and methods for artificial intelligence (AI)-assisted communication within video game

An AI-assisted communication system in video games addresses latency and relevance issues by using AI models to process and deliver messages aligned with the player's game state and context, enhancing engagement and reducing confusion.

JP2024161342AActive Publication Date: 2024-11-19SONY INTERACTIVE ENTERTAINMENT LLC
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
JP2024074024
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-01
Filing Date
2024-04-30
Publication Date
2024-11-19
Estimated Expiration
2044-04-30

AI Technical Summary

Technical Problem

In real-time video games, there are challenges in timely and relevant communication of messages, particularly gesture-based messages, due to latency and changing game states, leading to potential confusion and adverse impacts on player experience.

Method used

An AI-assisted communication system that includes an input processor, message identification engine, message interpretation engine, message relevance evaluation engine, and communication delivery engine, utilizing AI models to identify, interpret, and deliver messages relevant to the player's current game state and context.

Benefits of technology

The system accelerates message processing and delivery, ensuring timely and relevant communications, reducing latency and improving player engagement by aligning messages with the game's dynamic context.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024161342000001_ABST
    Figure 2024161342000001_ABST
Patent Text Reader

Abstract

To convey a message in an appropriate manner.SOLUTION: A portion of the input data receiving input data including a message for a target player and defining the message is automatically identified using a first artificial intelligence model component. A meaning of the message defined by the portion of the input data is automatically determined using a second artificial intelligence model component. A third artificial intelligence model component is used to automatically determine, based on the meaning of the message, whether or not the message is relevant to a current game state and game context of the target player. A communication to the target player that conveys the meaning of the message is automatically composed through use of a fourth artificial intelligence model component, when the message is determined to be relevant to the current game state and game context of the target player. The communication is delivered to the target player.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Background technology]

[0001] The video game industry has undergone many changes over the years, seeking to find ways to improve a player's video game playing experience and increase player engagement with video games and / or online gaming systems. Increasing a player's engagement with a video game makes the player more likely to continue playing the video game and / or play the video game frequently, ultimately increasing revenues for video game developers and providers and the video game industry as a whole. Thus, video game developers and providers continue to seek improvements in video game operation, particularly with respect to methods of messaging communication between players of a video game to increase player engagement and improve player experience. It is against this background that embodiments of the present disclosure are made. Summary of the Invention

[0002] A system for artificial intelligence (AI) assisted communication within a video game is disclosed. The system includes an input processor configured to receive input data including a message for communication to a target player of the video game. The system also includes a message identification engine having a first AI model component configured and trained to automatically process the input data to identify portions of the input data that define the message. The system also includes a message interpretation engine having a second AI model component configured and trained to automatically determine a meaning of the message defined by the portions of the input data identified by the message identification engine. The system also includes a message relevance assessment engine having a third AI model component configured and trained to automatically determine whether a message is relevant to a current game state of the target player and a current game context of the target player based on the meaning of the message determined by the message interpretation engine. The system also includes a communication delivery engine having a fourth AI model component configured and trained to automatically create a communication to the target player that conveys the meaning of the message determined by the message interpretation engine if the message is determined to be relevant by the message relevance assessment engine. The system also includes an output processor configured to deliver the communication created by the communication delivery engine to the target player.

[0003] A method of AI-assisted communication within a video game is disclosed. The method includes receiving input data including a message for communication to a target player of the video game. The method also includes automatically identifying, through execution of a first AI model component, a portion of the input data that defines the message. The method also includes automatically determining, through execution of a second AI model component, a meaning of the message defined by the portion of the input data. The method also includes automatically determining, through execution of a third AI model component, whether the message is relevant to a current game state and game context of the target player based on the meaning of the message. The method also includes automatically creating, through execution of a fourth AI model component, a communication to the target player that conveys the meaning of the message if the message is determined to be relevant to the current game state and game context of the target player. The method also includes delivering the communication to the target player. [Brief description of the drawings]

[0004] [Figure 1] 1 shows a diagram of an AI-assisted communication system within a video game, according to some embodiments. [Diagram 2] 2 shows a chart illustrating the acceleration of message communication provided by the system of FIG. 1 in accordance with some embodiments. [Diagram 3] 1 illustrates an example AI model that implements a neural network, according to some embodiments. [Figure 4] 1 illustrates a flowchart of a method for AI-assisted communication within a video game, according to some embodiments. [Diagram 5]1 illustrates various components of an exemplary server device within a cloud-based computing system that can be used to execute aspects of the system of FIG. 1 and the method of FIG. 4 for AI-assisted communication within a video game, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] In the following detailed description, several specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that embodiments of the present disclosure may be practiced without some or all of these specific details. In other instances, well-known process operations have not been described in detail in order not to unnecessarily obscure the present disclosure.

[0006] Many modern computer applications, such as video games, virtual reality applications, augmented reality applications, virtual world applications, and the like, provide various forms of communication to users of the computer applications. For ease of explanation, the term "video game" is used herein to refer to any of the above-mentioned types of computer applications in which a user can participate in the execution of the computer application. Also, for ease of explanation, the term "player" (as in video game player) is used herein to refer to a user who participates in the execution of any of the above-mentioned types of computer applications.

[0007] In various embodiments, in-game communication occurs between different players of the video game. In some embodiments, in-game communication occurs between a spectator of the video game and a player of the video game. In some embodiments, communication occurs between a virtual entity (e.g., a video game generated entity) and a player of the video game. In some embodiments, communication occurs between a spectator and a virtual entity. In some embodiments, communication occurs between two or more spectators of the video game. In various embodiments, the spectators of the video game can be real people and / or virtual (e.g., AI generated) spectators. In some embodiments, virtual spectators can be instantiated to represent real people. In various embodiments, communication conveyed to players in the video game can have one or more of a text format, an image format, a video format, an audio format, and a haptic format, among other formats that can essentially be implemented in a video game. In various embodiments, the content of the communication made in the video game is one or more of a gesture (made by either a real human body or a virtual entity in the video game), a spoken language sentence / phrase (made in either spoken or written form), and a video game controller input. In various embodiments, a video game controller can be any type of device used to communicate any type of user input to a computer system that executes a video game. For example, in various embodiments, the video game controller is one or more of a handheld video game controller, a head mounted display (HMD) device, a sensor embedded wearable device (e.g., gloves, glasses, vests, shirts, pants, capes, hats, etc.), and a brandishable control device (e.g., a stick, a club, a gun, a bow and arrow, a sword, a knife, a bat, a racket, a shield, etc.).

[0008] In various embodiments, communications are provided to a player of a video game within the context of the video game, and therefore these communications are communicated in a manner appropriate to the game state and game context of the video game. For example, if a target player of a video game is presented with a heads-up display as part of normal video game play, communications between players may be communicated as images and / or text messages in the target player's heads-up display. For ease of explanation, the term "target player" is used herein to refer to a player of a video game to whom a message is addressed, regardless of the message's source. Also, for ease of explanation, the term "message originator" is used herein to refer to an entity that originates a message that needs to be communicated to a target player. In various embodiments, the message originator is one or both of a human and a virtual entity, where the human is either playing or spectating the video game, and the virtual entity is generated as an object or aspect of the video game. In various embodiments, the message originator may be one or more of a video game player, a video game spectator, a computer-generated entity within the video game (e.g., an avatar, an object, etc.), among other entities that may essentially reside within or interface with the environment / context of the video game, and the message originator is permitted by the rules of the video game to communicate with the target player.

[0009] In many modern video games, the game state and / or game context change rapidly during the play of the video game, which can make the timely processing and transmission of messages from a message originator to a target player difficult. Also, as computer processing power continues to improve, many video games are moving toward using more gesture-based inputs to control player actions, interactions, and communications within the video game. In some embodiments, if the context of the message is a human gesture made by the message originator, the video game system may need to identify and convert the human gesture into a verbal communication that can be transmitted to the target player. In traditional linear procedural processing techniques, the procedure for identifying and converting the gesture and ultimately transmitting the verbal communication to the target player may take a significant amount of time compared to the rate at which the game state and / or game context are changing during the execution of the video game, so that when the verbal communication is received by the target player, the verbal communication may no longer be relevant to the current game state and / or game context associated with the target player, which may confuse and / or annoy the target player and, in some cases, adversely affect the target player's play of the video game. For example, if a target player is moving through a maze in a video game where the first right turn leads to the exit but the second right turn leads to a bottomless pit, a gesture message is processed from a message sender in verbal communication to the target player to encourage the player to make the next right turn, but if there is excessive latency / delay in processing the gesture message and in generating / transmitting the corresponding verbal communication to the target player, the target player may fall into the bottomless pit.Therefore, due to latency / delays in message processing (especially gesture-based message processing), a message that is well-intentioned for the benefit of the target player may actually end up having a detrimental effect on the target player.

[0010] In real-time video game development, timely communication is always a challenge. Some of the reasons for communication delays within a real-time video game play context include the slowness of the message originator in sending the message, the latency in the system processing the message and conveying the corresponding communication to the target player, and the type of message being processed. In particular, gesture-based messages require significant computational processing to be interpreted and converted into a communication format that can be conveyed to the target player. In a video game context, a gesture-based message can be any type of message conveyed through the positioning and movement of one or more real human body parts of a person participating in a video game and / or through the positioning and movement of one or more visual parts of a virtual entity within a video game context. In various embodiments, a gesture-based message can take the form of essentially any posture and / or movement of one or more body parts (or virtual entity parts) that is meaningful within the context of a video game. Also, in some embodiments, a gesture-based message can take the form of a formal / standardized sign language, such as American Sign Language.

[0011] Depending on the complexity of the gesture message, the video game system may not be able to determine the meaning of the gesture message and / or may not be able to create words corresponding to the gesture message quickly enough for the message to make sense to the target player, or to a third party, or to the video game system itself. In various embodiments disclosed herein, if the video game system determines, based on the current game state and game context, that the gesture message does not make sense at that current point in the video game, it may stop processing the gesture message so that the corresponding communication is not sent to the target player. For example, if the gesture message is to warn the target player that there is a bear in the woods and the target player has already seen the bear, the video game system may stop processing the gesture message and not send the communication to the target player. In some embodiments, the video game system operates to notify the message originator if the gesture message is not sent to the target player for any reason. For example, a message originator that instructed the target player with a gesture to beware of bears in the woods may be notified by the video game system that the target player has already seen the bear.

[0012] In some embodiments, if a particular gesture message is not sent due to latency (or expected latency), the meaning of that particular gesture message may be examined by the video game system to determine whether some other corresponding alternative communication may instead be sent to the target player in a timely manner. For example, in the bear-in-the-woods example above, the video game system may generate and send an alternative communication informing the target player that the target player is currently located between the bear and its cubs. In this way, the message originator is engaged in the video game and does not feel left out.

[0013] Also, in some embodiments, if a message originator is attempting to send a message that is too complex, the video game system may automatically shorten or summarize the complex message into a simpler message for communication to the target player. For example, if a message originator gestures to indicate the following message to a target player: "I see a bear. It's hiding behind a tree, so watch out," any latency in processing the gesture message may ultimately result in the communication of the message to the target player arriving too late to be useful. If latency is determined / predicted by the video game system to be an issue in processing the gesture communication, the video game system may automatically generate a summarized or truncated communication to convey the intent of the original message provided by the message originator. For example, with reference to the exemplary message above regarding a bear hiding behind a tree, the video game system may operate to automatically generate a truncated sentence such as "there's a bear behind the tree," thereby allowing communication to be made to the target player in a more timely manner.

[0014] FIG. 1 illustrates a diagram of an artificial intelligence (AI) assisted communication system 100 within a video game, according to some embodiments. The system 100 includes an input processor 111 configured to receive input data from one or more message source(s) 101, as indicated by arrow 113. The input data includes a message for communication to a target player 149 of the video game. In various embodiments, the one or more message source(s) 101 include a human 103, an audio clip 107, and a game controller 109. In some embodiments, the audio clip 107 is generated by the human 103. In some embodiments, the audio clip 107 is provided by an entity other than the human 103, such as a virtual entity within the video game. The game controller 109 can be any type of game controller, including, but not limited to, one or more of a handheld video game controller, an HMD device, a sensor-embedded wearable device, and a wieldable control device. In some embodiments, the human 103 is a co-player of the target player 149 in the video game, and messages are sent from the message origin(s) 101 to the target player 149. In some embodiments, the human 103 is a spectator of the video game being played by the target player 149 to which messages are sent by the message origin(s) 101. In some embodiments, the human 103 uses equipment that allows the human 103 to interface with and participate in the video game. For example, in FIG. 1, the human 103 is shown using an HMD 104 and a camera 102. The HMD 104 allows the human 103 to participate in the video game in an immersive manner. The camera 102 is provided to capture images / videos that include messages based on gestures provided by the human 103. For example, the human 103 is shown making a gesture 105 with his left arm and hand 106 that is intended to be interpreted as a message directed to the target player 149.Further, it should be understood that essentially any type of gesture can be made by human 103, so long as the gesture can be interpreted by system 100 as having meaning within the context of the video game.

[0015] In some embodiments, the input processor 111 includes an input filter 111A configured to implement a filtering process to advantageously isolate message-related input data by removing portions of the input data received from the one or more message source(s) 101 that are not relevant to the message that needs to be communicated to the target player 149. In some embodiments, the input data received from the one or more message source(s) 101 includes a video of the human 103 making a gesture, and the message for communication to the target player 149 is provided at least in part by the gesture. In some embodiments, the portion of the input data that defines the message is identified as a pixel region within each of several video frames of the video of the human 103 making the gesture. In some embodiments, in addition to the video of the human 103 making the gesture, the input data received from the one or more message source(s) 101 also includes one or more of an audio clip 107 and an input signal from a video game controller 109, and the message for communication to the target player 149 is provided at least in part by one or more of the audio clip 107 and an input signal from the video game controller 109.

[0016] It should also be appreciated that the system 100 has real-time access to the current game state 153A and game context 153B of the video game being played by the target player 149. In some embodiments, the game state 153A defines the state of gameplay of a particular player of the video game at a particular point in time. In some embodiments, the game state 153A includes user / player saved data, which includes information that personalizes the video game for the corresponding player. For example, the game state 153A can include information related to a user's character, such that the video game renders a character, e.g., in terms of shape, appearance, clothing, weapons, etc., which may be specific or unique to that user. And, the game context 153B is essentially any and all data that defines the scene in which the player of the video game is present and / or with which the player of the video game is associated. In some embodiments, the game context 153B also includes the history of the play of the video game by a particular player of the video game. In some embodiments, game context 153B also includes information regarding one or more other scenes of the video game to which a particular player of the video game may transition depending on actions / decisions performed by various entities within the video game.

[0017] The system 100 also includes a message identification engine 115 that receives message-related input data from the input processor 111, as indicated by arrow 117. The message identification engine 115 is configured to process the message-related input data to identify portions of the message-related input data that define a message that needs to be communicated to the target player 149. In some embodiments, the message identification engine 115 includes a gesture message identifier AI model component 119 that is configured and trained to automatically identify portions of the image / video data that include a gesture made by the human 103, the gesture conveying a message to the target player 149. In some embodiments, the message identification engine 115 also includes an audio message identifier AI model component 121 that is configured and trained to automatically identify portions of an audio clip that include a message to the target player 149 or that augment a message based on a gesture to the target player 149. Additionally, in some embodiments, the message identification engine 115 includes a controller message identifier AI model component 123 that is configured and trained to automatically identify controller input signals that include a message for the target player 149 or that augment a gesture-based message for the target player 149. Thus, within the overall system 100, the message identification engine 115 includes at least a first AI model component (one or more of 119, 121, and 123) that is configured and trained to process input data received from the message origin(s) 101 and automatically identify portions of the input data that define a message for the target player 149.

[0018] The system also includes a message interpretation engine 125 that receives the portion of the input data defining the message to the target player 149 as determined by the message identification engine 115, as indicated by arrow 127. The message interpretation engine 125 is configured to process the portion of the input data defining the message to the target player 149 and determine the meaning of the message to the target player 149. In some embodiments, the message interpretation engine 125 includes a gesture interpreter AI model component 129 that is configured and trained to automatically determine the meaning of the gestures in the message to the target player 149. In some embodiments, the gesture interpreter AI model component 129 is configured and trained to determine the meaning of the gesture message through identification of the body part used to perform the gesture and through identification of the positioning of the body part and / or the manner in which the body part moves. In some embodiments, the gesture interpreter AI model component 129 is configured and trained to provide acceleration of the delivery of the message to the target player 149 by determining the meaning of the message to the target player 149 before the human 103 has finished performing the gesture.

[0019] Also, in some embodiments, the message interpretation engine 125 includes an audio interpreter AI model component 131 that is configured and trained to automatically determine the meaning of an audio clip associated with a message for the target player 149. Also, in some embodiments, the message interpretation engine 125 includes a controller interpreter AI model component 133 that is configured and trained to automatically determine the meaning of a controller input signal associated with a message for the target player 149. Thus, within the overall system 100, the message interpretation engine 125 includes at least a second AI model component (one or more of 129, 131, and 133) that is configured and trained to automatically process the portion of the input data that defines the message for the target player 149 (as determined by the message identification engine 115) and automatically determine the meaning of the message for the target player 149.

[0020] In some embodiments, the message interpretation engine 125 is used to implement a message auto-completion process. In some embodiments, based on the current game state 153A and game context 153B, a probabilistic decision can be made about the message that the message originator 101 is about to provide before the message originator 101 actually finishes providing the message. When the probability of a given message being provided reaches a certain threshold, the message interpretation engine 125 can automatically determine that a given message is being provided even if the message originator 101 has not yet finished providing the message. This auto-completion process is particularly useful with respect to interpreting gesture messages. In various embodiments, the message interpretation engine 125 can use many different input types, signals, and data to narrow down the possible options of messages being provided by the message originator 101. For example, gaze direction, emotion detection, and / or any other input parameters associated with the message originator 101 can be used to intuit the intent of the message being provided by the message originator 101. In some embodiments, different gesture interpreter AI model components 129 can be built for different message originators 101 to facilitate the probabilistic decisions underlying the message auto-completion process.

[0021] In some embodiments, records of message communications made during video game play, along with corresponding game state and game context data, can be analyzed offline, e.g., post-game play, to build and train AI models 129, 131, and 133 that enable prediction of message communications for / within a particular game state and / or game context. Then, as the video game is played, the AI ​​models 129, 131, and 133 can be used to make predictive decisions regarding the intent of the message communication and the interpretation of the message to speed up the message communication during video game play. In some embodiments, the audio interpreter AI model component 131 can learn keywords and responses to aid in the accuracy and speed of determining the intent and interpretation of the message communication. For example, when the system 100 hears the keyword "glitch" from the target player 149, the system 100 can use the keyword "glitch" as a type of negative feedback to learn that there was some glitch in the interpretation and / or delivery of the message. Post-game analysis of communications made by one or more message source(s) 101 to target players 149 can be used to train AI models 129, 131, and 133 in making predictive interpretations of messages. This type of AI model training can be done across various video games played by message source(s) 101 to learn tendencies and modes of message communication.

[0022] The system 100 also includes a message relevance assessment engine 135 that receives the meaning of the message to the target player 149 determined by the message interpretation engine 125, as indicated by arrow 137. The message relevance assessment engine 135 is configured to automatically determine whether a message intended for the target player 149 is relevant to the current game state 153A of the target player 149 and the current game context 153B of the target player 149 based on the meaning of the message determined by the message interpretation engine 125. In this manner, the system 100 is configured to ensure that a message from one or more message origin(s) 101 is meaningfully relevant to the current game state 153A and the current game context 153B before communicating with the target player 149 via the message. In some embodiments, the message relevance assessment engine 135 includes an AI model component 136 configured and trained to assess the relevance of a message to the target player 149, taking into account the current game state 153A and game context 153B of the video game. In some embodiments, the AI ​​model component 136 is configured to determine the scope of the current game context 153B based on whether the message is intended to be a communication between individuals or whether the message is intended to be broadcast to multiple people, and the scope of the established current game context 153B encompasses all target recipients of the message. The message relevance engine 135 continuously analyzes the current game state 153A and game context 153B associated with the target player 149 of the message to determine whether communication of the message to the target player 149 is appropriate, timely, and / or beneficial to the target player 149. In this manner, the system 100, via the message relevance evaluation engine 135, uses its understanding of what is currently (in real time) happening by the target player 149 to determine whether a message should be communicated to the target player 149.

[0023] In some embodiments, the AI ​​model component 136 is configured and trained based on a large population of message relevance outcomes for many different messages communicated in various game states and game contexts. In some embodiments, if the message relevance assessment engine 135 determines that the message is not sufficiently relevant to the current game state 153A of the target player 149 and / or the current game context 153B of the target player 149, the message relevance assessment engine 135 can direct that no further processing of the message by the system 100 is performed. Thus, within the overall system 100, the message relevance assessment engine 135 includes a third AI model component (136) configured and trained to automatically determine whether a message is relevant to the current game state 153A of the target player 149 and the current game context 153B of the target player 149 based on the meaning of the message determined by the message interpretation engine 125.

[0024] In some embodiments, the AI ​​model component 136 of the message relevance assessment engine 135 is configured and trained to evaluate the meaning of a message to the target player 149 to identify the subject of the message and actions associated with the subject of the message. In these embodiments, the AI ​​model component 136 of the message relevance assessment engine 135 is configured and trained to consider a message to the target player 149 as relevant if it is determined that the subject of the message is still present within the current game context 153B of the target player 149 and that the actions associated with the subject of the message are still relevant to the current game state 153A of the target player 149. In some embodiments, the AI ​​model component 136 of the message relevance assessment engine 135 is configured and trained to cancel further processing of the message to the target player 149 by the system 100 if it is determined that the subject of the message is no longer present within the current game context 153B of the target player 149 or that the actions associated with the subject of the message are no longer relevant to the current game state 153A of the target player 149. Additionally, in some embodiments, the AI ​​model component 136 of the message relevance assessment engine 135 is configured and trained to cancel further processing of a message by the system 100 for the target player 149 if it is determined that actions related to the subject of the message are not beneficial for improving the current game state 153A of the target player 149. Additionally, in some embodiments, the AI ​​model component 136 of the message relevance assessment engine 135 is configured and trained to cancel further processing of a message by the system 100 for the target player 149 if it is determined that the subject of the message and actions related to the subject of the message are redundant with respect to another communication already communicated to the target player 149.

[0025] The system 100 also includes a communication delivery engine 139 that receives both the meaning of the message to the target player 149 as determined by the message interpretation engine 125, as indicated by arrow 141, and the determination of the relevance of the message to the target player 149 as determined by the message relevance assessment engine 135, as indicated by arrow 142. The communication delivery engine 139 is configured to automatically create a communication to the target player 149 that conveys the meaning of the message to the target player 149 as determined by the message interpretation engine 125, if the message to the target player 149 is determined to be relevant by the message relevance assessment engine 135. In some embodiments, the communication delivery engine 139 includes an AI model component 140 that is configured and trained to create a communication to the target player 149 based on the current game state 153A and the current game context 153B of the target player 149 to provide the communication in an optimized manner commensurate with the level of urgency and importance of the message to the target player 149. It should be appreciated that the communications delivery engine 139 determines the format and content of the communication to be delivered to the target player 149. For example, if the target player 149 is using an HMD 152, the communications delivery engine 139 creates a communication for delivery via the HMD 152. In various embodiments, the format of the communication created by the communications delivery engine 139 is one or more of an image, a video, a text message, an audio clip, and a haptic action, among essentially any other possible communication format applicable to the video game being played.In some embodiments, the AI ​​model component 140 of the communication delivery engine 139 is configured and trained to automatically determine a format of the communication to the target player 149, the format being one or more of a text format, an audible format, a graphical format, and a haptic format. The communication delivery engine 139 also determines the extent to which the message to the target player 149 is shortened in the communication that is ultimately delivered to the target player 149. Thus, within the overall system 100, the communication delivery engine 139 includes a fourth AI model component (140) that is configured and trained to automatically create a communication to the target player 149 that conveys the meaning of the message as determined by the message interpretation engine 125 if the message is determined to be relevant by the message relevance assessment engine 135.

[0026] In some embodiments, the AI ​​model component 140 of the communication transfer engine 139 is configured and trained to automatically create a unified communication to the target player 149 that conveys a meaning determined by the message interpretation engine 125 to be similar to the multiple messages, instead of automatically creating multiple separate communications to the target player 149 for each of the multiple messages. For example, in a video game involving group play of multiple players, if several players are simultaneously attempting to communicate the same or similar information to the target player 149, the AI ​​model component 140 of the communication transfer engine 139 detects the situation and responds by generating a single timely and concise communication to the target player 149, instead of multiple redundant communications from the several players. In some embodiments, the AI ​​model component 140 of the communication transfer engine 139 is configured and trained to automatically recognize a higher urgency level for communicating a communication to the target player 149. In some of these embodiments, the AI ​​model component 140 of the communication delivery engine 139 is configured and trained to automatically create an abbreviated communication to the target player 149 that conveys a shorter version of the message in response to recognizing a higher level of urgency for delivering the communication to the target player 149.

[0027] Through the communication delivery engine 139, the system 100 uses its knowledge of what is currently (real-time) happening to the target player 149 to determine how best to deliver a message to the target player 149, e.g., what content and format a communication to the target player 149 should have to best deliver the message to the target player 149's benefit. In some embodiments, the communication delivery engine 139 operates to visually highlight certain elements within the target player's 149's current game context 153B to efficiently and effectively deliver one or more nuances of the message. Using both the message relevance assessment engine 135 and the communication delivery engine 139, the system 100 operates to process and deliver messages based on the current game state 153A and game context 153B associated with the target player 149, such that only communications that are relevant, timely, concise, and helpful to the target player 149 are delivered.

[0028] The system also includes an output processor 143 configured to receive communications created by the communication delivery engine 139, as indicated by arrow 145, and to deliver communications created by the communication delivery engine 139 to the target player 149, as indicated by arrow 147. In some embodiments, the output processor 143 is defined to prepare and transmit communications in data packets to the target player 149 over a network, such as one or more of a local area network (wired and / or wireless and / or optical), a wide area network (wired and / or wireless and / or optical), a cellular network, a satellite network, and the Internet, among essentially any other type of network capable of transmitting a data signal. In these embodiments, the data packets are prepared by the output processor 143 according to any known and available network communication protocol. In some embodiments, the output processor 143 includes a network interface card (NIC) to provide packetization of outgoing data transmitted from the system 100 to the target player 149.

[0029] It should be appreciated that the arrangement of the system 100 shown in FIG. 1 is one example of various ways in which the system 100 for AI-assisted communication in video games can be configured to implement various operations and methods disclosed herein. For example, in alternative embodiments, two or more of the message identification engine 115, the message interpretation engine 125, the message relevance assessment engine 135, and the communication propagation engine 139 are implemented in accordance with a combined AI model. Also, in some embodiments, the message relevance engine 135 is either omitted or turned off, so that all messages are considered relevant by default. Also, in some embodiments, the incoming data to the input processor 111 includes metadata that identifies the message portions of the input data. In these embodiments, the message identification engine 115 is bypassed, so that the message portions of the input data identified through the associated metadata are provided directly from the input processor 111 to the message interpretation engine 125. Also, in some embodiments, the system 100 is implemented as an accessibility option, so that the system 100 can be enabled or disabled by the message originator 101 and / or the target player 149.

[0030] FIG. 2 shows a chart illustrating the acceleration of message communication provided by the system 100 of FIG. 1 according to some embodiments. It should be appreciated that by utilizing AI within the system 100, it is possible to process and convey messages to the target player 149 in a more rapid and optimized manner than is possible through linear procedure (non-AI) processing of messages to the target player 149. FIG. 2 shows a first duration 201 required to deliver a given message using linear procedure (non-AI) processing of the given message. The first duration 201 extends from time zero (0) to a first delivery time (DT1) 203. FIG. 2 also shows a second duration 205 required to deliver a given message using the AI-based system 100. The second duration 205 is the time from time zero (0) to a second delivery time (DT2) 215. FIG. 2 shows that the system 100 accelerates the processing and delivery of the message to the target player 149 by time 217. 2 also illustrates various points in the non-linear AI-based processing of a given message by the system 100. For example, the given message is identified by the message identification engine 115 at a time indicated by arrow 207. The given message is then interpreted by the message interpretation engine 125 at a time indicated by arrow 209. The relevance of the given message is then determined by the message relevance assessment engine 135 at a time indicated by arrow 211. A communication to a target player 149 for the given message is then created by the communication delivery engine 139 at a time indicated by arrow 213. The communication is then finally delivered to the target player 149 at a second delivery time (DT2) 215.

[0031] As shown in FIG. 2, the system 100 provides faster message communication from a message originator 101 to a target player 149. In some embodiments, faster message communication is provided, at least in part, through faster AI-based identification of messages in input data using a message identification engine 115. In some embodiments, faster message communication is provided, at least in part, through faster AI-based interpretation of messages using a message interpretation engine 125. In some embodiments, faster message communication is provided, at least in part, through generation and delivery of shortened versions of messages using an AI model component 140 in a communication delivery engine 139. In some embodiments, faster message communication is provided, at least in part, through determination and selection of the most rapid / efficient and effective method of message delivery using an AI model component 140 in a communication delivery engine 139.

[0032] 3 illustrates an example AI model 300 implementing a neural network 301 according to some embodiments. In some embodiments, the AI ​​model 300 represents each AI model component in the system 100, including the gesture message identifier AI model component 119, the audio message identifier AI model component 121, the controller message identifier AI model component 123, the gesture interpreter AI model component 129, the audio interpreter AI model component 131, the controller interpreter AI model component 133, the AI ​​model component 136 in the message relevance assessment engine 135, and the AI ​​model component 140 in the communication propagation engine 139.

[0033] Given an input, the AI ​​model 300 can analyze the input and provide an appropriate response to the input. For example, if the AI ​​model 300 is trained as a gesture message identifier AI model component 119, the AI ​​model 300 can be used to identify gestures that convey messages in an image and / or video frame and / or sequence of video frames. If the AI ​​model 300 is trained as an audio message identifier AI model component 121, the AI ​​model 300 can be used to identify messages in audio clips. If the AI ​​model 300 is trained as a controller message identifier AI model component 123, the AI ​​model 300 can be used to identify messages associated with a controller input signal or combination of controller input signals. If the AI ​​model 300 is trained as a gesture interpreter AI model component 129, the AI ​​model 300 can be used to determine the meaning of a message conveyed by a given gesture. If the AI ​​model 300 is trained as an audio interpreter AI model component 131, the AI ​​model 300 can be used to determine the meaning of a message in an audio clip. Additionally, when the AI ​​model 300 is trained as a controller interpreter AI model component 133, the AI ​​model 300 can be used to determine the meaning of a message conveyed by one or more controller input signals. Additionally, when the AI ​​model 300 is trained as an AI model component 136 in a message relevance assessment engine 135, the AI ​​model 300 can be used to determine the relevance of a message to a current game state 153A and a current game context 153B. Additionally, when the AI ​​model 300 is trained as an AI model component 140 in a communication delivery engine 139, the AI ​​model 300 can be used to create a communication to a target player 149 and deliver the message in the most appropriate and efficient manner.

[0034] In some embodiments, the message identification engine 115 includes one or more modelers configured to build each of the gesture message identifier AI model component 119, the audio message identifier AI model component 121, and the controller message identifier AI model component 123 as needed to learn about the message identification process for a given game state 153A and game context 153B. In some embodiments, the message interpretation engine 125 includes one or more modelers configured to build each of the gesture interpreter AI model component 129, the audio interpreter AI model component 131, and the controller interpreter AI model component 133 as needed to learn about the message interpretation process for a given game state 153A and game context 153B. In some embodiments, the message relevance assessment engine 135 includes a modeler configured to build an AI model component 136 as needed to learn about the message relevance assessment process for a given game state 153A and game context 153B. In some embodiments, the communication propagation engine 139 includes a modeler that is configured to build AI model components 140 as needed to learn about the communication creation process for a given game state 153A and game context 153B. In various embodiments, the system 100 utilizes AI, including deep learning algorithms, reinforcement learning, and / or other AI-based algorithms, to build and train the various AI model components 119, 121, 123, 129, 131, 133, 136, and 140. Also, in some embodiments, the system 100 is configured to continuously refine the training of one or more of the various AI model components 119, 121, 123, 129, 131, 133, 136, and 140 given any updated training data.

[0035] In various embodiments, neural network 301 can be implemented as a deep neural network with supervised or unsupervised training, a deep convolutional neural network, and / or a recurrent neural network. In some embodiments, neural network 301 includes a deep learning network that supports reinforcement learning (e.g., through the use of success criteria, success metrics, etc.) or reward-based learning. For example, in some embodiments, neural network 301 is set up as a Markov decision process (MDP) that supports a reinforcement learning algorithm.

[0036] Neural network 301 represents a network of interconnected nodes, such as an artificial neural network. In FIG. 3, each circle represents a node. Each node learns some information from training data. Through the interconnections, knowledge can be exchanged between nodes. In FIG. 3, each arrow between nodes represents an interconnection. An input to neural network 301 activates a set of nodes. This set of nodes then activates other nodes, thereby propagating knowledge about the input. This activation process is repeated across other nodes until an output is provided. Exemplary neural network 301 includes a hierarchy of nodes. At the bottom hierarchy, there is an input layer 302. The input layer 302 includes a set of input nodes. In some embodiments, intermediate predictions of AI model 300 are determined through a classifier that creates labels, such as outputs, features, nodes, classifications, etc. At the top hierarchy, there is an output layer 304. The output layer 304 includes a set of output nodes. Each output node represents a possible output of AI model 300 for a given set of inputs provided to AI model 300. In various embodiments, these results generated by the AI ​​model 300 can be compared to predetermined and true results, or learned changes and results, to refine and / or modify parameters used by the system 100 to iteratively determine predicted or expected responses and / or changes appropriate for a given set of inputs. Nodes in the neural network 301 learn parameters of the trained AI model 300 that can be used to make such decisions when refining parameters.

[0037] In some embodiments, there are one or more hidden layer(s) 303 between the input layer 302 and the output layer 304 in the neural network 301. The hidden layer(s) 303 include "X" hidden layers, where "X" is an integer equal to or greater than 1. Each of the hidden layer(s) 303 includes a set of hidden nodes. The input nodes of the input layer 302 are interconnected to the hidden nodes of the first hidden layer 303. The hidden nodes of the last ("Xth") hidden layer 303 are interconnected to the output nodes of the output layer 304, such that the input nodes are not directly interconnected to the output nodes. When multiple hidden layers 303 are present, the input nodes of the input layer 302 are interconnected to the hidden nodes of the bottom (first) hidden layer 303. The hidden nodes of the first hidden layer 303 are then interconnected to the hidden nodes of the next hidden layer 303, and so on until the hidden nodes of the top ("Xth") hidden layer 303 are interconnected to the output nodes of the output layer 304.

[0038] An interconnect connects two nodes in the neural network 301. The interconnects in the exemplary neural network 301 are indicated by arrows. Each interconnect has a numerical weight that can be learned, allowing the neural network 301 to adapt to the input and thus learn. In general, the hidden layer(s) 303 allow knowledge about the input nodes of the input layer 302 to be shared among all tasks corresponding to output nodes of the output layer 304. In this regard, in some embodiments, a transformation function f is applied to the input nodes of the input layer 302 via the hidden layer(s) 303. In some cases, the transformation function f is nonlinear. Also, various nonlinear transformations f are available, including, for example, a rectified linear function f(x)=max(0,x).

[0039] In some embodiments, the neural network 301 also uses a cost function c to find the optimal solution. The cost function c measures the deviation between the predicted value output by the neural network 301, defined as f(x), for a given input x, and the ground truth or target value y (e.g., expected outcome). The optimal solution represents a situation where there is no solution with a lower cost than the cost of the optimal solution. An example of the cost function c is the mean squared error between the predicted value and the ground truth for data where ground truth labels are available. During the learning process, the neural network 301 can employ various optimization methods to learn model parameters that minimize the cost function c (e.g., learn the weights of the interconnections between nodes in the hidden layer(s) 303) using an error backpropagation algorithm. One example of such an optimization method is stochastic gradient descent.

[0040] FIG. 4 illustrates a flowchart of a method for AI-assisted communication in a video game, according to some embodiments. The method includes an operation 401 for receiving input data including a message for communication to a target player 149 of the video game. In some embodiments, the input data includes one or more of: (A) a video of the person 103 making a gesture; and (B) tracking metadata associated with the person 103 making the gesture, where the message for communication to the target player 149 is provided at least in part by the gesture. In some embodiments, the tracking metadata defines one or more of a pose of the person 103, a movement of the person 103, a position of a controller device, an orientation of the controller device, a movement of the controller device, a position of a wearable device worn by the person 103, an orientation of a wearable device worn by the person 103, and a movement of a wearable device worn by the person 103. In some embodiments, the tracking metadata is defined in terms of six degrees of freedom (6DOF), including three translational degrees of freedom (x, y, z) and three rotational degrees of freedom (about the x-axis, the y-axis, and the z-axis). In some embodiments, the input data includes both gestures and one or more of an audio clip 107 and a video game controller 109 input, and the message for communication to the target player 149 is provided at least in part by one or more of the audio clip 107 and the video game controller 109 input. In some embodiments, a portion of the input data defining the message is identified as a pixel region within each of several video frames of a video of the person 103 performing the gesture as captured by the camera 102. The method also includes an operation 403 for automatically identifying the portion of the input data defining the message through execution of a first AI model component (e.g., one or more of 119, 121, 123).

[0041] The method also includes an operation 405 for automatically determining a meaning of the message defined by the portion of the input data through execution of a second AI model component (e.g., one or more of 129, 131, 133). In some embodiments, the method includes executing the second AI component to automatically identify a body part of the person 103 used to perform the gesture, and automatically identify one or more of a positioning of the body part and a manner of movement of the body part. Also, in these embodiments, the method includes executing the second AI component to analyze the body part, the positioning of the body part, and / or the manner of movement of the body part to determine the meaning of the message. It should be appreciated that the body part of the person 103 used to perform the gesture can be any body part of the person (e.g., face, eyes, arms, legs, hands, feet, fingers, etc.) or any combination of body parts of the person. In some embodiments, the body part(s) of the person 103 used to perform the gesture include the person's eyes. In these embodiments, the second AI component is configured to analyze the gaze direction and / or the manner of movement of one or both eyes of the person 103 to determine the meaning of the message conveyed by the eye(s) of the person 103. It should also be appreciated that in some embodiments, the body part of the person 103 used to make the gesture is the person's face. In these embodiments, the second AI component is configured to analyze the facial expression of the person 103 to determine the meaning of the message conveyed by the face of the person 103. Also, in some embodiments, the second AI model is configured and trained to process the gesture information of the message in the form of tracking metadata corresponding to the tracking of various components and / or parts associated with the person 103, such as controller tracking metadata (e.g., 6DOF), limb tracking metadata, hand(s) / finger(s) tracking metadata, among essentially any other type of tracking metadata. The processing of tracking metadata by the second AI model is particularly relevant to video games implementing virtual reality and augmented reality.In some embodiments, the method includes providing acceleration of the processing and delivery of the message to the target player 149 by executing a second AI model component to determine the meaning of the message before the person 103 finishes making the gesture.

[0042] The method also includes an operation 407 for automatically determining, based on the meaning of the message through execution of a third AI model component (e.g., 136), whether the message is relevant to the current game state 153A and the current game context 153B of the target player 149. In some embodiments, the method includes executing the third AI model component to evaluate the meaning of the message and identify a subject of the message and an action associated with the subject of the message. In these embodiments, the method also includes executing the third AI model component to consider the message relevant if it is determined that the subject of the message is still present in the current game context 153B of the target player 149 and the action associated with the subject of the message is still relevant to the current game state 153A of the target player 149. In these embodiments, the method also includes executing the third AI model component to cancel further processing of the message if it is determined that the subject of the message is no longer present in the current game context 153B of the target player 149 or the action associated with the subject of the message is no longer relevant to the current game state 153A of the target player 149.

[0043] In some embodiments, the method includes executing a third AI model component to cancel further processing of the message if it is determined that actions associated with the subject of the message are not beneficial for improving the current game state 153A of the target player 149. Also, in some embodiments, the method includes executing a third AI model component to cancel further processing of the message if it is determined that the subject of the message and actions associated with the subject of the message are redundant with another communication already communicated to the target player 149.

[0044] The method also includes an operation 409 for automatically creating, through execution of a fourth AI model component (e.g., 140), a communication to the target player 149 that conveys the meaning of the message if the message is determined to be relevant to the current game state 153A and current game context 153B of the target player 149. The method also includes an operation 411 for delivering the communication to the target player 149.

[0045] In some embodiments, the method includes executing a fourth AI model component to automatically create a unified communication to the target player 149 that conveys a similar meaning to the multiple messages instead of automatically creating multiple communications to the target player 149 for each of the multiple messages determined by the second AI model component. Also, in some embodiments, the method includes executing a fourth AI model component to automatically recognize a higher urgency level for communicating the communication to the target player 149. In these embodiments, the method also includes executing the fourth AI model component to automatically create an abbreviated communication to the target player 149 that conveys a shorter version of the message in response to recognizing a higher urgency level for communicating the communication to the target player 149. Furthermore, in some embodiments, the method includes executing the fourth AI model component to automatically determine a format of the communication to the target player 149, the format being one or more of a text format, an audible format, a graphical format, and a haptic format.

[0046] The system 100 is configured to recognize that the message originator 101 is attempting to communicate with a message to the target player 149 (whether through gestures, speech language, and / or controller input). Once the system 100 understands the intent of the message, the system 100 utilizes its knowledge of the current game state 153A and the current game context 153B of the target player 149 to make an informed decision on how to process the message and convey the message to the target player 149. In this manner, the system 100 mitigates the occurrence of misaligned communication, which occurs when the communication received by the target player 149 does not match the intent of the message originator 101. In some embodiments, the system 100 implements a continuous filtering process (by the message relevance engine 135) to allow either the message to be sent to the target player 149 or to block the message from being sent to the target player 149. It should be appreciated that the ability to interrupt and stop the processing of messages that are either not timely or relevant to the target player 149 can save processor cycles and memory usage, thereby improving the overall computational efficiency of the video game system. Also, in some embodiments, the system implements a message optimization process (by the communication delivery engine 139) to automatically modify the content and / or format of messages before being delivered to the target player 149 based on what is most appropriate for the target player 149, given the target player's 149 current game state 153A and / or current game context 153B.It should be appreciated that the system 100 provides management of message communication from message originator(s) 101 to the target player 149 (or multiple target players) in at least the following ways: A) ensuring that messages are communicated to the target player 149 in a timely manner; B) shortening messages as necessary to reduce latency in communicating messages to the target player 149; C) avoiding communication of messages to the target player 149 that are not relevant or have become irrelevant to the target player's 149 current game state 153A and current game context 153B; D) avoiding communication of messages to the target player 149 that are not beneficial to the target player 149; and E) avoiding communication of redundant messages to the target player 149.

[0047] FIG. 5 illustrates various components of an exemplary server device 500 in a cloud-based computing system that can be used to execute aspects of the system 100 and method for AI-assisted communication in a video game, according to some embodiments. The block diagram illustrates the server device 500, which can incorporate or be a personal computer, a video game console, a personal digital assistant, a head-mounted display (HMD), a wearable computing device, a laptop or desktop computing device, a server, or any other digital computing device suitable for implementing embodiments of the present disclosure. The server device (or simply referred to as a "server" or "device") 500 includes a central processing unit (CPU) 502 for executing software applications and optionally an operating system. The CPU 502 can be configured with one or more homogeneous or heterogeneous processing cores. For example, the CPU 502 is one or more general-purpose microprocessors with one or more processing cores. Further embodiments can be implemented using one or more CPUs with a microprocessor architecture that is particularly adapted for highly parallel and computationally intensive applications, such as interpreting queries, identifying contextually relevant resources, and immediately implementing and rendering the contextually relevant resources in a video game. The device 500 may be localized to the player playing the game segment (e.g., a game console), or remote from the player (e.g., a back-end server processor), or to one of many servers using virtualization in a cloud-based gaming system 500 for remote gameplay streaming to a client.

[0048] Memory 504 stores applications and data used by CPU 502. Storage 506 provides non-volatile storage and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input devices 508 communicate user input from one or more users to device 500, and examples of user input devices 508 may include a keyboard, mouse, joystick, touchpad, touch screen, still or video recorder / camera, gesture-recognizing tracking device, and / or microphone. Network interface 514 enables device 500 to communicate with other computer systems over an electronic communications network, which may include wired or wireless communications over local area networks and wide area networks such as the Internet. The audio processor 512 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 502, the memory 504, and / or the storage 506. The components of the device 500, including the CPU 502, the memory 504, the data storage 506, the user input device 508, the network interface 514, and the audio processor 512, are connected via one or more data buses 522.

[0049] A graphics subsystem 520 is further coupled to the data bus 522 and the components of the device 500. The graphics subsystem 520 includes a graphics processing unit (GPU) 516 and a graphics memory 518. The graphics memory 518 includes a display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. The graphics memory 518 may be integrated in the same device as the GPU 516, may be coupled as a separate device from the GPU 516, and / or may be implemented within the memory 504. The pixel data may be provided to the graphics memory 518 directly from the CPU 502. Alternatively, the CPU 502 may provide data and / or instructions defining a desired output image to the GPU 516, from which the GPU 516 generates pixel data for one or more output images. The data and / or instructions defining the desired output image may be stored in the memory 504 and / or the graphics memory 518. In an embodiment, GPU 516 includes 3D rendering capabilities that generate pixel data for output images from instructions and data that define scene geometry, lighting, shading, texture, motion, and / or camera parameters. GPU 516 may further include one or more programmable execution units capable of executing shader programs.

[0050] Graphics subsystem 520 periodically outputs pixel data of an image from graphics memory 518 for display on display device 510. Display device 510 may be any device capable of displaying visual information in response to a signal from device 500, including CRT, LCD, plasma, and OLED displays. In addition to display device 510, the pixel data may also be projected onto a projection surface. Device 500 may provide analog or digital signals to display device 510, for example.

[0051] Implementations of the present disclosure for communication between computing devices may be practiced with a variety of computing device configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, head mounted displays, wearable computing devices, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0052] In some embodiments, communication may be facilitated using wireless technologies. Such technologies may include, for example, 5G wireless communication technologies. 5G is the fifth generation of cellular network technology. 5G networks are digital cellular networks in which the service area covered by a provider is divided into small geographical areas known as cells. Analog signals representing sound and video are digitized by the telephone, converted by an analog-to-digital converter, and transmitted as a bit stream. All 5G wireless devices within a cell communicate with local antenna arrays and low-power automatic transceivers (transmitters and receivers) within the cell by electromagnetic waves over frequency channels assigned by the transceivers from a frequency pool reused by other cells. The local antennas are connected to the telephone network and the Internet by high-bandwidth optical fiber or wireless backhaul connections. As with other cellular networks, mobile devices moving from one cell to another are automatically moved to the new cell. Of course, 5G networks are merely an example of a type of communication network, and embodiments of the present disclosure may use previous generations of wireless or wired communication, as well as later generations of wired or wireless technologies that come after 5G.

[0053] In view of the above embodiments, it will be appreciated that the present disclosure may employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the present disclosure are useful machine operations. The present disclosure also relates to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purposes, or the apparatus may be a general-purpose computer selectively enabled or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used with computer programs written in accordance with the teachings herein, or it may prove more convenient to construct a more specialized apparatus to perform the required operations.

[0054] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be coordinated to occur at slightly different times, or may be distributed within a system that allows for the occurrence of processing operations at various intervals related to the process.

[0055] One or more embodiments may also be fabricated as computer readable code (program instructions) on a computer readable medium. A computer readable medium is any data storage device that can store data which can subsequently be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read only memory, random access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium may include computer readable tangible media distributed over network-connected computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0056] Although the foregoing embodiments have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments are not to be limited to the details set forth herein, but may be modified within the scope of the appended claims and their equivalents.

[0057] It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations that use various features disclosed herein. Thus, the examples provided are only some of the possible examples and are not intended to limit the various implementations that may be defined by combining various elements. In some examples, an implementation may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

Claims

1. 1. A system for artificial intelligence assisted communication within a video game, comprising: an input processor configured to receive input data including a message for communication to a target player within the video game; a message identification engine having a first artificial intelligence model component configured and trained to automatically process the input data to identify portions of the input data that define the message; a message interpretation engine having a second artificial intelligence model component configured and trained to automatically determine the meaning of the message defined by the portion of the input data identified by the message identification engine; a message relevance assessment engine having a third artificial intelligence model component configured and trained to automatically determine whether the message is relevant to the target player's current game state and the target player's current game context based on the meaning of the message determined by the message interpretation engine; a communication delivery engine having a fourth artificial intelligence model component configured and trained to automatically create a communication to the target player that conveys the meaning of the message as determined by the message interpretation engine if the message is determined to be relevant by the message relevance assessment engine; an output processor configured to deliver the communication created by the communication delivery engine to the target player; A system having

2. the input data includes one or more of a video of a person performing a gesture and tracking metadata associated with the person performing the gesture, and the message for communication to the target player is provided at least in part by the gesture. The system of claim 1 .

3. the tracking metadata defines one or more of a pose of the person, a movement of the person, a position of a controller device, an orientation of the controller device, a movement of the controller device, a position of a wearable device worn by the person, an orientation of the wearable device worn by the person, and a movement of the wearable device worn by the person; The system of claim 2.

4. the input data includes one or more of an audio clip and a video game controller input, and the message for communication to the target player is provided at least in part by one or more of the audio clip and the video game controller input. The system of claim 2.

5. the portion of the input data defining the message is identified as a pixel region within each of a number of video frames of the video of the person performing the gesture; The system of claim 2.

6. the second artificial intelligence model component is configured and trained to determine the meaning of the message through identification of the body part used to perform the gesture and through identification of one or more of a position of the body part and a manner of movement of the body part; The system of claim 2.

7. the second artificial intelligence model component is configured and trained to determine the meaning of the message before the person finishes making the gesture. The system of claim 2.

8. the third artificial intelligence model component is configured and trained to evaluate the meaning of the message to identify a subject of the message and an action associated with the subject of the message, the third artificial intelligence model component is configured and trained to consider the message relevant if the subject of the message is still present within the current game context of the target player and the action associated with the subject of the message is still determined to be relevant to the current game state of the target player; The system of claim 1 .

9. the third artificial intelligence model component is configured and trained to cancel further processing of the message by the system if it is determined that the subject of the message is no longer present within the current game context of the target player or if it is determined that the action associated with the subject of the message is no longer relevant to the current game state of the target player; The system of claim 8.

10. the third artificial intelligence model component is configured and trained to cancel further processing of the message by the system if it is determined that the action related to the subject of the message is not beneficial for improving the current game state of the target player; The system of claim 8.

11. the third artificial intelligence model component is configured and trained to cancel further processing of the message by the system if the subject of the message and the action related to the subject of the message are determined to be redundant with another communication already communicated to the target player; The system of claim 8.

12. the fourth artificial intelligence model component is configured and trained to automatically create a unified communication to the target player that conveys a similar meaning to the plurality of messages determined by the message interpretation engine, instead of automatically creating a plurality of communications to the target player for the plurality of messages; The system of claim 1 .

13. the fourth artificial intelligence model component is configured and trained to automatically recognize a higher urgency level for communicating the communication to the target player, and the fourth artificial intelligence model component is configured and trained to automatically create an abbreviated communication to the target player conveying a shorter version of the message in response to recognizing the higher urgency level for communicating the communication to the target player. The system of claim 1 .

14. the fourth artificial intelligence model component is configured and trained to automatically determine a format of the communication to the target player, the format being one or more of a text format, an audible format, a graphical format, and a haptic format; The system of claim 1 .

15. 1. A method for artificial intelligence assisted communication within a video game, comprising: receiving input data including a message for communication to a target player of the video game; automatically identifying, through execution of a first artificial intelligence model component, portions of said input data that define said message; automatically determining a meaning of the message defined by the portion of the input data through execution of a second artificial intelligence model component; automatically determining, based on the meaning of the message, whether the message is relevant to the target player's current game state and game context through execution of a third artificial intelligence model component; if the message is determined to be relevant to the current game state and the game context of the target player, automatically creating a communication to the target player through execution of a fourth artificial intelligence model component that conveys the meaning of the message; Delivering the communication to the target player; method.

16. the input data includes one or more of a video of a person performing a gesture and tracking metadata associated with the person performing the gesture, and the message for communication to the target player is provided at least in part by the gesture. The method of claim 15.

17. the tracking metadata defines one or more of a pose of the person, a movement of the person, a position of a controller device, an orientation of the controller device, a movement of the controller device, a position of a wearable device worn by the person, an orientation of the wearable device worn by the person, and a movement of the wearable device worn by the person; 17. The method of claim 16.

18. the input data includes one or more of an audio clip and a video game controller input, and the message for communication to the target player is provided at least in part by one or more of the audio clip and the video game controller input.

17. The method of claim 16.

19. the portion of the input data defining the message is identified as a pixel region within each of a number of video frames of the video of the person performing the gesture; 17. The method of claim 16.

20. Executing the second artificial intelligence model component to automatically identify a body part used to perform the gesture, automatically identify one or more of a positioning of the body part, and a manner of movement of the body part; Executing the second artificial intelligence model component to analyze the body part, the positioning of the body part, and the manner of movement of the body part to determine the meaning of the message; Further comprising:

17. The method of claim 16.

21. executing the second artificial intelligence model component to determine the meaning of the message before the person finishes performing the gesture.

17. The method of claim 16.

22. executing the third artificial intelligence model component to evaluate the meaning of the message and identify a subject of the message and an action associated with the subject of the message; executing the third artificial intelligence model component to determine that the subject of the message is still present within the current game context of the target player and that the action related to the subject of the message is still relevant to the current game state of the target player; Further comprising: The method of claim 15.

23. and executing the third artificial intelligence model component to cancel further processing of the message if it is determined that the subject of the message is no longer present within the current game context of the target player or if it is determined that the action associated with the subject of the message is no longer relevant to the current game state of the target player.

23. The method of claim 22.

24. and canceling further processing of the message if the third artificial intelligence model component executes the third artificial intelligence model component to determine that the action associated with the subject of the message is not beneficial for improving the current game state of the target player.

23. The method of claim 22.

25. and executing the third artificial intelligence model component to cancel further processing of the message if it is determined that the subject of the message and the action associated with the subject of the message are redundant with another communication already communicated to the target player.

23. The method of claim 22.

26. executing the fourth artificial intelligence model component to automatically create a unified communication to the target player that conveys a similar meaning to the plurality of messages determined by the message interpretation engine, instead of automatically creating a plurality of communications to the target player for the plurality of messages; The method of claim 15.

27. further executing the fourth artificial intelligence model component to automatically recognize a higher urgency level for delivering the communication to the target player; executing the fourth artificial intelligence model component to automatically create an abbreviated communication to the target player conveying a shorter version of the message in response to recognizing the higher urgency level for conveying the communication to the target player; The method of claim 15.

28. and executing the fourth artificial intelligence model component to automatically determine a format of the communication to the target player, the format being one or more of a text format, an audible format, a graphical format, and a haptic format. The method of claim 15.

Citation Information

Patent Citations

  • Methods and apparatus for unsupervised machine learning for classification of gestures and estimation of applied forces

    CN113557069A

  • Program, information storage medium and game system

    JP2011212296A

  • Computer program and game device

    JP2013135761A

  • Distribution server, distribution system, distribution method, and program

    JP2020031903A

  • Program, method, and information processing device

    JP2020065656A