Machine Learning-Based Gaming Platform Messaging Risk Management Using Gamer Behavior

Abstract

A machine learning (ML) model (200) is used to identify spam messages and spammers in a computer game setting based on learned gamer behavior. An evaluator module (500, 502) receives a score(s) from the ML model indicating whether a particular message is spam or other undesirable, and if the score meets a threshold, passes the message to a punishment module (412) to determine whether to warn, suspend, or ban the sending account.

Description

[Technical field]

[0001] This application relates generally to machine learning based gaming platform messaging risk management. [Background technology]

[0002] Computer simulations, such as computer games, may allow messaging between players and / or spectators for added entertainment. Summary of the Invention

[0003] As understood herein, such messaging poses the risk that players will be inundated with spam and / or messages from spam bots, reducing the enjoyment of the game, especially online games, and users will need to report such messages and have another human analyze them to determine the appropriate course of action.

[0004] As will also be appreciated herein, computer games, such as online games, offer a unique opportunity to detect spam and spammers (human or bot) that other computer-based activities do not provide. In particular, significant details of player (gamer) behavior are recorded that typically do not accompany other computer-based activities, and this game-specific behavioral data can be advantageously used to reduce spam in computer games.

[0005] Thus, the present principles use data, specifically attributes available only on the gaming platform across all games. The unique in-game behavior patterns of players within the gaming platform are also used. Multiple machine learning systems automate the risk moderation process. Prediction systems and decision engines specific to the unique needs of the gaming platform are provided. All messages sent during game play may be analyzed, not just messages that users may report as potentially problematic.

[0006] In one aspect, the system includes at least one computer medium including instructions executable by at least one processor to receive from at least a first machine learning (ML) model at least one indication of whether a first message sent from a first account during play of the computer simulation is of a predetermined type rather than a transient signal. The instructions are executable to receive the indication, determine an action, and send a signal to at least one action module to implement the action against the first account. The first ML model is configured to receive as input information related to user behavior within the computer simulation.

[0007] The predetermined type may be one or more of spam, abusive language, and inappropriate language.

[0008] Metrics related to user behavior may include metrics of gaming behavior, such as one or more of games played, time played, gameplay history, number of trophies earned, in-game activities, and success rate of in-game activities.

[0009] Additionally or alternatively, metrics related to user behavior may include one or more of user messaging behaviors, such as frequency of message sending, number of different accounts to which messages were sent, regions of accounts to which messages were sent, and message type.

[0010] Additionally or alternatively, metrics related to user behavior may include one or more of the following: number of games owned, number of paid games purchased.

[0011] In another aspect, a method includes using a machine learning (ML) model to process data associated with the messaging to represent user behavior associated with a computer simulation, and using an output of the ML model to determine whether to apply a corrective action to an account associated with the messaging.

[0012] In another aspect, the device includes at least one processor configured to send information relating to all messages sent within the computer simulation by a player account to a machine learning (ML) model. The information provided to the ML model represents user behavior within the computer simulation. The processor is configured to selectively apply corrective actions to the player account using an output of the ML model.

[0013] The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts and in which: [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram of an exemplary system including an embodiment consistent with the present principles; [Diagram 2] Illustrates an exemplary machine learning (ML) risk management architecture at a high level. [Diagram 3] 1 illustrates an exemplary ML risk management information flow. [Figure 4] 1 illustrates further details of an exemplary ML risk management information flow. [Diagram 5] 1 illustrates further details of an exemplary ML risk management information flow. [Figure 6]1 illustrates an exemplary sequence of messaging when the risk level of a message is below a threshold. [Figure 7] 1 illustrates an exemplary sequence of messaging when the risk level of a message exceeds a threshold. [Figure 8] 1 illustrates, in exemplary flow chart form, exemplary logic for training an ML model using gaming behavior. [Figure 9] 1 illustrates, in exemplary flow chart form, exemplary logic for training an ML model using message content. [Figure 10] 1 illustrates, in exemplary flow chart form, exemplary logic for training a composite ML model. [Figure 11] 1 illustrates, in exemplary flow chart form, exemplary logic for using one or more ML models for message risk management consistent with the present principles. [Figure 12] 1 illustrates, in exemplary flow chart form, alternative exemplary logic for using one or more ML models for message risk management consistent with present principles. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The present disclosure generally relates to computer ecosystems, including aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks. The systems herein may include server and client components that may be connected via a network such that data may be exchanged between the client and server components. The client components may include one or more computing devices, including gaming consoles, such as Sony PlayStation® or Microsoft, or Nintendo, or other manufacturer's gaming consoles, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart televisions, Internet-enabled televisions), portable computers, such as laptops and tablet computers, other mobile devices, including smartphones, as well as additional examples described below. These client devices may operate in a variety of operating environments. For example, some client computers may employ, by way of example, a Linux operating system, a Microsoft operating system, or a Unix operating system, or an Apple or Google operating system. These operating environments may be used to execute one or more browsing programs, such as browsers created by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by Internet servers described below. Additionally, an operating environment consistent with the present principles may be used to run one or more computer game programs.

[0016] The server and / or gateway may include one or more processors that execute instructions that configure the server to receive and transmit data over a network, such as the Internet. Alternatively, the clients and servers may be connected via a local intranet or a virtual private network. The server or controller may be exemplified by a game console, such as a Sony PlayStation®, a personal computer, or the like.

[0017] Information may be exchanged between the clients and the servers over a network. For this purpose and for security, the servers and / or clients may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus that implements a method for providing a secure community, such as an online social website, to network members.

[0018] The processor may be a single-chip or multi-chip processor capable of implementing logic through various wiring such as address, data and control lines, as well as registers and shift registers.

[0019] Components included in one embodiment may be used in other embodiments in any suitable combination. For example, any of the various components described herein and / or illustrated in the drawings may be combined, interchanged, or excluded from other embodiments.

[0020] "A system having at least one of A, B, and C" (and similarly "a system having at least one of A, B, or C", "a system having at least one of A, B, C") includes systems having A only, B only, C only, A and B, A and C, B and C, and / or A, B, and C, etc.

[0021] Referring now specifically to FIG. 1, an exemplary system 10 is shown, which may include one or more of the exemplary devices described above and further below in accordance with the present principles. The first of the exemplary devices included in the system 10 is a consumer electronics (CE) device, such as an audio video device (AVD) 12, such as, but not limited to, an Internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). Alternatively, the AVD 12 may also be a computerized Internet-enabled ("smart") phone, a tablet computer, a notebook computer, an HMD, a wearable computerized device, a computerized Internet-enabled music player, a computerized Internet-enabled headphones, a computerized Internet-enabled implantable device such as an implantable skin device, and the like. It should be understood that the AVD 12, regardless of what it is, is configured to implement the present principles (e.g., to communicate with other CE devices to implement the present principles, to execute the logic described herein, and to perform any other functions and / or operations described herein).

[0022] Accordingly, to realize such principles, the AVD 12 may be established by some or all of the components shown in FIG. 1. For example, the AVD 12 may include one or more displays 14, which may be implemented by a high definition or ultra-high definition "4K" or higher flat screen and may be touch-enabled to receive user input signals via touch on the display. The AVD 12 may include one or more speakers 16 for outputting audio in accordance with the present principles, and at least one additional input device 18, such as a voice receiver / microphone, for inputting audible commands to the AVD 12 to control the AVD 12. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a WAN, a LAN, etc., under the control of one or more processors 24. Thus, the interface 20 may be a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12 to implement the present principles, including other elements of AVD 12 described herein, such as controlling display 14 to present images thereon and receiving input from display 14. It should further be noted that network interface 20 may be a wired or wireless modem or router, or a wireless telephone transceiver, or other suitable interface, such as the Wi-Fi transceiver mentioned above.

[0023] In addition to the above, AVD 12 may include one or more input and / or output ports 26, such as a High-Definition Multimedia Interface (HDMI®) port or a USB port for physically connecting to another CE device, and / or a headphone port for connecting headphones to AVD 12 so that audio from AVD 12 is presented to a user through the headphones. For example, input port 26 may be connected to a source 26a of audio-video content, either wired or wireless, cable or satellite. Thus, source 26a may be a separate or integrated set-top box, or a satellite receiver. Alternatively, source 26a may be a game console or disc player containing content. Source 26a when implemented as a game console may include some or all of the components described below in connection with CE device 48.

[0024] The AVD 12 may further include one or more computer memories 28, such as disk-based or solid-state storage that is not a transitory signal, possibly embodied within the AVD housing as a standalone device, or as a personal video recording device (PVR) or video disk player for playing AV programs either inside or outside the AVD housing, or as a removable memory medium or server as described below. In some embodiments, the AVD 12 may also include a location or position receiver, such as, but not limited to, a cellular receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or cellular base station and provide that information to the processor 24 and / or in conjunction with the processor 24 to determine the altitude at which the AVD 12 is disposed. The component 30 may also be implemented by an inertial measurement unit (IMU), typically including a combination of accelerometers, gyroscopes, and magnetometers, or by event-based sensors to determine the location and attitude of the AVD 12 in three dimensions.

[0025] Continuing with the description of the AVD 12, in some embodiments, the AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras such as webcams, event-based sensors, and / or cameras integrated into the AVD 12 and controllable by the processor 24 to collect pictures / images and / or video in accordance with present principles. The AVD 12 may also include a Bluetooth transceiver 34 and other NFC elements 36 for communicating with other devices using Bluetooth and / or Near Field Communication (NFC) technologies, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0026] Additionally, the AVD 12 may include one or more auxiliary sensors 38 (e.g., motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands)) that provide input to the processor 24. The AVD 12 may include an over-the-air (OTA) television broadcast port 40 for receiving OTA television broadcasts that provide input to the processor 24. In addition to the above, it is noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR Data Association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, and may also be a kinetic energy harvester that may convert kinetic energy into power to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may also be included. One or more haptic generators 47 may be provided to generate haptic signals that can be sensed by a person holding or touching the device.

[0027] Continuing to refer to FIG. 1, in addition to the AVD 12, the system 10 may include one or more other CE device types. In one embodiment, the first CE device 48 may be a computer game console that can be used to transmit audio and video of a computer game to the AVD 12 via commands sent directly to the AVD 12 and / or through a server, as described below, while the second CE device 50 may include similar components to the first CE device 48. In the illustrated example, the second CE device 50 may be configured as a computer game controller operated by a player or a head mounted display (HMD) worn by a player. In the illustrated example, only two CE devices are shown, but it should be understood that a fewer or greater number of devices may be used. The devices herein may implement some or all of the components shown for the AVD 12. Any of the components shown in the following figures may incorporate some or all of the components shown for the AVD 12.

[0028] Referring now to the at least one server 52 mentioned above, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56, such as disk-based or solid-state storage, and at least one network interface 58 that, under the control of the server processor 54, enables communication with other devices of Figure 1 via the network 22, and may indeed facilitate communication between the server and client devices in accordance with the present principles. It should be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as a wireless telephone transceiver.

[0029] Thus, in some embodiments, server 52 may be an entire Internet server or server "farm," and may include and perform "cloud" functions such that, for example, in an exemplary embodiment for a network gaming application, devices of system 10 may access the "cloud" environment via server 52. Alternatively, server 52 may be implemented by one or more game consoles or other computers in the same room or nearby as the other devices shown in FIG.

[0030] The components illustrated in the following figures may include some or all of the components illustrated in Figure 1. The user interfaces (UIs) described herein may be integrated, extended, and UI elements may be mixed and matched between UIs.

[0031] The present principles may employ a variety of machine learning models, including deep learning models. Machine learning models consistent with the present principles may use a variety of algorithms trained in ways including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms may be implemented by computer circuitry and include one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a type of RNN known as a long short-term memory (LSTM) network. Support vector machines (SVMs) and Bayesian networks may also be considered examples of machine learning models. In addition to the types of networks mentioned above, the models herein may be implemented by classifiers.

[0032] Thus, as understood herein, performing machine learning may include accessing training data and then training a model on the training data so that the model can process additional data and make inferences. Thus, an artificial neural network / artificial intelligence model trained through machine learning may include an input layer, an output layer, and multiple hidden layers in between that are configured and weighted to make inferences regarding the appropriate outputs.

[0033] Reference is now made to Figure 2. As an overview, Figure 2 illustrates two separate ML models used to leverage in-game data to determine whether a message is undesirable, e.g., unsolicited spam or from a human or bot spammer. All messages sent within a computer simulation, such as a computer game, are evaluated on the fly by the models in some example implementations, and appropriate action can be taken given the model's output. One model is a content-based model, which looks at the content of messages sent throughout the game platform, including message text, metadata, Uniform Resource Locator (URL) or Uniform Resource Identifier (URI), and characters. Another model is a "user model" that takes user behavior as input.

[0034] A client-facing message routing and decision (MD) engine 200 communicates with a content-based prediction service 202 and an account-based prediction service 204. The content-based prediction service 202 in turn loads a content-based natural language processing (NLP) machine learning (ML) model (or "content model" for short) 206, which may be updated daily, while the account-based prediction service 204 may communicate with a user behavior ML model (or "user model" for short) 208 through a user search database 210.

[0035] The content models 206 include one or more machine learning models based on the content of messages shared between players that can be applied to predict the likelihood that a message will fall into multiple risk moderation categories, such as spam, offensive or abusive content, child protection, hate speech, pornographic or obscene content, etc.

[0036] In contrast, the user model 208 includes one or more machine learning models based on user behavior on the gaming platform, including gaming activity, purchasing habits, and messaging behavior, which may be applied to predict the likelihood that a target user will behave in violation of community guidelines, for example, by spamming within the gaming platform.

[0037] The content-based ML model 206 may have access to a variety of data sources 212, including a general message log 214 that covers all messages sent within the simulation system, as well as a "grief" message log 216 that contains messages that have been flagged for review by a moderator and / or reviewed by a moderator.

[0038] Similarly, the user behavior ML model 208 may access data sources 212 , including accessing a user behavior log 218 and a list of banned user identifiers 220 .

[0039] In some embodiments, the MD engine 200 can receive information from a ML-based or rule-based composite model 222 that aggregates scores from other ML models described herein and outputs a single score on which action is taken consistent with the present principles. Additionally, Figure 2 shows that the content-based prediction service 202 can receive information from a Uniform Resource Identifier (URI) evaluation module 224 and an inappropriate text filter 226.

[0040] The system shown in Figure 2 can evaluate every message sent in a computer simulation system or computer game on the fly using data not available anywhere else: game data. As described in further detail below, this data includes user (also referred to herein as player or gamer) behavior, such as number of boss kills, game genre (some games may naturally have more vulgar language than others, so vulgar language is less of a target for enforcement), and other user behavior information specific to the simulation community and / or the simulation platform itself, including the amount of time a particular user who originated the message under test played the simulation, whether that user purchased the simulation, how often that user plays that particular simulation, etc., to determine whether a particular user is a human spammer or a bot spammer, as described herein. This basic information can vary from game to game, as behavior in one game may not be unusual in a different game.

[0041] To explain FIG. 2 in more detail, both the content-based model and the user-based model can be trained on four types of data sources. These include a regular message log 214 that stores and processes regular messages and communications sent between users, and a grief message log 216 that stores moderated messages and moderation results from our internal moderation team. The data sources shown in FIG. 2 also include user data 218 that stores a vast variety of user behavior data, including users' purchase history, gameplay history, in-game activity history, messaging history, and the like. These user data are unique to the gaming platform. For example, gameplay history may include which games a user played, for how long, and how many sessions, all recorded in user data 218. As another example, for in-game activity history, user data may record how successful a user was at a particular in-game activity, how often a user killed a boss, how often a user completed a task, how many trophies a user earned, and the user's estimated skill level.

[0042] The messaging history is also unique to the gaming platform, as users may adopt special language unique to their gaming community, and special abbreviations unique to the games they play. The patterns learned through machine learning are unique to the gaming platform.

[0043] FIG. 2 also shows a banned ID list that stores account IDs that have been banned in the past.

[0044] Both models can be trained on these data sources (after data processing and feature extraction). The trained models are then loaded into the respective online real-time services 202, 204 to perform predictions on whether a message is spam and / or whether the message sender is a spammer. When the consumer-facing MD engine 200, which may be implemented as an application programming interface (API), receives a message sent through an account ID, the message body and the account ID are sent to the content-based prediction service 202 and the account prediction service 204. The two services return a machine learning prediction score for the message body based on the content model and a machine learning prediction score for the account ID based on the user model.

[0045] In an embodiment, a composite model 222 may be provided to combine the predicted scores from the content model 206 and the user model 208 to predict a single actionable score for spam risk management. The score is used to implement appropriate moderation actions.

[0046] For an understanding of the ML workflow adopted in line with the current principles, please refer to Figure 3. A data extract, transform and load (ETL) module 300 reads data from the regular message log 214, the "grief" message log 216, and the user action log 218, and writes the combination of these logs to a pre-processed database 302.

[0047] The data logs contain unique, gaming platform specific data, and machine learning models can be built for each individual computer simulation, if desired, so that, for example, each online computer game in a gaming system can use its own unique ML model to predict spam and spammer accounts. It should be understood that any of the data listed below may be game specific, or, if desired, may not be game specific but instead represent a composite score aggregated across all games associated with a user account.

[0048] Data that may be included in user data 218 may include static user data such as wallet balance, age, gender, country, city, language, whether or not an account is active, email verification status, account registration status, and membership status. User data may also include the user's gaming behavior, optionally grouped by game, such as games played, time played, play history, number of trophies earned by an account, in-game activities, and activity success rates. User data may further include game specific information regarding the user's messaging behavior, frequency of message sending, number of different accounts to which messages were sent, regions of accounts to which messages were sent, and message types. User data may include commercial user behavior, such as number and titles of games owned, number and titles of paid game purchases.

[0049] Such data can be used to train models as to whether certain user behavior is risky and indicative of problematic accounts that require punishment, such as spammer accounts. For example, if an account sends messages at a high frequency, this may indicate a spambot and therefore a risk. If an account sends a large number of group messages that meet a threshold, a spambot may be indicated. If an account has never purchased a computer game and has acquired a large number of friends that meet a threshold, a spambot may be indicated. If an account sends group messages to a large number of recipients that meet a threshold, a spambot may be indicated.

[0050] Game-specific or composite (aggregated) user social data may also be stored in user data 218. Such data may include links to social profile pictures and friend status.

[0051] The messaging data contained in the regular and grief message logs 214, 216 may be game specific or compound and may include the message body, as well as the message type, length, and start time.

[0052] The pre-processed database 302 is read by a content feature extraction module 304 and a user feature extraction module 306. The content feature extraction model outputs feature vectors representing the data read from the pre-processed database 302 to an offline content feature vector database 308 and an online content feature vector database 309. The user feature extraction model 306 outputs feature vectors representing the data read from the pre-processed database 302 to an offline user feature database 310. The data is converted into a data format suitable for machine learning and designed into a data vector representation that can be consumed by a machine learning algorithm.

[0053] The feature vectors in the offline databases 308, 310 are read during the training phase by the content model in training 206A, the user model in training 208A, respectively, and also by the composite model in training 222A, if provided, which also reads the preprocessed database 302 as shown. Also as shown, in embodiments where there is currently no need to update the scores in real time for the user model, the user model in training 208A may also write information to the online user score search database 311, in which case the scores may be calculated for the user and cached in the online user score search database 311. The real-time user score service 204 retrieves the scores from the online user score search database 311 using the user ID as an input argument. It should be appreciated that the user features from the user feature extraction model 306 may also be sent directly to the online user score search database 311.

[0054] In FIG. 3, so far, it will be appreciated that feature engineering and model training involves reading input data and extracting patterns from that data (e.g., how often a user played a game recently, how well a user completed a task within the game) that are stored in respective offline databases 308, 310 and used to train models 206, 208, 222 on these patterns, with the goal of predicting how dangerous a target user and message is, e.g., the likelihood of it being spam, threatening other users, and not following our community guidelines. Note that training data may include ground truth, such as messages annotated by humans as spam, user behavior annotated by humans as possible spammer accounts, etc.

[0055] The models-in-training 206A, 208A, 222A can be written to the offline model store 312. In this step, three categories of models can be built from the respective models-in-training 206A, 208A, 222A for their specific tasks: content models 206 for rating message content as spam, user models 208 for rating user behavior patterns as spammer accounts, and composite models / services 222.

[0056] As shown in Figure 3, each of the content, user and composite prediction services 202, 204, 222 retrieves data from the model store 312 and the online content feature database 309 and the online user feature database 311. At this stage, the online prediction services 202, 204, 222 are constructed to return prediction scores based on user (account ID) and message (message body) inputs. All three services can retrieve the models trained as described above, retrieve the features and extracted patterns, and make predictions in real time.

[0057] 4 and 5 illustrate details of how the MD engine 200 shown in FIG. 2 uses ML to determine the likelihood of a particular message or user meeting "risk" criteria, such as whether a message is spam, sexually explicit, or overly inappropriate.

[0058] For example, an ML model trained as described above can be trained with appropriate data (including data described herein) to evaluate a message and the likelihood that it is a spam message. The ML model can be trained to evaluate the message and sender to determine if the user is a spambot. Once that is determined (a score between 0.0 and 1.0, with scores closer to 1.0 being more likely to meet the criteria), the score is then passed to one or more decision engines to determine what, if any, punishment should be based on the score and an inference engine.

[0059] 4, assume that a spammer account 400 sends a spam message to an in-game messaging app 402. The app 402 sends the message to a message server 404, which logs the message in a log 406, and to a content router 408, which may run one or more ML models described herein to determine if the message is spam. The content router 408 may return the results to the message server 404, which may then relay the results back to the app 402. The content router 408 may also send the evaluation results to the log 406 and / or a review queue 410 for human review, and may optionally send annotations for input into further training of the ML model being run.

[0060] The content router 408 can send a request to the punishment engine 412 to ban or otherwise punish the spammer account 400. The punishment engine 412 can write a message to the grief message log 216 that can be read by the ML models 206, 208 along with the user date from the user log 218. Note that the MD engine 200 can also communicate with the punishment engine 412.

[0061] The content router 408 can also send the message to the MD engine 200, which can send the message to a message queue 414 that can also be read by the ML models 206, 208. The content and user prediction services 202, 204 (and, if provided, the composite prediction engine 222) are accessed by the MD engine 200 and return a score representing the likelihood that the message is spam and / or that the account 400 is a spammer account based on the output of the prediction services.

[0062] FIG. 5 illustrates an embodiment in which the MD engine 200 and / or other components embody multiple decision engines, in the example shown, an account decision engine 500 that determines whether a score from the user ML model 208 indicates that a message sending account is risky, e.g., a spammer, a content decision engine 502 that determines whether a score from the content ML model 206 indicates that the message content is risky, e.g., spam, and a composite decision engine 504 that determines whether a single composite score representing the user and content scores indicates risk.

[0063] The decision engine can be automated and therefore does not require human interaction. The decision engine may be specialized to take into account a particular inference engine that is specific to the game's data and treatment. The decision engine is specific to the game's treatment and based on years of moderation experience, and may be extensible so that additional inference engines can be added without recoding. Furthermore, the decision engine may be programmable so that the penalties and matrices are programmable (e.g., using yaml) and can be updated and changed without modifying the engine.

[0064] The engine can also log all actions (because what to log for each action is programmable in YML) and these are retained for human evaluation at a later date in the event of review or appeal.

[0065] All decision engines are loaded at startup with their scores and actions associated with the scores. As scores arrive from the ML engines, they are collected until all have arrived or a timeout occurs. The scores are then compiled into a list and sent to a decision engine selector 506 which first goes through the list of scores and corresponding ML models and selects one to act on. The decision engine selector 506 then decides which decision engine 500, 502, 504 to send the scores to from a decision engine configuration database 507. Each decision engine thus corresponds to one or more ML models.

[0066] As will be further explained shortly, the selected decision engine then uses the score to initiate a script that will act based on the score, which may be to do nothing, issue a warning, temporarily suspend the account, permanently cancel the account, etc.

[0067] In Figure 5, group messages 508, friend requests 510, and recipient-specific messages 512 are sent from user accounts (clients) of the computer simulation to an analysis tool selector 514 of the content router 408. As shown, the selector 514 sends its input to block 516 of the MD engine 200, which receives ML models from a database 518 and sends input to the user and content prediction models 208, 206. The analysis tool selector 514 may also send its input to various other analysis tools, including a URI evaluator 224 to evaluate whether any URIs in the message are on a restricted list, an inappropriate text filter 226 to evaluate whether the message body contains inappropriate language or other objectionable speech, an image evaluator 520 to determine whether any images sent by the user contain objectionable material, and an anti-virus evaluator 522 to determine whether any messages contain a virus. Any messages that cause violations in the output from the analysis tools may be deleted and / or the account that sent the message may be penalized.

[0068] 5 shows that the ML models 208, 206 send their outputted numbers (e.g., between 0 and 1, with 1 indicating the most likely violation and 0 indicating the least likely) to a predicted response collection block 524 in the MD engine 200. The responses are aggregated in an aggregation block 526 of the MD engine 200, and the aggregate score is communicated from the aggregation block 526 to the decision engine selector 506.

[0069] 5, the selected decision engine block 500 (or 502 or 504) of the MD engine 200 can send the output of its decision to the punishment engine 412 to determine the appropriate punishment for the account sending the message flagged as risky by the decision engine. The decision engine block 500 can further send the decision to a log of messages and scores for human review at 530 or send it to the log 406 for recording.

[0070] It will be readily appreciated that actions and decisions may be configurable and based on game play.

[0071] 6 and 7 illustrate the message flows, in this case assuming messages that do not require punishment and messages that require punishment, respectively, when the ML model is trained to test for spam. It should be understood that the principles can be employed to test for other types of risk, such as abusive language, inappropriate language, etc.

[0072] 6, at 600, a message is sent from the message server 404 to the content router 408, as described above. At 602, the message is forwarded from the content router 408 to the MD engine 200, which returns the message to the server 404 through the content router 408, at 604.

[0073] The MD engine 200 may read 606 the context from the server 404 as needed and receive 608 the context.

[0074] Message loops 610 and 612 show that the MD engine 200 accesses the content model 202 and the user model 204, respectively, to obtain respective scores indicating whether the message content and user account, respectively, pose a spam risk therein.

[0075] Figure 6 assumes no risk (e.g., no spam). At 614, the MD engine 200 sends the message and score to a message queue 616. At 618, the scores from the ML models can be aggregated and a punishment action can be obtained at 620 from a decision table or matrix, as described above. Because Figure 6 does not assume spam, no action is taken other than writing the message and its risk score to the log 406 at step 622 and, optionally, writing the message and score to a review process 626 at 624 for use in further training of the ML models.

[0076] 7, at 700, a message is sent from the message server 404 to the content router 408, as described above. At 702, the message is forwarded from the content router 408 to the MD engine 200, which returns the message at 704 through the content router 408 to the server 404.

[0077] The MD engine 200 may read 706 and receive 708 the context from the server 404, as needed.

[0078] Message loops 710 and 712 show that the MD engine 200 accesses the content model 202 and the user model 204, respectively, to obtain respective scores indicating whether the message content and user account, respectively, now pose a spam risk.

[0079] FIG. 7 assumes the case of risk (e.g., spam). At 714, the MD engine 200 sends the message and score to the message queue 616. At 718, the scores from the ML model can be tallied, and at 720, the punishment action can be obtained from the decision table or matrix, as described above. Since FIG. 7 assumes spam, in addition to writing the message and its risk score to the log 406 at step 722, and optionally writing the message and score to the review process 626 at 724 for use in further training of the ML model, at 726, the MD engine 200 sends the score(s) indicative of risk to the punishment engine 412. As described elsewhere herein, the punishment engine then determines what punishment to impose based on the score(s), e.g., deleting the message so that it cannot be seen by the gamer, warning the sending account, temporarily suspending the sending account, or permanently banning the suspended account. The punishment is performed by any of the processes, components, and processors described herein.

[0080] With particular reference to the composite model 222 and its decision engine 504, it can accept inputs between 0 and 1 from the user and content models and output a composite score based thereon. The composite model may be a classifier or evaluator and may employ a matrix of combined scores adjusted for costs that reflect human judgments versus the cost of mistakes. The composite model outputs a number between 0 and 1 indicating the risk of the message, such as spam. Some types of risk, such as child abuse, may always be flagged as requiring punishment, while other types of risk, such as swearing, may not always be flagged depending on, for example, the context of a game. Thus, the matrix may apply weights to the composite score based on the associated risk of the underlying messages, with riskier types of messages triggering reporting to the punishment engine based on a lower score in the ML engine than types of messages that are less risky but still undesirable.

[0081] An example matrix is ​​shown below. Parameter Threshold(Spam) Threshold(Abusive) Threshold... User Game Behavior .5 .1 .2 Message Content .4 .1 .3 Composite .6 .1 .5

[0082] Thus, in the illustrated example, the thresholds are what the model predicted scores must meet (e.g., by meeting or exceeding) to trigger a corrective action message to the punishment engine. In the illustrated example, the first row of thresholds is used to determine whether a score from the user model 208 is worth reporting to the punishment engine, the second row of thresholds is used to determine whether a score from the content model 206 is worth reporting to the punishment engine, and the third row of thresholds is used to determine whether a score from the composite model 222 is worth reporting to the punishment engine. The first column of thresholds is associated with a first message type, in this example spam, the second column is associated with a second message type, in this case abusive language, and the subsequent columns of thresholds are associated with each subsequent message type, e.g., inappropriate language, sexual content, etc. The thresholds can be changed as frequently as necessary to reflect updates to risk / penalty assessment decisions.

[0083] Thus, it will be readily appreciated that for any particular message, the score threshold from the ML engine at which the message is reported may depend on both the type of message into which the message is grouped and the particular parameters associated with the message that are the subject of the ML engine prediction.

[0084] Thus, a first message of a first message type and a first numerical score output by a first ML engine may be reported to the punishment engine, while the same first message of the first message type and the same first numerical score but output by a second, different ML engine may not be reported to the punishment engine.

[0085] Similarly, a first message of a first message type and a first numerical score output by the first ML engine may be reported to the punishment engine, while a second message of a second message type and the same first numerical score output by the first ML engine may not be reported to the punishment engine.

[0086] Turning to the punishment engine 412, it too can be updated as frequently as necessary to reflect updated punishment assignments. In one embodiment, the lowest level of punishment may be sending a warning message to an account associated with a message or user behavior or content of a composite score that met the reporting threshold in the matrix above only once, while a higher number of reports may result in more severe punishments ranging from warnings to message deletion, temporary account suspension, to implementing a permanent account ban. For example, a warning may be sent to a violating account after a single violation, messages may be deleted after N violations, where N is an integer greater than 1, and temporary account suspension may be implemented after M violations, where M is an integer greater than N.

[0087] In another heuristic, for example, a warning may be sent to the violating account after the predicted score is equal to a threshold, the message may be deleted in response to the predicted score exceeding the threshold by a first amount, and a temporary suspension of the account may be implemented in response to the predicted score exceeding the threshold by a second amount greater than the first amount. Combinations of these heuristics may also be implemented. Also, different penalties may be imposed depending on the type of violating message.

[0088] Thus, a first message that exceeds a first threshold by a first amount may receive a first type of punishment, while a second message that exceeds the first threshold by a first amount may receive a second type of punishment. A first message that meets the first threshold a first number of times may receive a first type of punishment, while a second message that meets the second threshold a first number of times may receive a second type of punishment. A first message that meets the first threshold a first number of times may receive a first type of punishment, while a second message that meets the first threshold a second number of times may receive a second type of punishment.

[0089] 8-10 relate to the training of various ML models described herein. Beginning at block 800 of FIG. 8, feature vectors are extracted from data representing a user's gaming behavior. At block 810, the feature vectors may be annotated with classifications to generate a ground truth set of vectors. At block 820, the ground truth feature vectors are input into the user model 208 to train the model.

[0090] 9, starting at block 900, feature vectors are extracted from data representing message content. At block 910, the feature vectors may be annotated with classifications to generate a ground truth set of vectors. At block 920, the ground truth feature vectors are input into the content model 206 to train the model.

[0091] Beginning at block 1000 of Figure 10, the output numerical scores from the content model and the user model are received. At block 1010, the output scores may be annotated with classifications to generate a ground truth set, where pairs of received scores are annotated into a single overall composite score. At block 1020, the ground truth scores are input to the composite model 222 to train the model.

[0092] Figure 11 illustrates the principles discussed above in flow chart form for purposes of illustration of the principles. The logic represented by Figure 11 may employ machine learning techniques and components and be executed by any one or more of the processors described herein.

[0093] Beginning at block 1100, for at least some, and preferably each message transmitted within the computer simulation or system of computer simulations, the message may be input to a content model 206 at block 1102. At block 1104, the message is also input to a user model 208. Optionally, the outputs of the content model 206 and the user model 208 may be sent to a composite model 222 at block 1106 to provide a single score representing both the content and user behavior associated with the message under test.

[0094] Moving to block 1108, the score(s) are evaluated for whether the message and / or user behavior associated with the account from which the message originates indicates risk. If a single composite score is used, then that composite score may be the only score used in block 1108.

[0095] Moving to decision diamond 1110, it is determined whether the score(s) associated with the message indicates a risk, i.e., whether the message is of a first type that does not require corrective action, or of a second type (danger) that does require corrective action. If corrective action is not required, the message and / or user action associated therewith are recorded / logged in block 1112 and then input as training to the associated model in block 1114.

[0096] On the other hand, if corrective action is required, a request for corrective action is sent to the punishment engine 412 along with the message, score(s), and other relevant data in block 1116. The output of the punishment engine is received and implemented in block 1118. The message and / or user action associated therewith is recorded / logged in block 1120 and then input into the associated model as training in block 1122.

[0097] FIG. 12 illustrates a simplified embodiment in which only user actions unique to the particular computer simulation being monitored are used.

[0098] Beginning at block 1200, for at least some, and preferably for each message sent within the computer simulation or system of computer simulations, the message may be input to a user model 208 at block 1202. An output score from the user model 208 is received at block 1204.

[0099] Moving to decision diamond 1206, it is determined whether the score(s) associated with the message indicates a risk, i.e., whether the message is of a first type that does not require corrective action, or of a second type (danger) that does require corrective action. If corrective action is not required, the message and / or user action associated therewith are recorded / logged in block 1208 and then input into the associated model as training in block 1210.

[0100] On the other hand, if corrective action is required, a request for corrective action is sent to the punishment engine 412 along with the message, score(s), and other relevant data in block 1212. The output of the punishment engine is received and implemented in block 1214. The message and / or user action associated therewith is recorded / logged in block 1216 and then input into the associated model as training in block 1218.

[0101] Although particular embodiments have been shown and described in detail herein, it is understood that the subject matter encompassed by the present invention is limited only by the scope of the claims.

Claims

1. 1. A system comprising: at least one computer medium that is not a transitory signal and that includes instructions, the instructions being executed by at least one processor to: receiving at least one indication from at least a first machine learning (ML) model indicating whether a first message sent from a first account during play of the computer simulation is of a predetermined type; receiving the indication and sending a signal to at least one treatment module to determine a treatment; Implementing the action on the first account, wherein the first ML model is configured to receive as input information related to user behavior within the computer simulation.

2. The system of claim 1 , wherein the predetermined type is spam.

3. The system of claim 1 , wherein the predetermined type is a bad word.

4. The system of claim 1 , wherein the predetermined type is profanity.

5. The system of claim 1 , wherein the metrics related to user behavior include metrics of gaming behavior.

6. The system of claim 5 , wherein the gaming behavior includes games played.

7. The system of claim 5 , wherein the gaming behavior includes playing time.

8. The system of claim 5 , wherein the gaming behavior includes a game play history.

9. The system of claim 5 , wherein the gaming behavior includes a number of trophies won.

10. The system of claim 5 , wherein the gaming behavior comprises an activity within a game.

11. The system of claim 5 , wherein the gaming behavior comprises a success rate of an activity within a game.

12. The system of claim 1 , wherein the indicators related to user behavior include user messaging behavior.

13. The system of claim 1 , wherein the indicators related to user behavior include a frequency of sending messages.

14. The system of claim 1 , wherein the metrics related to user behavior include a number of different accounts to which messages were sent.

15. The system of claim 1 , wherein the indicators related to user behavior include a geography of an account to which a message was sent.

16. The system of claim 1 , wherein the indicators related to user behavior include message type.

17. The system of claim 1 , wherein the metrics related to user behavior include number of games owned.

18. The system of claim 1 , wherein the indicators related to user behavior include a number of paid game purchases.

19. 1. A method comprising: processing the data associated with the messaging and using a machine learning (ML) model to represent user behavior associated with the computer simulation; and using an output of the ML model to determine whether to apply a corrective action to an account associated with the messaging.

20. A device, comprising: At least one processor, the at least one processor comprising: sending information relating to all messages sent within a computer simulation by player accounts to a machine learning (ML) model, said information representing user behavior within said computer simulation; A device configured to use the output of the ML model to selectively apply corrective actions to the player account.

Images