Machine Learning-Based Gaming Platform Messaging Risk Management

Abstract

A machine learning (ML) model (200) is used to identify spam messages and spammers in a computer gaming environment based on learned gamer behavior. Message text may also be used. An evaluator module (500, 502) receives score(s) from the ML model(s) indicating whether a particular message is spam or not, and if the score(s) meet a threshold, passes the message to a punishment module (412) to determine whether to warn, suspend, or ban the sending account.

Description

[Technical Field]

[0001] This application relates generally to machine learning-based gaming platform messaging risk management. [Background technology]

[0002] Computer simulations, such as computer games, may allow messaging between players and / or spectators, adding to the enjoyment. Summary of the Invention

[0003] As understood herein, such messaging poses the risk of flooding players with spam and / or messages from spambots, reducing their enjoyment of the game, particularly online games. Users must report such messages, which must be analyzed by other humans to determine the appropriate action to take.

[0004] As will be further understood herein, computer games, such as online games, offer unique opportunities for detecting spam and spammers (human or bot) that other computer-based activities do not. In particular, important details of player (gamer) behavior that are not associated with other computer-based activities are typically recorded, and this game-specific behavioral data can be advantageously used to reduce spam in computer games.

[0005] Thus, the principles use data, particularly attributes, that span games and are only available on the gaming platform. They also use the unique in-game behavior patterns of players on the gaming platform. Multiple machine learning systems automate the risk management process. Prediction systems and decision engines are provided that are specific to the unique needs of the gaming platform. Any messages sent during gameplay can be analyzed, not just those that users may report as potentially problematic.

[0006] Thus, the system includes at least one computer medium that is not a transitory signal, the computer medium itself including instructions executable by at least one processor to receive, from at least a first machine learning (ML) model, at least one indication of whether a first message sent from a first account during play of a computer simulation is spam or not, and in response to the indication, to send a signal to at least one action module to determine an action and to perform the action on the first account.

[0007] In an example, the first ML model can include a content model configured to receive as input content from the first message, and the instructions can be executable to receive from at least the second ML model at least one indication of whether the first message sent from the first account during play of the computer simulation is spam. The instructions can be executable to signal at least one action module to determine an action and to implement the action against the first account in response to the indication from the second ML model satisfying a threshold. The second ML model can be configured to receive as input information related to user behavior in the computer simulation.

[0008] In some embodiments, the instructions can be further executable to receive, from at least the composite model, at least one indication of whether a first message sent from the first account during play of the computer simulation is spam. The instructions can be executable to signal at least one action module to determine an action and to implement the action against the first account in response to the indication from the composite model satisfying a threshold. The composite model can be configured to receive information from the first and second ML models as input.

[0009] In non-limiting embodiments, the action may include one or more of deleting the message, sending a warning to the first account, temporarily suspending the first account, or permanently banning the first account.

[0010] In some embodiments, the ML model includes a first ML model, the first message is of a first message type, the instructions from the first ML model include a first numerical score, and the instructions are executable to report the first message to a punishment engine and receive corrective action therefrom, and to not report to the punishment engine a second message of the first message type and the first numerical score output by a second, different ML engine.

[0011] In some embodiments, the ML model includes a first ML model, the first message is a first message type, the instructions from the first ML model include a first numerical score, and the instructions are executable to report the first message to a punishment engine and receive corrective action therefrom, and to not report to the punishment engine a second message of a second message type and the same first numerical score output by the first ML engine.

[0012] The action taken on the account may be established at least in part by one or more of the number of times the ML engine outputs the instruction and / or the magnitude of the score output by the ML engine.

[0013] In a non-limiting example, the instructions may be executable to, in response to the instruction exceeding a first threshold by a first amount, take a first action on a first account associated with the first message and take a second action on a second account associated with a second message associated with the instruction from the ML engine exceeding the first threshold by the first amount.

[0014] If desired, the instructions may be executable to, in response to instructions being returned a first number of times from the ML engine, perform a first action on a first account associated with the first message and perform a second action on a second account associated with a second message associated with a second number of instructions from the ML engine.

[0015] In another aspect, a method includes using a first machine learning (ML) model to process content from at least a first message sent from a first account in a computer simulation. The method also includes using a second ML model to process data representing user behavior associated with the first account. The method includes receiving outputs of the first and second ML models and selectively implementing a corrective action on the first account based at least in part on the outputs.

[0016] In another aspect, an assembly includes at least one processor programmed to receive at least a first message from a first account associated with a computer game. The processor is programmed to send content of the first message to a first machine learning (ML) model and receive first instructions from the first ML model representing processing of the content. The processor is also programmed to send data representing user behavior associated with the first account to a second ML model and receive second instructions from the second ML model representing processing of the data. At least in part in response to the first and second instructions, the processor is programmed to implement corrective action against the first account.

[0017] The processor is further programmed to receive at least a second message from a second account associated with the computer game. The processor is programmed to: send content of the second message to the first ML model; receive third instructions from the first ML model indicating processing of the content of the second message; send data representing user behavior associated with the second account to the second ML model; and receive fourth instructions from the second ML model indicating processing of the data representing user behavior associated with the second account. At least in part in response to the third and fourth instructions, the processor is programmed to not take corrective action against the second account.

[0018] Without limitation, the processor may be a component of a computer game console, a computer game controller, or a computer game server.

[0019] The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts and in which: [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a block diagram of an exemplary system including an example according to the present principles; [Figure 2] A high-level example of a machine learning (ML) risk management architecture. [Figure 3] Illustrates an example of ML risk management information flow. [Figure 4] 1 illustrates further details of an example ML risk management information flow. [Figure 5] 1 illustrates further details of an example ML risk management information flow. [Figure 6] 1 illustrates an example messaging sequence when the risk level of a message is below a threshold. [Figure 7] 1 illustrates an example messaging sequence when the risk level of a message exceeds a threshold. [Figure 8] Example logic for training an ML model using game behavior is illustrated in example flowchart form. [Figure 9] 1 illustrates example logic for training an ML model using message content in example flowchart form. [Figure 10] Example logic for training a composite ML model is illustrated in example flowchart form. [Figure 11] Example logic for using one or more ML models for message risk management consistent with the present principles is illustrated in example flowchart form. [Figure 12] An example of alternative logic for using one or more ML models for message risk management consistent with the present principles is illustrated in example flow chart form. DETAILED DESCRIPTION OF THE INVENTION

[0021] The present disclosure generally relates to computer ecosystems, including aspects of consumer electronics (CE) device networks (e.g., without limitation, computer gaming networks). Systems herein may include server and client components that may be connected via a network such that data may be exchanged between the client and server components. The client components may include one or more computing devices, such as game consoles such as Sony PlayStation®, or game consoles manufactured by Microsoft, Nintendo, or other manufacturers, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart TVs, internet-enabled TVs), portable computers such as laptops and tablet computers, and other mobile devices such as smartphones, as well as additional examples described below. These client devices may operate using a variety of operating environments. For example, some of the client computers may use, by way of example, the Linux® operating system, an operating system from Microsoft, or a Unix® operating system, or an operating system manufactured by Apple or Google. These operating environments may be used to run one or more browsing programs, such as browsers made by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by internet servers, as described below. An operating environment according to the present principles may also be used to run one or more computer game programs.

[0022] The server and / or gateway may include one or more processors that execute instructions that configure the server to send and receive data over a network such as the Internet. Alternatively, the clients and servers may be connected via a local intranet or a virtual private network. The server or controller may be instantiated by a game console, e.g., a Sony PlayStation®, a personal computer, etc.

[0023] Information may be exchanged between the client and the server over a network. For this purpose and for security, the server and / or client may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus that implements a method for providing a secure community, such as an online social website, to network members.

[0024] The processor may be a single-chip or multi-chip processor capable of implementing logic through various wiring such as address lines, data lines, and control lines, as well as registers and shift registers.

[0025] Components included in one embodiment may be used in other embodiments in any suitable combination, for example, any of the various components described herein and / or illustrated in the figures may be combined, interchanged, or excluded from other embodiments.

[0026] "A system having at least one of A, B, and C" (and similarly "a system having at least one of A, B, or C" and "a system having at least one of A, B, and C") includes systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.

[0027] Referring now specifically to FIG. 1 , an exemplary system 10 is shown. System 10 may include one or more of the exemplary devices described above and further below in accordance with the present principles. A first exemplary device included in system 10 is a consumer electronics (CE) device, e.g., an audio-video device (AVD) 12, such as, without limitation, an Internet-enabled TV with a TV tuner (as well as a set-top box that controls the TV). AVD 12 may also alternatively be a computerized Internet-enabled (“smart”) phone, a tablet computer, a notebook computer, an HMD, a wearable computerized device, a computerized Internet-enabled music player, a computerized Internet-enabled headphones, a computerized Internet-enabled implantable device, e.g., an implantable skin device, etc. Regardless of what it is, it should be understood that AVD 12 is configured to perform the present principles (e.g., communicate with other CE devices to perform the present principles, execute the logic described herein, and perform any other functions and / or operations described herein).

[0028] Accordingly, to implement such principles, the AVD 12 may be established by some or all of the components shown in FIG. 1 . For example, the AVD 12 may include one or more displays 14, which may be implemented by high-definition or ultra-high-definition “4K” or higher flat screens and may be touch-enabled to receive user input signals via touching on the display. The AVD 12 may also include one or more speakers 16 for outputting audio in accordance with the present principles and at least one additional input device 18, such as an audio receiver / microphone for inputting audible commands to the AVD 12 to control it. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a WAN, a LAN, etc., under the control of one or more processors 24. Thus, the interface 20 may be, without limitation, a Wi-Fi transceiver (which is an example of a wireless computer network interface, such as, without limitation, a mesh network transceiver). It should be understood that processor 24 controls AVD 12 (including the other elements of AVD 12 described herein) to carry out the present principles, e.g., controls display 14 to present images thereon and receives input therefrom. It should further be noted that network interface 20 may be a wired or wireless modem or router, or other suitable interface, e.g., a wireless telephone transceiver, or the Wi-Fi transceiver described above.

[0029] In addition to the foregoing, AVD 12 may also include one or more input and / or output ports 26 (e.g., a High-Definition Multimedia Interface (HDMI®) port or a USB port) and may physically connect to another CE device and / or a headphone port (for connecting headphones to AVD 12 and presenting audio from AVD 12 to the user via the headphones). For example, input port 26 may connect, wired or wirelessly, to a cable or satellite source 26a of audio-video content. Thus, source 26a may be a separate or integrated set-top box, or a satellite receiver. Or source 26a may be a game console or disc player containing content. If implemented as a game console, source 26a may include some or all of the components described below in connection with CE device 48.

[0030] AVD 12 may further include one or more computer memories 28, such as non-transitory disk-based or solid-state storage devices, possibly embodied within the AVD's chassis, as a standalone device, or as a personal video recording device (PVR) or video disc player either internal or external to the AVD's chassis for playing AV programs, or as a removable memory medium or server, as described below. In some embodiments, AVD 12 may also include a location or position receiver, such as, without limitation, a cellular receiver, a GPS receiver, and / or an altimeter 30 (configured to receive geographic location information from satellites or cellular towers and provide that information to processor 24 and / or determine the altitude at which AVD 12 is located with processor 24). Component 30 may also be implemented by an inertial measurement unit (IMU) (typically including a combination of accelerometers, gyroscopes, and magnetometers to determine the location and orientation of AVD 12 in three dimensions) or by an event-based sensor.

[0031] Continuing with the description of the AVD 12, in some embodiments, the AVD 12 may include one or more cameras 32 (which may be thermal imaging cameras, digital cameras, e.g., webcams, event-based sensors, and / or cameras integrated within the AVD 12 and controllable by the processor 24 to collect photos / images and / or video) in accordance with present principles. Also included on the AVD 12 may be a Bluetooth transceiver 34 and other NFC elements 36 for communicating with other devices using Bluetooth and / or near field communication (NFC) technology, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0032] Still further, the AVD 12 may include one or more auxiliary sensors 38 (e.g., motion sensors, e.g., accelerometers, gyroscopes, cyclometers, or magnetic sensors; infrared (IR) sensors; optical sensors; speed and / or cadence sensors; event-based sensors; gesture sensors (e.g., for detecting gesture commands)) to provide input to the processor 24. The AVD 12 may include an over-the-air TV broadcast port 40 for receiving over-the-air TV broadcasts and providing input to the processor 24. Note that in addition to the foregoing, the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, e.g., an IR data association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, and a kinetic energy harvester may be provided that can convert kinetic energy into electrical power to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field-programmable gate array 46 may be included. One or more haptic generators 47 may be provided to generate haptic signals that can be detected by a person holding or interacting with the device.

[0033] Continuing with reference to FIG. 1 , in addition to the AVD 12, the system 10 may include one or more other CE device types. In one example, the first CE device 48 may be a computer game console (which may be used to transmit computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or via a server, as described below), while the second CE device 50 may include similar components to the first CE device 48. In the illustrated example, the second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by the player. While the illustrated example shows only two CE devices, it should be understood that fewer or more devices may be used. The devices herein may implement some or all of the components shown for the AVD 12. Any of the components shown in the following figures may incorporate some or all of the components shown for the AVD 12.

[0034] Referring now to the aforementioned at least one server 52, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56 (e.g., disk-based storage or solid-state storage), and at least one network interface 58 (which, under the control of the server processor 54, enables communication with other devices of FIG. 1 via network 22 and may actually facilitate communication between the server and client devices in accordance with the present principles). It should be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (e.g., a wireless telephone transceiver, etc.).

[0035] Thus, in some embodiments, server 52 may be an entire Internet server or server "farm" and may include and perform "cloud" functionality, such that, for example, in an example embodiment for a network gaming application, devices of system 10 may access the "cloud" environment via server 52. Alternatively, server 52 may be implemented by one or more game consoles or other computers in the same room or nearby as the other devices shown in FIG. 1 .

[0036] The components shown in the following figures may include some or all of the components shown in Figure 1. The user interfaces (UIs) described herein may be integrated and extended, and UI elements may be mixed and matched between UIs.

[0037] The present principles may use various machine learning models, e.g., deep learning models. Machine learning models consistent with the present principles may use various algorithms trained using methods including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms, which may be implemented by computer circuitry, include one or more neural networks, e.g., convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a type of RNN known as a long short-term memory (LSTM) network. Support vector machines (SVMs) and Bayesian networks may also be considered examples of machine learning models. In addition to the types of networks mentioned above, the models herein may be implemented by classifiers.

[0038] Thus, as understood herein, performing machine learning may involve accessing training data and then training a model on the training data so that the model can process and infer additional data. Thus, an artificial neural network / artificial intelligence model trained through machine learning may include an input layer, an output layer, and multiple hidden layers therebetween. The hidden layers are configured and weighted to make inferences regarding appropriate outputs.

[0039] Reference is now made to Figure 2. At a high level, Figure 2 illustrates two separate ML models that utilize in-game data to determine whether a message is unwanted, e.g., annoying spam, or whether it is from a human or bot spammer. Every message sent in a computer simulation, e.g., a computer game, may, in some embodiments, be evaluated on the fly by a model, and appropriate action may be taken in response to the model output. One model is a content-based model that looks at the content of messages sent throughout the game platform, such as message text, metadata, Uniform Resource Locators (URLs) or Uniform Resource Identifiers (URIs), and characters. Another model is a "user model" that takes user behavior as input.

[0040] A client-facing message routing and decision (MD) engine 200 communicates with a content-based prediction service 202 and an account-based prediction service 204. As a result, the content-based prediction service 202 loads a content-based natural language processing (NLP) machine learning (ML) model (or "content model" for short) 206, which can be updated daily. Meanwhile, the account-based prediction service 204 can communicate with a user behavior ML model (or "user model" for short) 208 via a user search database 210.

[0041] The content models 206 include one or more machine learning models based on the content of messages shared between players that can be applied to predict the likelihood of a message falling into multiple risk management categories (e.g., spam, offensive or abusive content, child protection, hate speech, pornographic or obscene content, etc.).

[0042] In contrast, the user model 208 includes one or more machine learning models based on user behavior on the gaming platform, such as gaming activity, purchasing habits, and messaging behavior, which can be applied to predict the likelihood that a target user will behave in a manner that violates community guidelines, for example, by engaging in spamming on the gaming platform.

[0043] The content-based ML model 206 may access various data sources 212, such as a regular message log 214 covering all messages sent within the simulation system, as well as a "griefed" message log 216 containing messages flagged for review by a moderator and / or reviewed by a moderator.

[0044] Similarly, the user behavior ML model 208 may have access to data sources 212, such as a user behavior log 218 and a list of banned user identities 220.

[0045] In some embodiments, the MD engine 200 may receive information from an ML-based or rule-based composite model 222 that aggregates scores from other ML models described herein and outputs a single score upon which action is taken consistent with the present principles. Additionally, Figure 2 shows that the content-based prediction service 202 may receive information from a uniform resource identifier (URI) evaluation module 224 and a text profanity filter 226.

[0046] The system shown in FIG. 2 can evaluate every message sent within a computer simulation system or computer game on the fly using data not available anywhere else (i.e., game data). As described in further detail below, this data can include user (also referred to herein as player or gamer) behavior, such as number of boss kills, game genre (because some games may naturally involve more vulgar language than others and therefore vulgar language is less actionable in such games), and other user behavior information specific to the simulation community and / or the simulation platform itself (e.g., how long a particular user originating a tested message played the simulation in which the message was sent, whether the user purchased the simulation, how often the user plays a particular simulation, etc.), all of which may be used as described herein to determine whether a particular user is a human or bot spammer. This basic information can vary from game to game, since behavior in one game may not be unusual in another game.

[0047] Explaining FIG. 2 in more detail, both the user-based model and the content-based model may be trained by four types of data sources. These include a regular message log 214 that stores and processes regular messages and communications sent between users, and a griefed message log 216 that stores content-moderated messages and their content moderation results from an internal content moderation team. The data sources shown in FIG. 2 also include user data 218 that stores a variety of user behavior data (e.g., a user's purchase history, gameplay history, in-game activity history, messaging history, etc.). User data is specific to the game platform. For example, in gameplay history, which games a user played, for how long, and in how many sessions may all be recorded in user data 218. As another example, in the case of in-game activity history, user data may record how successful a user was at a particular in-game activity, how often they killed bosses, completed tasks, how many trophies they earned, and their estimated skill level.

[0048] Messaging history is also specific to the gaming platform because users may use special language specific to their gaming community and special abbreviations specific to the games they play. The patterns learned through machine learning are specific to the gaming platform.

[0049] FIG. 2 also shows a banned ID list that stores historically banned account IDs.

[0050] Both models can be trained on these data sources (after data processing and feature extraction). The trained models are then loaded into the respective online real-time services 202, 204 to perform predictions on whether a message is spam and / or whether a message sender is a spammer. When the consumer-facing MD engine 200 (which may be implemented as an application programming interface (API)) receives a message sent through an account ID, the message body and the account ID are sent to the content-based prediction service 202 and the account prediction service 204. The two services return a machine learning prediction score for the message body based on the content model and a machine learning prediction score for the account ID based on the user model.

[0051] In an example, a composite model 222 may be provided to combine predicted scores from the content model 206 and the user model 208 to predict one single actionable score for spam risk management, using the score to take appropriate content management actions.

[0052] To understand the ML workflow used consistent with the present principles, reference is now made to Figure 3. A data extract, transform, and load (ETL) module 300 reads data from regular message logs 214, "griefed" message logs 216, and user behavior logs 218, and writes the combination of these logs to a preprocessed database 302.

[0053] The data logs may optionally include unique, gaming platform-specific data for building machine learning models for each individual computer simulation, such that each online computer game in a gaming system may, for example, use its own unique ML model to predict spam and spammer accounts. It should be understood that any of the data listed below may be game-specific or, if desired, non-game-specific and may instead represent a composite score aggregated across all games associated with a user account.

[0054] Data that may be included in user data 218 may include, for example, static user data, such as wallet balance, age, gender, country, city, language, whether the account is active, email verification status, registration status, and membership status. User data may also include the user's gaming behavior (broken down by game, if necessary), such as games played, play time, play history, number of trophies earned by the account, in-game activities, and activity success rate. User data may further include game-specific information regarding the user's messaging behavior, frequency of sending messages, number of different accounts messaged, regions of accounts messaged, and message types. User data may also include the user's commercial behavior, such as number and titles of games owned, number and titles of non-free game purchases.

[0055] Such data may be used to train a model as to whether certain user behavior is risky and indicative of a problematic account, e.g., a spammer account, that requires punishment. For example, if an account sends messages frequently, this may indicate a spambot, and therefore a risk. If an account sends a number of group messages that meet a threshold, this may indicate a spambot. If an account has never purchased a computer game but has gained a number of friends that meet a threshold, this may indicate a spambot. If an account sends group messages to a number of recipients that meet a threshold, this may indicate a spambot.

[0056] Game-specific or composite (aggregated) user social data may also be stored in user data 218. Such data may include links to social profile photos and relationship status.

[0057] The messaging data contained in the regular and griefed message logs 214, 216 may be game-specific or compound and may include the message body, as well as the message type, length, and start time.

[0058] The preprocessed database 302 is read by a content feature extraction module 304 and a user feature extraction module 306. The content feature extraction model outputs feature vectors representing the data read from the preprocessed database 302 to an offline content feature vector database 308 and an online content feature vector database 309. The user feature extraction model 306 outputs feature vectors representing the data read from the preprocessed database 302 to an offline user feature database 310. The data is converted into a data format suitable for machine learning and designed into a data vector representation that can be digested by a machine learning algorithm.

[0059] The feature vectors in the offline databases 308, 310 are read during the training phase by the content model in training 206A, the user model in training 208A, and, if provided, the composite model in training 222A, respectively. The composite model in training 222A also reads the preprocessed database 302, as shown. Also, as shown, the user model in training 208A may write information to the online user score retrieval database 311 in embodiments where real-time updates to the current score for the user model are not required; in this case, scores may be calculated for the user and cached in the online user score retrieval database 311. The real-time user score service 204 reads scores from the online user score retrieval database 311 using the user ID as an input argument. It should be understood that user features from the user feature extraction model 306 may also be sent directly to the online user score retrieval database 311.

[0060] 3, it is understood that feature engineering and model training involves reading input data, extracting patterns from the data (e.g., how often a user has recently played a game, how successful the user was in completing tasks within the game), storing these patterns in respective offline databases 308, 310, and using these patterns to train models 206, 208, 222 to predict how likely target users and messages are to be risky, e.g., how likely they are to be spam, how likely they are to pose a threat to other users, and how likely they are to not follow community guidelines. Note that training data may include ground truth, e.g., messages annotated by humans as spam, user behavior annotated by humans as potential spammer accounts, etc.

[0061] The models-in-training 206A, 208A, 222A may be written to the offline model store 312. In this step, three categories of models may be built from each model-in-training 206A, 208A, 222A for their specific tasks: content models 206 that evaluate message content as spam, user models 208 that evaluate user behavior patterns as spammer accounts, and composite models / services 222.

[0062] As shown in Figure 3, each content, user, and composite prediction service 202, 204, 222 reads data from the model store 312, the online content feature database 309, and the online user feature database 311. At this stage, the online prediction services 202, 204, 222 are configured to return prediction scores based on user (account ID) and message (message body) input. All three services may load the trained models as described above, read the features and extracted patterns, and make real-time predictions.

[0063] 4 and 5 provide further details on how the MD engine 200 shown in FIG. 2 uses ML to determine the probability that a particular message or user meets "risk" criteria (e.g., whether the message is spam, sexually explicit, or gratuitously profane).

[0064] For example, an ML model trained as described above may be trained on appropriate data (including data defined herein) to evaluate messages and their likelihood of being spam messages. The ML model may also be trained to evaluate messages and senders to determine whether a user is a spambot. Once determined (as a score between 0.0 and 1.0, with scores closer to 1.0 indicating a higher probability of meeting the criteria), the score is passed to one or more decision engines, which determine what, if any, punishment should be imposed based on the score and an inference engine.

[0065] 4, assume that a spammer account 400 sends a spam message to an in-game messaging app 402. The app 402 sends the message to a message server 404. The message server 404 logs the message in a log 406 and sends it to a content router 408. The content router 408 may run one or more ML models described herein to determine whether the message is spam. The content router 408 may return the results to the message server 404, which may then relay the results back to the app 402. The content router 408 may also send the evaluation results to the log 406 and / or a review queue 410 for human review and, if necessary, annotation for input into further training of a running ML model.

[0066] The content router 408 can send a request to a punishment engine 412 to ban or otherwise punish the spammer account 400. The punishment engine 412 can write the message to a griefed message log 216, which can be read by the ML models 206, 208 along with the user date from a user log 218. Note that the MD engine 200 can communicate with the punishment engine 412.

[0067] The content router 408 can also send messages to the MD engine 200. The MD engine 200 can send messages to a message queue 414, which may be read by the ML models 206, 208. The content and user prediction services 202, 204 (and, if provided, a composite prediction engine 222) are accessed by the MD engine 200 and return a score representing the likelihood that the message is spam and / or that the account 400 is a spammer account based on the output of the prediction services.

[0068] FIG. 5 shows an example in which the MD engine 200 and / or other components embody multiple decision engines. In the illustrated example, there is an account decision engine 500 that determines whether the score from the user ML model 208 indicates that the message sending account is risky (e.g., a spammer), a content decision engine 502 that determines whether the score from the content ML model 206 indicates that the message content is risky (e.g., spam), and a composite decision engine 504 that determines whether a single composite score representing the user and content scores indicates risk.

[0069] The decision engine can be automated so that no human interaction is required. The decision engine can be specific, taking into account a particular reasoning engine (and being specific to the game data and actions). The decision engine can be specific to gaming actions, based on years of content management experience, and can be extensible so that additional reasoning engines can be added without recoding. Additionally, the decision engine can be programmable, allowing penalties and matrices to be programmable (e.g., using YAML) and updated and changed without modifying the engine.

[0070] The engine can also record all actions (because recording for each action is programmable in YAML) and set these aside for later human evaluation in case of review or appeal of the decision.

[0071] All decision engines are loaded with scores and score-related actions at startup. As scores arrive from the ML engines, they are collected until they are all arrived or a timeout occurs. The scores are then consolidated into a list and sent to a decision engine selector 506. The decision engine selector 506 first looks through the list of scores and corresponding ML models and selects one to operate on. The decision engine selector 506 then determines which decision engine 500, 502, 504 to send the scores to from a decision engine configuration database 507. Thus, each decision engine corresponds to one or more ML models.

[0072] As will be further described shortly, the selected decision engine then uses the score and initiates a script that executes based on the score, which could range from doing nothing, issuing a warning, suspending the account, to permanently canceling the account.

[0073] In FIG. 5 , group messages 508, friend requests 510, and recipient-specific messages 512 are sent from computer-simulated user accounts (clients) to an analysis tool selector 514 in the content router 408. The selector 514 sends its input to a block 516 in the MD engine 200. The block 516 receives ML models from a database 518 and sends input to the user and content prediction models 208, 206, as shown. The analysis tool selector 514 may also send its input to various other analysis tools, such as a URI evaluator 224 that evaluates whether any URIs in a message are on a restricted list; a text profanity filter 226 that evaluates whether the message body contains profanity or other objectionable language; an image evaluator 520 that determines whether any images sent by a user contain objectionable material; and an antivirus evaluator 522 that determines whether any message contains a virus. Any messages that trigger a violation output from the analysis tools may be removed, and / or the account sending the message may be disciplined.

[0074] 5 shows that the ML models 208, 206 send their output numerical values ​​(e.g., zero to one, with one indicating the highest probability of violation and zero indicating the lowest) to a predicted response collection block 524 within the MD engine 200. The responses are aggregated in an aggregation block 526 of the MD engine 200, and the aggregate score is communicated from the aggregation block 526 to the decision engine selector 506.

[0075] 5, a selected decision engine block 500 (or 502 or 504) of the MD engine 200 can send its decision output to a punishment engine 412 to determine an appropriate punishment for the account sending the message flagged as risky by the decision engine. The decision engine block 500 may then send its decision to a message and score log for human review at 530, as well as to the log 406 for recording.

[0076] It can be readily appreciated that actions and decisions can be configurable and based on gameplay.

[0077] 6 and 7 illustrate example message flows assuming messages that do not require punishment and messages that do require punishment, respectively (in this case, the ML model is trained to test for spam). It should be understood that the principles may also be used to test for other types of risk (e.g., abusive, profane, etc.).

[0078] 6, at 600, a message is sent from the message server 404 to the content router 408, as previously described. At 602, the message is forwarded from the content router 408 to the MD engine 200. At 604, the MD engine 200 returns the message to the server 404 through the content router 408.

[0079] The MD engine 200 may read 606 and receive 608 the context from the server 404 as needed.

[0080] Message loops 610 and 612 show that the MD engine 200 accesses the content model 202 and the user model 204, respectively, to obtain their respective scores indicating whether the message content and user account, respectively, pose a risk of spam, in this case.

[0081] Figure 6 assumes there is no risk (e.g., no spam). At 614, the MD engine 200 sends the message and score to a message queue 616. At 618, the scores from the ML model may be aggregated, and at 620, a punishment action may be derived from a decision table or matrix as previously described. Because Figure 6 assumes there is no spam, no action is taken other than writing the message and its risk score to the log 406 at step 622, and optionally writing the message and score at 624 to a review process 626 for use in further training of the ML model.

[0082] 7, at 700, a message is sent from the message server 404 to the content router 408, as previously described. At 702, the content router 408 forwards the message to the MD engine 200. At 704, the MD engine 200 returns the message to the server 404 through the content router 408.

[0083] The MD engine 200 may read 706 and receive 708 the context from the server 404 as needed.

[0084] Message loops 710 and 712 show that the MD engine 200 accesses the content model 202 and the user model 204, respectively, to obtain their respective scores indicating whether the message content and user account, respectively, pose a risk of spam, in this case.

[0085] FIG. 7 assumes the case of risk (e.g., spam). At 714, the MD engine 200 sends the message and score to the message queue 616. At 718, the scores from the ML model may be aggregated, and at 720, a punishment action may be obtained from a decision table or matrix as described above. Because FIG. 7 assumes spam, in addition to writing the message and its risk score to the log 406 at step 722 and optionally writing the message and score to the review process 626 at 724 for further training of the ML model, the MD engine 200 sends the score(s) indicating the risk to the punishment engine 412 at 726. As described elsewhere herein, the punishment engine then determines what punishment to impose based on the score(s) (e.g., removing the message from the gamer's view, warning the sending account, suspending the sending account, or permanently banning a suspended account). The punishment is performed by any of the processes, components, and processors described herein.

[0086] With regard to the composite model 222 and its decision engine 504 in particular, it may accept inputs of zero to one from the user and content models and output a composite score based thereon. The composite model may be a classifier or estimator and may use a matrix of combined scores adjusted for costs that reflect human judgment on the cost of error. The composite model outputs a number between zero and one indicating the risk of a message, such as spam. Some types of risk, such as child abuse, may always be flagged as requiring punishment, while other types of risk, such as swearing, may not always be flagged, depending, for example, on the game context. Thus, the matrix may apply weights to the composite score based on the associated risk of the underlying message, with riskier types of messages triggering reporting to the punishment engine based on a lower score from the ML engine than less risky but still undesirable types of messages.

[0087] An example matrix is ​​shown below. Parameter Threshold (Spam) Threshold (Abuse) Threshold... User gaming behavior .5 .1 .2 Message Content .4 .1 .3 Composite .6 .1 .5

[0088] Thus, in the illustrated example, the thresholds are those that a model-predicted score must meet (e.g., by meeting or exceeding) to trigger a corrective action message to the punishment engine. In the illustrated example, the first row of thresholds is used to determine whether a score from the user model 208 is worth reporting to the punishment engine, the second row of thresholds is used to determine whether a score from the content model 206 is worth reporting to the punishment engine, and the third row of thresholds is used to determine whether a score from the composite model 222 is worth reporting to the punishment engine. The first column of thresholds pertains to a first message type (spam in this example), the second column pertains to a second message type (abuse in this case), and the subsequent columns of thresholds pertain to each subsequent message type (e.g., profanity, sexual content, etc.). The thresholds may be changed as often as necessary to reflect updated risk / penalty assessment decisions.

[0089] Thus, it is readily understood that for any particular message, the threshold for the score from the ML engine at which the message is reported may depend on both the type of message into which the message is classified and the particular parameters associated with the message that are the subject of the ML engine prediction.

[0090] Thus, a first message of a first message type and a first numerical score output by a first ML engine may be reported to the punishment engine, while the same first message of the first message type and a numerical score output by a second, different ML engine may not be reported to the punishment engine.

[0091] Similarly, a first message of a first message type and a numerical score output by a first ML engine may be reported to a punishment engine, while a second message of a second message type and the same first numerical score output by the first ML engine may not be reported to the punishment engine.

[0092] Reference is made to the punishment engine 412, which may also be updated as often as necessary to reflect updated punishment assignments. In one example, the lowest level of punishment may be to send a warning message only once to an account associated with the message content or user behavior or composite score that met the reporting threshold in the matrix, with more severe punishments being implemented for subsequent reports, ranging from warnings to message deletions, temporary account suspensions, and permanent account bans. For example, a warning may be sent to a violating account after one violation, a message may be deleted after N violations (where N is an integer greater than 1), and an account suspension may be implemented after M violations (where M is an integer greater than N).

[0093] In other heuristics, for example, a warning may be sent to a violating account after the predicted score equals a threshold, a message may be deleted in response to the predicted score exceeding the threshold by a first amount, and an account may be suspended in response to the predicted score exceeding the threshold by a second amount greater than the first amount. Combinations of these heuristics may also be implemented. Different types of violating messages may have different penalties.

[0094] Thus, a first message that exceeds a first threshold by a first amount may receive a first type of punishment, while a second message that exceeds the first threshold by the first amount may receive a second type of punishment. A first message that meets the first threshold a first number of times may receive a first type of punishment, while a second message that meets the second threshold a first number of times may receive a second type of punishment. A first message that meets the first threshold a first number of times may receive a first type of punishment, while a second message that meets the first threshold a second number of times may receive a second type of punishment.

[0095] 8-10 relate to training various ML models described herein. Beginning at block 800 in FIG. 8, feature vectors are extracted from data representing a user's gaming behavior. The feature vectors may be annotated with classifications to generate a ground truth set of vectors (block 810). At block 820, the ground truth feature vectors are input into the user model 208 to train the model.

[0096] 9, starting at block 900, feature vectors are extracted from data representing message content. The feature vectors may be annotated with classifications to generate a ground truth set of vectors (block 910). At block 920, the ground truth feature vectors are input to the content model 206 to train the model.

[0097] 10, starting at block 1000, output numerical scores are received from the content model and the user model. The output scores may be annotated with classifications to generate a ground truth set in which pairs of received scores are annotated into a single overall composite score (block 1010). At block 1020, the ground truth scores are input to the composite model 222 to train the model.

[0098] The previously described principles are illustrated in flowchart form for illustrative purposes in Figure 11. The logic represented by Figure 11 may be executed by any one or more of the processors described herein using machine learning techniques and components.

[0099] Beginning at block 1100, for at least some, and preferably each, message transmitted within the computer simulation or computer simulation system, the message may be input to a content model 206 at block 1102. The message may also be input to a user model 208 at block 1104. Optionally, the outputs of the content model 206 and the user model 208 may be sent to a composite model 222 at block 1106 to render a single score representing both the content and user behavior associated with the message being tested.

[0100] Proceeding to block 1108, the score(s) are evaluated for whether user behavior associated with the message and / or the account from which the message originates indicates risk. If a single composite score is used, the composite score may be the only score used in block 1108.

[0101] Moving to decision diamond 1110, it is determined whether the score(s) associated with the message indicate a risk, i.e., whether the message is of a first type that does not require corrective action, or a second type (risky) that does require corrective action. If no corrective action is required, the message and / or its associated user behavior are recorded / logged (block 1112) and then input as training into the associated model (block 1114).

[0102] On the other hand, if corrective action is required, a request is sent to the punishment engine 412 along with the message, score(s), and other relevant data for corrective action in block 1116. The output of the punishment engine is received and implemented in block 1118. The message and / or associated user behavior is recorded / logged (block 1120) and then input as training into the associated model (block 1122).

[0103] FIG. 12 illustrates a simplified embodiment that uses only user actions specific to the particular computer simulation being monitored.

[0104] Beginning at block 1200, for at least some, and preferably each, message transmitted within the computer simulation or computer simulation system, the message may be input to the user model 208 at input block 1202. An output score from the user model 208 is received (block 1204).

[0105] Moving to decision diamond 1206, it is determined whether the score(s) associated with the message indicate a risk, i.e., whether the message is of a first type that does not require corrective action, or a second type (risky) that does require corrective action. If no corrective action is required, the message and / or its associated user behavior are recorded / logged (block 1208) and then input as training into the associated model (block 1210).

[0106] On the other hand, if corrective action is required, a request is sent to the punishment engine 412 along with the message, score(s), and other relevant data for corrective action in block 1212. The output of the punishment engine is received and implemented in block 1214. The message and / or associated user behavior is recorded / logged (block 1216) and then input as training into the associated model (block 1218).

[0107] Although particular embodiments have been shown and described in detail herein, it should be understood that the subject matter encompassed by the present invention is limited only by the claims.

Claims

1. 1. A system comprising: At least one computer-readable storage medium that is not a transitory signal, receiving, from at least a first machine learning (ML) model, at least one indication of whether a first message sent from a first account during play of the computer simulation is spam; In response to the instruction, signaling at least one action module to determine an action; performing the action on the first account; and The first ML model includes a content model configured to receive content from the first message as input, and the instructions include: receiving, from at least a second ML model, at least one indication of whether the first message sent from the first account during play of a computer simulation is spam; signaling at least one action module to determine an action in response to the indication from the second ML model satisfying a threshold; and performing the action on the first account, wherein the second ML model is configured to receive as input information related to user behavior in the computer simulation.

2. The instruction: receiving, from at least the composite model, at least one indication of whether the first message sent from the first account during play of the computer simulation is spam; and signaling at least one action module to determine an action in response to the indication from the composite model satisfying a threshold; 2. The system of claim 1, wherein the composite model is configured to receive information from the first and second ML models as input.

3. The system of claim 1 , wherein the action includes deleting the message.

4. The system of claim 1 , wherein the action includes sending an alert to the first account.

5. The system of claim 1 , wherein the action includes temporarily suspending the first account.

6. The system of claim 1 , wherein the action includes permanently banning the first account.

7. 1. A system comprising: At least one computer-readable storage medium that is not a transitory signal, receiving, from at least a first machine learning (ML) model, at least one indication of whether a first message sent from a first account during play of the computer simulation is spam; In response to the instruction, signaling at least one action module to determine an action; performing the action on the first account; and The ML model includes a first ML model, the first message is a first message type, the indication from the first ML model includes a first numerical score, and the instruction is: A system including at least one computer-readable storage medium executable to report the first message to a punishment engine and receive corrective action therefrom, and to not report a second message of the first message type and the first numerical score output by a second, different ML engine to the punishment engine.

8. 1. A system comprising: At least one computer-readable storage medium that is not a transitory signal, receiving, from at least a first machine learning (ML) model, at least one indication of whether a first message sent from a first account during play of the computer simulation is spam; In response to the instruction, signaling at least one action module to determine an action; performing the action on the first account; and The ML model includes a first ML model, the first message is a first message type, the indication from the first ML model includes a first numerical score, and the instruction is: reporting the first message to a punishment engine and receiving corrective action therefrom; and not reporting to the punishment engine a second message of a second message type and of the same first numerical score output by the first ML engine.

9. 1. A system comprising: At least one computer-readable storage medium that is not a transitory signal, receiving, from at least a first machine learning (ML) model, at least one indication of whether a first message sent from a first account during play of the computer simulation is spam; In response to the instruction, signaling at least one action module to determine an action; performing the action on the first account; and The system includes at least one computer-readable storage medium, wherein the action is established at least in part by the number of times the ML engine outputs the instruction.

10. 1. A system comprising: At least one computer-readable storage medium that is not a transitory signal, receiving, from at least a first machine learning (ML) model, at least one indication of whether a first message sent from a first account during play of the computer simulation is spam; In response to the instruction, signaling at least one action module to determine an action; performing the action on the first account; and The system includes at least one computer-readable storage medium, wherein the action is established at least in part by a magnitude of a score output by the ML engine.

11. 1. A system comprising: At least one computer-readable storage medium that is not a transitory signal, receiving, from at least a first machine learning (ML) model, at least one indication of whether a first message sent from a first account during play of the computer simulation is spam; In response to the instruction, signaling at least one action module to determine an action; performing the action on the first account; and The instruction: In response to the indication exceeding a first threshold by a first amount, performing a first action on a first account associated with the first message; and performing a second action on a second account associated with a second message associated with an indication from the ML engine exceeding the first threshold by the first amount.

12. The instruction: In response to the instruction being returned a first number of times from the ML engine, performing a first action on a first account associated with the first message; 2. The system of claim 1, wherein the system is executable to perform a second action on a second account associated with a second message associated with a second number of instructions from the ML engine.

13. 1. A processor-implemented method comprising: processing content from at least a first message sent from a first account in a computer simulation using a first machine learning (ML) model; processing data representing user behavior associated with the first account using a second ML model; the processor receiving outputs of the first and second ML models; the processor selectively taking corrective action on the first account based at least in part on the output.

14. the processor inputting the outputs of the first and second ML models into a combination module; The method of claim 13 , further comprising: the processor selectively implementing the corrective action on the first account based at least in part on a numerical value output by the combination module.

15. 1. An assembly comprising: at least one processor, receiving at least a first message from a first account associated with a computer game; sending the content of the first message to a first machine learning (ML) model; receiving first instructions from the first ML model representing processing of the content; transmitting data representing user behavior associated with the first account to a second ML model; receiving second instructions from the second ML model representing processing of the data; performing a corrective action on the first account at least in part in response to the first and second instructions; receiving at least a second message from a second account associated with the computer game; sending the content of the second message to the first ML model; receiving third instructions from the first ML model representing processing of the content of the second message; sending data representing user behavior associated with the second account to the second ML model; receiving fourth instructions from the second ML model representing processing of the data representing user behavior associated with the second account; and and not taking any corrective action against the second account in response to at least part of the third and fourth instructions.

16. 16. The assembly of claim 15, wherein the processor is a component of a computer game console.

17. 16. The assembly of claim 15, wherein the processor is a component of a computer game controller.

18. The assembly of claim 15 , wherein the processor is a component of a computer game server.

19. 16. The assembly of claim 15, wherein the processor is programmed to provide the first and second instructions to a combination module and to implement the corrective action based on an output of the combination module.

Images