Machine Learning-Based Gaming Platform Messaging Risk Management

Abstract

A machine learning (ML) model (200) is used to identify spam messages and spammers in a computer gaming environment based on learned gamer behavior. Message text may also be used. An evaluator module (500, 502) receives score(s) from the ML model(s) indicating whether a particular message is spam or not, and if the score(s) meet a threshold, passes the message to a punishment module (412) to determine whether to warn, suspend, or ban the sending account.

Description

[Technical field]

[0001] This application relates generally to machine learning based gaming platform messaging risk management. [Background technology]

[0002] Computer simulations, such as computer games, may allow messaging between players and / or spectators, adding to the enjoyment. Summary of the Invention

[0003] As understood herein, such messaging poses the risk of flooding players with spam and / or messages from spambots, reducing enjoyment of the game, particularly online games, and users must report such messages, which must be analyzed by other humans to determine the appropriate action to take.

[0004] As will be further understood herein, computer games, such as online games, offer unique opportunities for detecting spam and spammers (human or bot) that other computer-based activities do not. In particular, important details of player (gamer) behavior that are not associated with other computer-based activities are typically recorded, and this game-specific behavioral data can be advantageously used to reduce spam in computer games.

[0005] Thus, the principles use data, particularly attributes, that span games and are only available on the gaming platform. They also use the unique in-game behavior patterns of players on the gaming platform. Multiple machine learning systems automate the risk management process. Prediction systems and decision engines are provided that are specific to the unique needs of the gaming platform. Any messages sent during game play may be analyzed, not just those that the user may report as potentially problematic.

[0006] Thus, the system includes at least one computer medium that is not a transitory signal, the computer medium itself including instructions executable by at least one processor to receive from at least a first machine learning (ML) model at least one indication of whether a first message sent from a first account during play of a computer simulation is spam or not, the instructions being executable in response to the indication to send a signal to at least one action module to determine an action and to implement the action against the first account.

[0007] In an example, the first ML model can include a content model configured to receive as input content from the first message, and the instructions can be executable to receive from at least the second ML model at least one indication of whether the first message sent from the first account during play of the computer simulation is spam or not. The instructions can be executable to signal at least one action module to determine an action and to implement the action against the first account in response to the indication from the second ML model satisfying a threshold. The second ML model can be configured to receive as input information related to user behavior in the computer simulation.

[0008] In some embodiments, the instructions can be further executable to receive from at least the composite model at least one indication of whether a first message sent from the first account during play of the computer simulation is spam or not. The instructions can be executable to signal at least one action module to determine an action and to implement the action against the first account in response to the indication from the composite model satisfying a threshold. The composite model can be configured to receive information from the first and second ML models as input.

[0009] In non-limiting implementations, the action may include one or more of deleting the message, sending a warning to the first account, temporarily suspending the first account, or permanently banning the first account.

[0010] In some embodiments, the ML model includes a first ML model, the first message is a first message type, and the instructions from the first ML model include a first numerical score, and the instructions are executable to report the first message to a punishment engine and receive a corrective action therefrom, and to not report a second message of the first message type and the first numerical score output by a second, different ML engine to the punishment engine.

[0011] In some embodiments, the ML model includes a first ML model, the first message is a first message type, and the instructions from the first ML model include a first numerical score, and the instructions are executable to report the first message to a punishment engine and receive a corrective action therefrom, and to not report to the punishment engine a second message of a second message type and the same first numerical score output by the first ML engine.

[0012] The action taken on the account may be established, at least in part, by one or more of the number of times the ML engine outputs the instruction and / or the magnitude of the score output by the ML engine.

[0013] In a non-limiting example, the instructions may be executable to, in response to the instructions exceeding a first threshold by a first amount, take a first action on a first account associated with the first message and take a second action on a second account associated with a second message associated with the instructions from the ML engine exceeding the first threshold by the first amount.

[0014] Optionally, the instructions may be executable in response to instructions being returned a first number of times from the ML engine to perform a first action on a first account associated with the first message and to perform a second action on a second account associated with a second message associated with a second number of instructions from the ML engine.

[0015] In another aspect, a method includes using a first machine learning (ML) model to process content from at least a first message sent from a first account in a computer simulation. The method also includes using a second ML model to process data representing user behavior associated with the first account. The method includes receiving outputs of the first and second ML models and selectively implementing a corrective action on the first account based at least in part on the outputs.

[0016] In another aspect, an assembly includes at least one processor programmed to receive at least a first message from a first account associated with a computer game. The processor is programmed to send content of the first message to a first machine learning (ML) model and receive first instructions from the first ML model representative of processing the content. The processor is also programmed to send data representative of user behavior associated with the first account to a second ML model and receive second instructions from the second ML model representative of processing the data. At least in part in response to the first and second instructions, the processor is programmed to implement a corrective action against the first account.

[0017] The processor is further programmed to receive at least a second message from a second account associated with the computer game. The processor is programmed to send content of the second message to the first ML model, receive third instructions from the first ML model representing processing the content of the second message, send data representing user behavior associated with the second account to the second ML model, and receive fourth instructions from the second ML model representing processing the data representing user behavior associated with the second account. At least in part in response to the third and fourth instructions, the processor is programmed to not implement a corrective action against the second account.

[0018] Without limitation, the processor may be a component of a computer game console, a computer game controller, or a computer game server.

[0019] The details of the present application, both as to its structure and operation, can best be understood with reference to the accompanying drawings, in which like reference numerals refer to like parts and in which: [Brief description of the drawings]

[0020] [Figure 1] 1 is a block diagram of an exemplary system including an example according to the present principles; [Diagram 2] A high-level example machine learning (ML) risk management architecture. [Diagram 3] Illustrates an example of ML risk management information flow. [Figure 4] 1 illustrates further details of an example ML risk management information flow. [Diagram 5] 1 illustrates further details of an example ML risk management information flow. [Figure 6] 1 illustrates an example messaging sequence when the risk level of a message falls below a threshold. [Figure 7] 1 illustrates an example messaging sequence when the risk level of a message exceeds a threshold. [Figure 8] Example logic for training an ML model using gaming behavior is illustrated in example flow chart form. [Figure 9] Example logic for training an ML model using message content is illustrated in example flow chart form. [Figure 10] Example logic for training a composite ML model is illustrated in example flow chart form. [Figure 11] 1 illustrates an example of logic for using one or more ML models for message risk management consistent with the present principles in example flow chart form. [Figure 12] An example of alternative logic for using one or more ML models for message risk management consistent with the present principles is illustrated in example flow chart form. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The present disclosure generally relates to a computer ecosystem, including aspects of a consumer electronics (CE) device network (e.g., without limitation, a computer gaming network). The system herein may include a server component and a client component that may be connected via a network such that data may be exchanged between the client component and the server component. The client components may include one or more computing devices, such as gaming consoles, such as Sony PlayStation®, or gaming consoles made by Microsoft or Nintendo or other manufacturers, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart TVs, Internet-enabled TVs), portable computers, such as laptops and tablet computers, and other mobile devices, such as smartphones, as well as further examples described below. These client devices may operate with a variety of operating environments. For example, some of the client computers may use, as examples, a Linux® operating system, an operating system from Microsoft, or a Unix® operating system, or an operating system made by Apple, Inc. or Google. These operating environments may be used to run one or more browsing programs, such as browsers made by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by Internet servers, as described below. An operating environment according to the present principles may also be used to run one or more computer game programs.

[0022] The server and / or gateway may include one or more processors that execute instructions that configure the server to send and receive data over a network, such as the Internet. Alternatively, clients and servers may be connected via a local intranet or a virtual private network. The server or controller may be instantiated by a game console, e.g., a Sony PlayStation®, a personal computer, etc.

[0023] Information may be exchanged between the client and the server over a network. For this purpose and for security, the server and / or the client may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus that implements the method for providing a secure community, such as an online social website, to network members.

[0024] The processor may be a single-chip or multi-chip processor capable of implementing logic through various wiring such as address lines, data lines, and control lines, as well as registers and shift registers.

[0025] Components included in one embodiment may be used in other embodiments in any suitable combination, for example, any of the various components described herein and / or illustrated in the figures may be combined, interchanged, or excluded from other embodiments.

[0026] A "system having at least one of A, B, and C" (and similarly "a system having at least one of A, B, or C" and "a system having at least one of A, B, C") includes systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.

[0027] Referring now specifically to FIG. 1, an exemplary system 10 is shown. System 10 may include one or more of the exemplary devices described above and further below in accordance with the present principles. A first of the exemplary devices included in system 10 is a consumer electronics (CE) device, such as an audio video device (AVD) 12, such as, without limitation, an Internet-enabled TV with a TV tuner (as well as a set-top box that controls the TV). AVD 12 may also alternatively be a computerized Internet-enabled ("smart") phone, a tablet computer, a notebook computer, an HMD, a wearable computerized device, a computerized Internet-enabled music player, a computerized Internet-enabled headphones, a computerized Internet-enabled embedded device, such as an embedded skin device, etc. Regardless of what it is, it should be understood that AVD 12 is configured to perform the present principles (e.g., communicate with other CE devices to perform the present principles, execute the logic described herein, and perform any other functions and / or operations described herein).

[0028] Thus, to carry out such principles, the AVD 12 may be established by some or all of the components shown in FIG. 1. For example, the AVD 12 may include one or more displays 14, which may be implemented by a high definition or ultra-high definition "4K" or higher flat screen and may be touch operable to receive user input signals via touch on the display. The AVD 12 may include one or more speakers 16 for outputting audio in accordance with the present principles, and at least one additional input device 18, such as an audio receiver / microphone for inputting audible commands to the AVD 12 to control the AVD 12. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a WAN, a LAN, etc., under the control of one or more processors 24. Thus, the interface 20 may be, without limitation, a Wi-Fi transceiver (which is an example of a wireless computer network interface, such as, without limitation, a mesh network transceiver). It should be understood that processor 24 controls AVD 12 (including the other elements of AVD 12 described herein) to carry out the present principles, e.g., to control display 14 to present images thereon and to receive input therefrom. It should further be noted that network interface 20 may be a wired or wireless modem or router, or other suitable interface, e.g., a wireless telephone transceiver, or a Wi-Fi transceiver as previously described.

[0029] In addition to the foregoing, AVD 12 may also include one or more input and / or output ports 26 (e.g., a High-Definition Multimedia Interface (HDMI®) port or a USB port) and may physically connect to another CE device and / or a headphone port (for connecting headphones to AVD 12 so that audio from AVD 12 is presented to the user via the headphones). For example, input port 26 may connect, wired or wirelessly, to a cable or satellite source 26a of audio-video content. Thus, source 26a may be a separate or integrated set-top box, or a satellite receiver. Or source 26a may be a game console or disc player containing content. When implemented as a game console, source 26a may include some or all of the components described below in connection with CE device 48.

[0030] AVD 12 may further include one or more computer memories 28, e.g., non-transitory disk-based or solid-state storage devices, possibly embodied within the AVD's chassis, as a stand-alone device, or as a personal video recording device (PVR) or video disk player, either internal or external to the AVD's chassis for playing AV programs, or as a removable memory medium or server, as described below. In some embodiments, AVD 12 may also include a location or position receiver, e.g., without limitation, a cellular receiver, a GPS receiver, and / or an altimeter 30 (configured to receive geographic location information from satellites or cellular towers and provide that information to processor 24 and / or determine the altitude at which AVD 12 is located with processor 24). Component 30 may also be implemented by an inertial measurement unit (IMU), typically including a combination of accelerometers, gyroscopes, and magnetometers to determine the location and orientation of AVD 12 in three dimensions, or by an event-based sensor.

[0031] Continuing with the description of the AVD 12, in some embodiments, the AVD 12 may include one or more cameras 32 (which may be thermal imaging cameras, digital cameras, e.g., webcams, event-based sensors, and / or cameras integrated within the AVD 12 and controllable by the processor 24 to collect pictures / images and / or videos) in accordance with present principles. Also included on the AVD 12 may be a Bluetooth transceiver 34 and other NFC elements 36 for communicating with other devices using Bluetooth and / or Near Field Communication (NFC) technologies, respectively. An exemplary NFC element may be a Radio Frequency Identification (RFID) element.

[0032] Still further, the AVD 12 may include one or more auxiliary sensors 38 (e.g., motion sensors, e.g., accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands)) to provide input to the processor 24. The AVD 12 may include an over-the-air TV broadcast port 40 for receiving OTA TV broadcasts and providing input to the processor 24. Note that in addition to the above, the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, e.g., an IR data association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, as well as a kinetic energy harvester that may convert kinetic energy into electricity to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may be included. One or more haptic generators 47 may be provided for generating a haptic signal that can be detected by a person holding or contacting the device.

[0033] With further reference to FIG. 1, in addition to the AVD 12, the system 10 may include one or more other CE device types. In one example, the first CE device 48 may be a computer game console (which may be used to transmit computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or via a server, as described below), while the second CE device 50 may include similar components as the first CE device 48. In the illustrated example, the second CE device 50 may be configured as a computer game controller operated by a player or a head mounted display (HMD) worn by the player. In the illustrated example, only two CE devices are shown, but it should be understood that fewer or more devices may be used. The devices herein may implement some or all of the components shown for the AVD 12. Any of the components shown in the following figures may incorporate some or all of the components shown for the AVD 12.

[0034] Referring now to the at least one server 52 previously mentioned, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56 (e.g., disk-based or solid-state storage), and at least one network interface 58 (which, under the control of the server processor 54, enables communication with other devices of FIG. 1 over the network 22, and may actually facilitate communication between the server and client devices in accordance with the present principles). It should be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (e.g., a wireless telephone transceiver, etc.).

[0035] Thus, in some embodiments, server 52 may be an entire Internet server or server "farm" and may include and perform "cloud" functionality such that, for example, in an example embodiment for a network gaming application, devices of system 10 may access a "cloud" environment via server 52. Alternatively, server 52 may be implemented by one or more game consoles or other computers in the same room or nearby as the other devices shown in FIG.

[0036] The components illustrated in the following figures may include some or all of the components illustrated in Figure 1. The user interfaces (UIs) described herein may be integrated and extended, and UI elements may be mixed and matched between UIs.

[0037] The present principles may use various machine learning models, e.g., deep learning models. Machine learning models consistent with the present principles may use various algorithms trained in ways including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms, which may be implemented by computer circuitry, include one or more neural networks, e.g., convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a type of RNN known as a long short-term memory (LSTM) network. Support vector machines (SVMs) and Bayesian networks may also be considered examples of machine learning models. In addition to the types of networks mentioned above, the models herein may be implemented by a classifier.

[0038] Thus, as understood herein, performing machine learning may include accessing training data and then training a model on the training data so that the model can process and infer additional data. Thus, an artificial neural network / artificial intelligence model trained through machine learning may include an input layer, an output layer, and multiple hidden layers between them. The hidden layers are configured and weighted to make inferences regarding the appropriate output.

[0039] Reference is now made to FIG. 2. As an overview, FIG. 2 illustrates two separate ML models that are used to utilize data in the game to determine whether a message is undesirable, e.g., nuisance spam, or from a human or bot spammer. Every message sent in a computer simulation, e.g., a computer game, may in some embodiments be evaluated on the fly by a model, and appropriate action may be taken in response to the model output. One model is a content-based model, which looks at the content of messages sent throughout the game platform, such as message text, metadata, Uniform Resource Locator (URL) or Uniform Resource Identifier (URI), and characters. Another model is a "user model" that takes user behavior as input.

[0040] A client-facing message routing and decision (MD) engine 200 communicates with a content-based prediction service 202 and an account-based prediction service 204. As a result, the content-based prediction service 202 loads a content-based natural language processing (NLP) machine learning (ML) model (or "content model" for short) 206, which can be updated daily. Meanwhile, the account-based prediction service 204 can communicate with a user behavior ML model (or "user model" for short) 208 via a user search database 210.

[0041] The content models 206 include one or more machine learning models based on the content of messages shared between players that can be applied to predict the likelihood of a message falling into multiple risk management categories (e.g., spam, offensive or abusive content, child protection, hate speech, pornographic or obscene content, etc.).

[0042] In response, the user model 208 includes one or more machine learning models based on user behavior on the gaming platform, such as gaming activity, purchasing habits, and messaging behavior, that can be applied to predict the likelihood that a target user will behave in a manner that violates community guidelines, for example, by engaging in spamming on the gaming platform.

[0043] The content-based ML model 206 may have access to various data sources 212, such as a regular message log 214 covering all messages sent within the simulation system, as well as a “griefed” message log 216 containing messages flagged for review by a moderator and / or messages that have been reviewed by a moderator.

[0044] Similarly, the user behavior ML model 208 may have access to data sources 212 , such as a user behavior log 218 and a list of banned user identities 220 .

[0045] In some embodiments, the MD engine 200 may receive information from a ML-based or rule-based composite model 222 that aggregates scores from other ML models described herein and outputs a single score upon which action is taken consistent with the present principles. Additionally, Figure 2 illustrates that the content-based prediction service 202 may receive information from a uniform resource identifier (URI) evaluation module 224 and a text profanity filter 226.

[0046] The system shown in FIG. 2 can evaluate any message sent within a computer simulation system or computer game on the fly using data not available anywhere else (i.e., game data). As described in more detail below, this data can include user (also referred to herein as player or gamer) behavior, such as number of boss kills, game genre (as some games naturally involve more vulgar language than others and thus vulgar language is less actionable in such games), and other user behavior information specific to the simulation community and / or the simulation platform itself (e.g., how long a particular user originating a message under test played the simulation in which the message was sent, whether the user purchased the simulation, how often the user plays a particular simulation, etc.), all of which may be used as described herein to determine whether a particular user is a human or bot spammer. This basic information can vary from game to game, as behavior in one game may not be unusual in another game.

[0047] Explaining FIG. 2 in more detail, both the user-based model and the content-based model may be trained by four types of data sources. These include a regular message log 214 that stores and processes regular messages and communications sent between users, and a grief message log 216 that stores content-moderated messages and their content moderation results from an internal content moderation team. The data sources shown in FIG. 2 also include user data 218 that stores a wide variety of user behavior data (e.g., a user's purchase history, game play history, in-game activity history, messaging history, etc.). User data is specific to the game platform. For example, in game play history, which games a user played, for how long, and in how many sessions may all be recorded in user data 218. As another example, in the case of in-game activity history, user data may record how successful a user was at a particular in-game activity, how often they killed bosses, completed tasks, how many trophies they earned, and their inferred skill level.

[0048] Messaging history is also specific to the gaming platform because users may use special language specific to their gaming community and special abbreviations specific to the games they play. The patterns learned through machine learning are specific to the gaming platform.

[0049] FIG. 2 also shows a banned ID list that stores historically banned account IDs.

[0050] Both models can be trained (after data processing and feature extraction) on these data sources. The trained models are then loaded into the respective online real-time services 202, 204 to perform predictions on whether a message is spam and / or whether the message sender is a spammer. Once the consumer-facing MD engine 200 (which may be implemented as an application programming interface (API)) receives a message sent through an account ID, the message body and the account ID are sent to the content-based prediction service 202 and the account prediction service 204. The two services return a machine learning prediction score for the message body based on the content model and a machine learning prediction score for the account ID based on the user model.

[0051] In an example, a composite model 222 may be provided to combine the predicted scores from the content model 206 and the user model 208 to predict one single actionable score for spam risk management. The score is used to take appropriate content management actions.

[0052] To understand the ML workflow used consistent with the present principles, reference is now made to Figure 3. A data extract, transform, and load (ETL) module 300 reads data from the regular message log 214, the "griefed" message log 216, and the user action log 218, and writes the combination of these logs to a preprocessed database 302.

[0053] The data logs may optionally include unique, gaming platform specific data for building machine learning models for each individual computer simulation, such that each online computer game in the gaming system may, for example, predict spam and spammer accounts using its own unique ML model. It should be understood that any of the data listed below may be game specific or may be non-game specific, if desired, and may instead represent a composite score aggregated across all games associated with a user account.

[0054] Data that may be included in user data 218 may include, for example, static user data, such as wallet balance, age, gender, country, city, language, whether an account is active, email verification status, registration status, and membership status. User data may also include the user's gaming behavior (broken down by game, if necessary), such as games played, play time, play history, number of trophies earned by the account, in-game activities, and activity success rates. User data may further include game-specific information regarding the user's messaging behavior, frequency of sending messages, number of different accounts messaged, regions of accounts messaged, and message types. User data may also include the user's commercial behavior, such as number and titles of games owned, number and titles of non-free game purchases.

[0055] Such data may be used to train models as to whether certain user behaviors are risky and indicative of problematic accounts, e.g., spammer accounts, that require punishment. For example, if an account sends messages frequently, this may indicate a spambot, and therefore a risk. If an account sends a number of group messages that meet a threshold, this may indicate a spambot. If an account has gained a number of friends that meet a threshold, without ever purchasing a computer game, this may indicate a spambot. If an account sends group messages to a number of recipients that meet a threshold, this may indicate a spambot.

[0056] Game-specific or composite (aggregated) user social data may also be stored in user data 218. Such data may include links to social profile photos and relationship status.

[0057] The messaging data contained in the regular and grief message logs 214, 216 may be game specific or compound and may include the message body, as well as the message type, length, and start time.

[0058] The preprocessed database 302 is read by a content feature extraction module 304 and a user feature extraction module 306. The content feature extraction model outputs feature vectors representing the data read from the preprocessed database 302 to an offline content feature vector database 308 and an online content feature vector database 309. The user feature extraction model 306 outputs feature vectors representing the data read from the preprocessed database 302 to an offline user feature database 310. The data is converted into a data format suitable for machine learning and designed into a data vector representation that can be digested by machine learning algorithms.

[0059] The feature vectors in the offline databases 308, 310 are read during the training phase by the content model in training 206A, the user model in training 208A, and, if provided, the composite model in training 222A, respectively. The composite model in training 222A also reads the preprocessed database 302, as shown. Also, as shown, the user model in training 208A may write information to the online user score search database 311 in embodiments where real-time updates of the current scores are not required for the user model, in which case scores may be calculated for the user and cached in the online user score search database 311. The real-time user score service 204 reads the scores from the online user score search database 311 using the user ID as an input argument. It should be appreciated that user features from the user feature extraction model 306 may also be sent directly to the online user score search database 311.

[0060] 3, it is understood that feature engineering and model training involves reading input data, extracting patterns from the data (e.g., how often a user has played a game recently, how successful the user was in completing tasks within the game), which are stored in respective offline databases 308, 310, and using these patterns to train models 206, 208, 222 with the goal of predicting how likely target users and messages are to be risky, e.g., how likely they are to be spam, how likely they are to pose a threat to other users, and how likely they are to not follow community guidelines. Note that training data may include ground truth, e.g., messages annotated by humans as spam, user behavior annotated by humans as potential spammer accounts, etc.

[0061] The in-training models 206A, 208A, 222A may be written to the offline model store 312. In this step, three categories of models may be built from each in-training model 206A, 208A, 222A for their specific tasks: content models 206 that evaluate message content as spam, user models 208 that evaluate user behavior patterns as spammer accounts, and composite models / services 222.

[0062] As shown in Figure 3, each content, user, and composite prediction service 202, 204, 222 reads data from the model store 312, the online content feature database 309, and the online user feature database 311. At this stage, the online prediction services 202, 204, 222 are constructed to return prediction scores based on user (account ID) and message (message body) inputs. All three services may load the trained models as described above, read the features and extracted patterns, and make real-time predictions.

[0063] 4 and 5 provide further details on how the MD engine 200 shown in FIG. 2 uses ML to determine the probability that a particular message or user meets "risk" criteria (e.g., whether a message is spam, sexually explicit, or gratuitously profane, etc.).

[0064] For example, an ML model trained as described above may be trained on suitable data (including data defined herein) to evaluate messages and their likelihood of being spam messages. An ML model may be trained to evaluate messages and senders to determine whether a user is a spambot. Once determined (with a score between 0.0 and 1.0, with scores closer to 1.0 indicating a higher probability of meeting the criteria), the score is passed to one or more decision engines, which determine what, if any, punishment should be based on the score and an inference engine.

[0065] In FIG. 4, it is assumed that a spammer account 400 sends a spam message to an in-game messaging app 402. The app 402 sends the message to a message server 404. The message server 404 logs the message in a log 406 and sends it to a content router 408. The content router 408 may run one or more ML models described herein to determine whether the message is spam or not. The content router 408 may return the results to the message server 404, which may then relay the results back to the app 402. The content router 408 may also send the evaluation results to the log 406 and / or a review queue 410 for human review and, if necessary, annotation for input into further training of a running ML model.

[0066] The content router 408 can send a request to a punishment engine 412 to ban or otherwise punish the spammer account 400. The punishment engine 412 can write the message to a griefed message log 216, which can be read by the ML models 206, 208 along with the user date from the user log 218. Note that the MD engine 200 can be in communication with the punishment engine 412.

[0067] The content router 408 can also send the message to the MD engine 200. The MD engine 200 can send the message to a message queue 414, which may be read by the ML models 206, 208. The content and user prediction services 202, 204 (and composite prediction engine 222, if provided) are accessed by the MD engine 200 and return a score representing the likelihood that the message is spam and / or that the account 400 is a spammer account based on the output of the prediction services.

[0068] FIG. 5 shows an example in which the MD engine 200 and / or other components embody multiple decision engines. In the illustrated example, there is an account decision engine 500 that determines whether the score from the user ML model 208 indicates that a message sending account is risky (e.g., a spammer), a content decision engine 502 that determines whether the score from the content ML model 206 indicates that the message content is risky (e.g., spam), and a composite decision engine 504 that determines whether a single composite score representing the user and content scores indicates risk.

[0069] The decision engine can be automated so that no human interaction is required. The decision engine can be specific considering a particular reasoning engine (and is specific to the game data and actions). The decision engine can be specific to gaming actions and based on years of content management experience, and can be extensible so that further reasoning engines can be added without recoding. Additionally, the decision engine can be programmable so that punishments and matrices are programmable (e.g., using YAML) and can be updated and changed without modifying the engine.

[0070] The engine can also log all actions (because logging for each action is programmable in YAML) and set these aside for later human evaluation in case of review or challenge of the decision.

[0071] All decision engines are loaded with scores and actions associated with the scores at startup. As scores come from the ML engines, they are collected until they are all there or a timeout occurs. The scores are then consolidated into a list and sent to the decision engine selector 506, which first looks through the list of scores and corresponding ML models and selects one to operate on. The decision engine selector 506 then decides which decision engine 500, 502, 504 to send the scores to from the decision engine configuration database 507. Thus, each decision engine corresponds to one or more ML models.

[0072] As will be further described shortly, the selected decision engine then uses the score to initiate a script that acts based on the score, which could range from doing nothing, issuing a warning, suspending the account, to permanently canceling the account.

[0073] In FIG. 5, group messages 508, friend requests 510, and recipient-specific messages 512 are sent from computer-simulated user accounts (clients) to an analysis tool selector 514 of the content router 408. The selector 514 sends its input to a block 516 of the MD engine 200. The block 516 receives ML models from a database 518 and sends input to the user and content prediction models 208, 206 as shown. The analysis tool selector 514 may also send its input to various other analysis tools, such as a URI evaluator 224 that evaluates whether any URIs in the message are on a restricted list, a text profanity filter 226 that evaluates whether the message body contains profanity or other objectionable language, an image evaluator 520 that determines whether any images sent by the user contain objectionable material, and an anti-virus evaluator 522 that determines whether any message contains a virus. Any messages that trigger a violation output from the analysis tools may be removed and / or the account sending the message may be penalized.

[0074] 5 shows that the ML models 208, 206 send their output numerical values ​​(e.g., zero to one, with one indicating the highest probability of violation and zero indicating the lowest) to a predicted response collection block 524 in the MD engine 200. The responses are aggregated in an aggregation block 526 of the MD engine 200, and the aggregate score is communicated from the aggregation block 526 to the decision engine selector 506.

[0075] 5, the selected decision engine block 500 (or 502 or 504) of the MD engine 200 can send its decision output to the punishment engine 412 to determine the appropriate punishment for the account sending the message flagged as risky by the decision engine. The decision engine block 500 can further send its decision to a message and score log for human review at 530 as well as to the log 406 for recording.

[0076] It can be readily appreciated that the actions and decisions can be configurable and based on gameplay.

[0077] 6 and 7 illustrate example message flows assuming messages that do not require punishment and messages that do require punishment, respectively (in this case the ML model is trained to test for spam). It should be understood that the principles may also be used to test for other types of risk (e.g., abusive, profane, etc.).

[0078] 6, at 600, a message is sent from the message server 404 to the content router 408, as previously described. At 602, the message is forwarded from the content router 408 to the MD engine 200. The MD engine 200 returns the message at 604 to the server 404 through the content router 408.

[0079] The MD engine 200 may read 606 and receive 608 the context from the server 404 as needed.

[0080] Message loops 610 and 612 show that the MD engine 200 accesses the content model 202 and the user model 204, respectively, to obtain their respective scores indicating whether the message content and user account, respectively, pose a risk of spam, in this case.

[0081] Figure 6 assumes no risk (e.g., no spam). At 614, the MD engine 200 sends the message and score to a message queue 616. At 618, the scores from the ML model may be tallied, and at 620, a punishment action may be derived from a decision table or matrix as previously described. Because Figure 6 assumes no spam, no action is taken other than writing the message and its risk score to the log 406 at step 622, and optionally writing the message and score at 624 to a review process 626 for use in further training of the ML model.

[0082] 7, at 700, a message is sent from the message server 404 to the content router 408, as previously described. At 702, the content router 408 forwards the message to the MD engine 200. The MD engine 200 returns the message to the server 404 through the content router 408, at 704.

[0083] The MD engine 200 may read 706 and receive 708 the context from the server 404 as needed.

[0084] Message loops 710 and 712 show that the MD engine 200 accesses the content model 202 and the user model 204, respectively, to obtain their respective scores indicating whether the message content and user account, respectively, pose a risk of spam, in this case.

[0085] FIG. 7 assumes the case of risk (e.g., spam). At 714, the MD engine 200 sends the message and score to the message queue 616. At 718, the scores from the ML model may be tallied, and at 720, the punishment action may be derived from a decision table or matrix as described above. Since FIG. 7 assumes spam, in addition to writing the message and its risk score to the log 406 at step 722 and optionally writing the message and score to the review process 626 at 724 for use in further training of the ML model, at 726, the MD engine 200 sends the score(s) indicative of risk to the punishment engine 412. As described elsewhere herein, the punishment engine then determines what punishment to impose based on the score(s) (e.g., remove the message from the gamer's view, warn the sending account, suspend the sending account, permanently ban the suspended account). The punishment is performed by any of the processes, components, and processors described herein.

[0086] With regard to the composite model 222 and its decision engine 504 in particular, it may accept inputs of zero to one from the user and content models and output a composite score based thereon. The composite model may be a classifier or estimator and may use a matrix of combined scores adjusted for costs that reflect human judgments on the cost of error. The composite model outputs a number between zero and one indicating the risk of the message, such as spam. Some types of risk, such as child abuse, may always be flagged as needing punishment, while other types of risk, such as swearing, may not always be flagged, depending on, for example, the game context. Thus, the matrix may apply weights to the composite score based on the associated risk of the message behind it, with riskier types of messages triggering reporting to the punishment engine based on a lower score from the ML engine than less risky but still undesirable types of messages.

[0087] An example matrix is ​​shown below. Parameter Threshold(Spam) Threshold(Abuse) Threshold... User gaming behavior .5 .1 .2 Message Content .4 .1 .3 Composite .6 .1 .5

[0088] Thus, in the illustrated example, the thresholds are what the model predicted scores must meet (e.g., by meeting or exceeding) to trigger a corrective action message to the punishment engine. In the illustrated example, the first row of thresholds is used to determine whether a score from the user model 208 is worth reporting to the punishment engine, the second row of thresholds is used to determine whether a score from the content model 206 is worth reporting to the punishment engine, and the third row of thresholds is used to determine whether a score from the composite model 222 is worth reporting to the punishment engine. The first column of thresholds pertains to a first message type (spam in this example), the second column pertains to a second message type (abuse in this case), and the subsequent columns of thresholds pertain to each subsequent message type (e.g., profanity, sexual content, etc.). The thresholds may be changed as many times as necessary to reflect updated risk / penalty assessment decisions.

[0089] Thus, it is readily understood that for any particular message, the threshold for the score from the ML engine at which the message is reported may depend on both the type of message into which the message is classified and the particular parameters associated with the message that are the subject of the ML engine prediction.

[0090] Thus, a first message of a first message type and a first numerical score output by a first ML engine may be reported to the punishment engine, while the same first message of the first message type and numerical score output by a second, different ML engine may not be reported to the punishment engine.

[0091] Similarly, a first message of a first message type and a numerical score output by a first ML engine may be reported to the punishment engine, while a second message of a second message type and the same first numerical score output by the first ML engine may not be reported to the punishment engine.

[0092] See the punishment engine 412, which may also be updated as many times as necessary to reflect updated punishment assignments. In one example, the lowest level of punishment may be to send a warning message only once to the account associated with the message content or user behavior or composite score that met the reporting threshold in the matrix described above, with more severe punishments being implemented for the number of subsequent reports, ranging from warnings, message deletion, temporary account suspension, to permanent account bans. For example, a warning may be sent to a violating account after one violation, a message may be deleted after N violations (N being an integer greater than 1), and an account suspension may be implemented after M violations (M being an integer greater than N).

[0093] In other rules of thumb, for example, a warning may be sent to a violating account after the predicted score equals a threshold, a message may be removed in response to the predicted score exceeding the threshold by a first amount, and an account suspension may be implemented in response to the predicted score exceeding the threshold by a second amount greater than the first amount. Combinations of these rules of thumb may also be implemented. Different types of violating messages may have different penalties.

[0094] Thus, a first message that exceeds a first threshold by a first amount may receive a first type of punishment, while a second message that exceeds the first threshold by a first amount may receive a second type of punishment. A first message that meets the first threshold a first number of times may receive a first type of punishment, while a second message that meets the second threshold a first number of times may receive a second type of punishment. A first message that meets the first threshold a first number of times may receive a first type of punishment, while a second message that meets the first threshold a second number of times may receive a second type of punishment.

[0095] 8-10 relate to training various ML models described herein. Beginning at block 800 in FIG. 8, feature vectors are extracted from data representing a user's gaming behavior. The feature vectors may be annotated with classifications to generate a ground truth set of vectors (block 810). At block 820, the ground truth feature vectors are input into the user model 208 to train the model.

[0096] 9, starting at block 900, feature vectors are extracted from data representing message content. The feature vectors may be annotated with classifications to generate a ground truth set of vectors (block 910). At block 920, the ground truth feature vectors are input to the content model 206 to train the model.

[0097] Beginning at block 1000 in Figure 10, output numerical scores are received from the content model and the user model. The output scores may be annotated with classifications to generate a ground truth set in which pairs of received scores are annotated into a single overall composite score (block 1010). At block 1020, the ground truth scores are input to the composite model 222 to train the model.

[0098] The principles previously discussed are illustrated in the form of an illustrative flow chart in Figure 11. The logic represented by Figure 11 may be executed by any one or more of the processors described herein using machine learning techniques and components.

[0099] Beginning at block 1100, for at least some, and preferably each, message transmitted within the computer simulation or computer simulation system, the message may be input to a content model 206 at block 1102. The message is also input to a user model 208 at block 1104. Optionally, the output of the content model 206 and the user model 208 may be sent to a composite model 222 at block 1106 to render a single score representative of both the content and user behavior associated with the message under test.

[0100] Proceeding to block 1108, the score(s) are evaluated for whether user behavior associated with the message and / or the account from which the message originates indicates risk. If a single composite score is used, the composite score may be the only score used in block 1108.

[0101] Moving to decision diamond 1110, it is determined whether the score(s) associated with the message indicate a risk, i.e., whether the message is of a first type that does not require corrective action, or a second type (risky) that does require corrective action. If no corrective action is required, the message and / or its associated user behavior is recorded / logged (block 1112) and then input as training into an associated model (block 1114).

[0102] On the other hand, if corrective action is required, a request is sent to the punishment engine 412 along with the message, score(s), and other relevant data for corrective action in block 1116. The output of the punishment engine is received and implemented in block 1118. The message and / or associated user behavior is recorded / logged (block 1120) and then input as training into an associated model (block 1122).

[0103] FIG. 12 illustrates a simplified embodiment that uses only user actions specific to the particular computer simulation being monitored.

[0104] Beginning at block 1200, for at least some, and preferably each, message transmitted within the computer simulation or computer simulation system, the message may be input to a user model 208 at an input block 1202. An output score from the user model 208 is received (block 1204).

[0105] Moving to decision diamond 1206, it is determined whether the score(s) associated with the message indicate a risk, i.e., whether the message is of a first type that does not require corrective action, or a second type (risky) that does require corrective action. If no corrective action is required, the message and / or its associated user behavior is recorded / logged (block 1208) and then input as training into an associated model (block 1210).

[0106] On the other hand, if corrective action is required, a request is sent to the punishment engine 412 along with the message, score(s), and other relevant data for corrective action in block 1212. The output of the punishment engine is received and implemented in block 1214. The message and / or associated user behavior is recorded / logged (block 1216) and then input as training into an associated model (block 1218).

[0107] Although particular embodiments have been shown and described in detail herein, it should be understood that the subject matter encompassed by the present invention is limited only by the scope of the claims.

Claims

1. 1. A system comprising: At least one computer medium that is not a transitory signal, receiving at least one indication from at least a first machine learning (ML) model of whether a first message sent from a first account during play of the computer simulation is spam; in response to said instruction, sending a signal to at least one action module to determine an action; and performing the action on the first account.

2. The first ML model includes a content model configured to receive as input content from the first message, and the instructions include: receiving from at least a second ML model at least one indication of whether the first message sent from the first account during play of the computer simulation is spam; in response to the indication from the second ML model satisfying a threshold, sending a signal to at least one action module to determine an action; 2. The system of claim 1, wherein the second ML model is configured to receive as input information related to user behavior in the computer simulation.

3. The instruction: receiving at least one indication from at least the composite model of whether the first message sent from the first account during play of the computer simulation is spam; and in response to the indication from the composite model satisfying a threshold, sending a signal to at least one action module to determine an action; 3. The system of claim 2, wherein the composite model is configured to receive information from the first and second ML models as input, and is executable to perform the action on the first account, the composite model being configured to receive information from the first and second ML models as input.

4. The system of claim 1 , wherein the action includes deleting the message.

5. The system of claim 1 , wherein the action includes sending an alert to the first account.

6. The system of claim 1 , wherein the action includes temporarily suspending the first account.

7. The system of claim 1 , wherein the action includes permanently banning the first account.

8. The ML model includes a first ML model, the first message is a first message type, the indication from the first ML model includes a first numerical score, and the instruction is:

2. The system of claim 1, wherein the system is executable to report the first message to a punishment engine and receive corrective action therefrom, and to not report to the punishment engine a second message of the first message type and the first numerical score output by a second, different ML engine.

9. The ML model includes a first ML model, the first message is a first message type, the indication from the first ML model includes a first numerical score, and the instruction is: reporting the first message to a punishment engine and receiving corrective action therefrom; 2. The system of claim 1, operable to: not report to the punishment engine a second message of a second message type and the same first numerical score output by the first ML engine.

10. The system of claim 1 , wherein the action is established at least in part by the number of times the ML engine outputs the instruction.

11. The system of claim 1 , wherein the action is established at least in part by a magnitude of a score output by the ML engine.

12. The instruction: in response to the indication exceeding a first threshold by a first amount, performing a first action on a first account associated with the first message; and implementing a second action on a second account associated with a second message associated with an indication from the ML engine exceeding the first threshold by the first amount.

13. The instruction: in response to the indication being returned a first number of times from the ML engine, performing a first action on a first account associated with the first message; 2. The system of claim 1, wherein the system is executable to: perform a second action on a second account associated with a second message associated with a second number of indications from the ML engine.

14. 1. A method comprising: using a first machine learning (ML) model to process content from at least a first message sent from a first account in the computer simulation; processing data representing user behavior associated with the first account using a second ML model; receiving outputs of the first and second ML models; and selectively taking a corrective action on the first account based at least in part on the output.

15. inputting the outputs of the first and second ML models into a combination module; and selectively implementing the corrective action on the first account based at least in part on a numerical value output by the composite module.

16. 1. An assembly comprising: At least one processor, receiving at least a first message from a first account associated with the computer game; sending content of the first message to a first machine learning (ML) model; receiving first instructions from the first ML model representing processing of the content; sending data representative of user behavior associated with the first account to a second ML model; receiving second instructions from the second ML model representing processing of the data; performing a corrective action on the first account at least in part in response to the first and second instructions; receiving at least a second message from a second account associated with the computer game; sending the content of the second message to the first ML model; receiving third instructions from the first ML model representing processing of the content of the second message; sending data representing user behavior associated with the second account to the second ML model; receiving fourth instructions from the second ML model representing processing of the data representing user behavior associated with the second account; and and in response at least in part to the third and fourth instructions, not to implement a corrective action against the second account.

17. The assembly of claim 16 , wherein the processor is a component of a computer game console.

18. The assembly of claim 16 , wherein the processor is a component of a computer game controller.

19. The assembly of claim 16 , wherein the processor is a component of a computer game server.

20. 17. The assembly of claim 16, wherein the processor is programmed to provide the first and second instructions to a combination module and to implement the corrective action based on an output of the combination module.

Images