Text overall risk identification method and system, electronic device and medium

Abstract

The present application relates to a kind of text overall risk identification method, system, electronic equipment and medium.Method includes: obtaining to be identified text and the behavior data of the user publishing the to be identified text;Behavior data includes entity information and scene information;According to entity information and scene information, determine the contribution degree of each entity to text overall risk in corresponding scene;According to entity information and scene information, determine the anomaly probability of each entity in corresponding scene;Risk degree of behavior data is determined based on contribution degree, the anomaly probability of each entity in corresponding scene and the weight of each entity in text overall risk;Determine the anomaly probability of to be identified text;The risk degree of to be identified text is determined based on the anomaly probability of to be identified text and the weight of to be identified text in text overall risk;Text overall risk is determined based on the risk degree of behavior data and the risk degree of to be identified text.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a text overall risk identification method and system, an electronic device and a medium. BACKGROUND

[0002] In the game industry, there are various important factors affecting the game ecology. These scenarios usually occur in user-generated content (UGC) such as chat channels, announcements, schedules, nicknames, signatures, in-game emails, forums, etc.

[0003] In the related art, an API interface is generally used to detect content, and methods such as natural language processing (NLP) are used to detect text abnormalities. However, this method has a high cost of resistance due to the complexity of text transformation, and therefore the effect is not significant. In addition, in order to detect text abnormalities, most cloud vendors only provide common solutions, and the individuality of the game is not high, and it is not possible to respond in a timely manner to game slang. SUMMARY

[0004] The main purpose of the present application is to provide a text overall risk identification method and system, an electronic device, a medium and a computer program product, which not only considers the abnormal probability of the text itself, but also considers the behavior data of the user, and by comprehensively calculating the joint probability of a statement, the identification probability of malicious text and the retention rate of players can be improved.

[0005] The first aspect of the present application discloses a text overall risk identification method, the method comprising: obtaining a text to be identified and behavior data of a user publishing the text to be identified; the behavior data comprising entity information and scenario information; determining the contribution degree of each entity to the overall risk of the text in the corresponding scenario according to the entity information and the scenario information; determining the abnormal probability of each entity in the corresponding scenario according to the entity information and the scenario information; determining the risk degree of the behavior data based on the contribution degree, the abnormal probability of each entity in the corresponding scenario and the weight of each entity in the overall risk of the text; determining the abnormal probability of the text to be identified; determining the risk degree of the text to be identified based on the abnormal probability of the text to be identified and the weight of the text to be identified in the overall risk of the text; and determining the overall risk of the text based on the risk degree of the behavior data and the risk degree of the text to be identified.

[0006] In a possible implementation of the above first aspect, the contribution degree of each entity to the overall risk of the text in the corresponding scenario is determined by an indicator function.

[0007] In a possible implementation of the first aspect, the method comprises determining an abnormal probability of each entity in a corresponding scenario according to the relevance of each entity and different scenarios.

[0008] In a possible implementation of the first aspect, the overall risk of the text P(X) is, wherein X is a sample of the text to be identified, (E i ,S j ), i∈{1,…,n} and j∈{1,…,m} are the corresponding scenarios of each entity, i.e. (entity, scenario) pairs, is a feature set under the (entity, scenario) pair, is an abnormal probability of entity i in scenario j, f i is a relevance abnormal probability of entity i in different scenarios, I i,j (E i ,S j ) is a contribution degree of entity i in scenario j to the overall risk of the text, ω i is a weight of each entity i in the overall risk of the text, X t is a text feature of the text to be identified, P(X t ) is an abnormal probability of the text to be identified, ω t is a weight of the text to be identified in the overall risk of the text.

[0009] In a possible implementation of the first aspect, the weight of each entity in the overall risk of the text and the weight of the text to be identified in the overall risk of the text are derived from positive and negative samples of all users.

[0010] In a possible implementation of the first aspect, subsequent processing measures are performed on the text to be identified based on the overall risk of the text.

[0011] The second aspect of the present application discloses a text overall risk identification system, the system comprising: an acquisition module configured to acquire a text to be identified and behavior data of a user publishing the text to be identified; the behavior data comprising entity information and scene information; a first determination module configured to determine a contribution degree of each entity to the text overall risk in a corresponding scene according to the entity information and the scene information; a second determination module configured to determine an abnormal probability of the each entity in the corresponding scene according to the entity information and the scene information; a third determination module configured to determine an abnormal probability of the text to be identified; a fourth determination module configured to determine a weight of an identification probability of each (entity, scene) pair and a weight of an identification probability of the text to be identified; and a fifth determination module configured to determine the text overall risk according to the abnormal probability of the each entity in the corresponding scene, the weight of the each entity in the text overall risk, the abnormal probability of the text to be identified, and the weight of the text to be identified in the text overall risk.

[0012] In a possible implementation of the second aspect, the system further comprises a processing module configured to perform subsequent processing measures on the text to be identified based on the text overall risk.

[0013] The third aspect of the present application discloses an electronic device comprising a memory storing computer executable instructions and a processor; when the instructions are executed by the processor, the device implements the method of the first aspect of the present application.

[0014] The fourth aspect of the present application discloses a computer readable storage medium storing one or more computer programs, the one or more computer programs being executed by one or more processors to cause the processors to execute the method of the first aspect of the present application.

[0015] The fifth aspect of the present application discloses a computer program product comprising a computer program, the computer program being executed by a processor to implement the method of the first aspect of the present application.

[0016] According to the text overall risk identification method, system, electronic device, medium and computer program product of the present application, the joint probability of text abnormality is obtained by comprehensively considering the risk degree of the behavior data of the user and the text to be identified, and the filtering of abnormal text messages is expanded to include multiple features of player game behavior and specific text content. Not only the abnormal probability of the text itself is considered, but also the behavior data of the user is considered, so that personalized game slang and seemingly normal dangerous speeches of abnormal players can be identified. The joint probability of a speech is comprehensively calculated, so that the identification probability of malicious text is improved, and the retention rate of players is improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0018] Figure 1 is a method for identifying malicious text in the related art;

[0019] Figure 2 is a flowchart of a text overall risk identification method according to an embodiment of the present application;

[0020] Figure 3 is a structural diagram of an electronic device according to an embodiment of the present application;

[0021] Figure 4 is a module diagram of a text overall risk identification system according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The present application will be further described below in conjunction with specific embodiments and drawings. It can be understood that the illustrative embodiments of the present disclosure include but are not limited to a text overall risk identification method, system, electronic device, medium and computer program product. The specific embodiments described herein are merely intended to explain the present application, and not to limit the present application. In addition, only the parts related to the present application are shown in the drawings, not all the structures or processes.

[0023] The specific embodiments will be described below to illustrate the implementation of the present application, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the specification. Although the description of the present application will be introduced in conjunction with the preferred embodiments, this does not mean that the features of the present application are limited to this implementation. On the contrary, the purpose of introducing the present application in conjunction with the implementation is to cover other options or modifications that can be extended based on the claims of the present application. In order to provide a deep understanding of the present application, many specific details will be included in the following description. The present application can also be implemented without using these details. In addition, in order to avoid confusion or obscure the focus of the present application, some specific details will be omitted in the description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0024] In addition, various operations will be described as a number of discrete operations; however, the order of the description should not be interpreted as implying that these operations must be performed in the order presented. In particular, these operations need not be performed in the order presented.

[0025] The terms "comprise", "have" and "include" are synonymous, unless the context dictates otherwise. The phrase "A / B" means "A or B". The phrase "A and / or B" means "(A and B) or (A or B)".

[0026] As used herein, the terms "module" or "unit" can refer to, be or include an Application Specific Integrated Circuit (ASIC), an electronic circuit, a processor and / or memory (shared, dedicated or group) that execute one or more software or firmware programs, a combinational logic circuit, and / or other suitable components that provide the described functionality.

[0027] In some cases, the disclosed embodiments can be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments can also be implemented as instructions carried by or stored on a transitory or non-transitory machine-readable (e.g., computer-readable) storage medium, which can be read and executed by one or more processors. For example, the instructions can be distributed over the network or by way of other computer readable media. Thus, a machine-readable medium can include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including without limitation floppy disks, optical disks, optical disks, CD-ROMs, magnetic cassettes or tapes, ROMs, RAMs, erasable programmable ROMs (EPROMs), electrically erasable programmable ROMs (EEPROMs), magnetic or optical cards, flash memories, or any other suitable device for storing or transmitting this information in a form that can be read by a machine. Thus, a machine-readable medium includes any type of media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0028] In the drawings, some structural or methodological features are shown in particular arrangements and / or orders. However, it should be understood that such particular arrangements and / or orders can not be required. In some embodiments, features can be arranged differently than shown in the illustrative figures. Also, including a structural or methodological feature in a particular figure does not imply that the feature is required in all embodiments, and in some embodiments, the feature can not be included or can be combined with other features.

[0029] It should be understood that although the terms "first", "second", etc. can be used herein to describe various elements or data, these elements or data should not be limited by these terms. These terms are only used to distinguish one element or data from another element or data. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the scope of the example embodiments.

[0030] It should be noted that in the present specification, similar reference numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0031] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0032] In the game industry, there are hidden dangers affecting content security in various places related to text, such as chat, mail, nickname, signature, avatar, space, community, forum, official website, etc. How to efficiently and accurately identify and handle various hidden dangers affecting content security in the game, optimize the game ecological environment, and improve the game experience of players has always been one of the core problems faced by game developers for a long time.

[0033] Figure 1 A method for identifying malicious text in the related art is shown. The text recognition method in the related art uses a natural language understanding algorithm (NLP) to determine whether the text is abnormal through an application programming interface (API) for user content UGC scenarios. However, for text with complex transformed variants, such as abbreviations, homophones, Chinese character splitting and combination, nicknames, transliteration, etc., the cost of resistance is high, and the recognition effect is not significant. In addition, the text recognition method in the related art mainly solves the malicious text caused by general reasons, and cannot handle the game slang with personalized features in time. Common game slang includes "selling fruits" and "out of marble" in resource transactions, which are difficult to be identified as malicious text. In addition, abnormal players will send exploratory statements that are indistinguishable from normal text, such as "hello" or "are you there", but the purpose of these statements is actually to attract people, which will cause user loss of the game platform. These exploratory statements use more information of the text itself in the malicious text recognition, making it impossible to effectively determine the malicious text, resulting in omissions.

[0034] To solve the above problems, one embodiment of the present application provides a text overall risk identification method 100, as shown in Figure 2 The method 100 includes:

[0035] S110, obtaining a to-be-identified text and behavior data of a user publishing the to-be-identified text; the behavior data includes entity information and scene information.

[0036] According to the associated information of the text in the user created content (UGC), the user publishing the text can be determined, and then the behavior data information of the user can be obtained. In the game industry, the behavior data of the user can be obtained from the behavior log containing the historical behavior of the player. The behavior data includes entity information and scene information at the time of publishing the text. The entity information includes account, role, IP, IP segment, device, guild, etc. The scene information is the application scene associated information of the entity (i.e. game role or device operating the game role), including installing application, creating account, logging in, recharging, private chatting, world chatting, etc.

[0037] In S120, the contribution degree of each entity to the overall risk of the text in the corresponding scene is determined according to the entity information and the scene information.

[0038] In the behavior data information of the user, although multiple entities and multiple scenes are included, not all entities in all scenes will contribute to the probability of the overall risk of the text. For example, when the IP of the obtained to-be-identified text is a local area network IP, rather than a public network IP; or the current account is an international server user, then the entity corresponding to the IP will not contribute to the overall risk of the text. Or, when the to-be-identified text is in the private chat of the account, rather than in the world chat, the behavior of the account publishing messages in the world chat will not contribute to the overall risk of the text.

[0039] In some embodiments, the contribution degree of each entity to the overall risk of the text in the corresponding scene can be derived by the indicator function I i,j (E i ,S j ).

[0040]

[0041] (E i ,S j ) is the each entity in the corresponding scene, and {(E,S)}κ is the entity and scene set effective in the application scene κ. The application scene usually refers to a specific game area server or a specific function play. In the present application, (E i ,S j ) is called a (entity, scene) pair, such as a (role, novice guide) pair when the role enters the novice guide process in the game. For example, when the IP of the obtained to-be-identified text is a public network IP, the entity corresponding to the IP will contribute to the overall risk of the text, I i,j (E i ,S j ) = 1. When the IP of the obtained to-be-identified text is a local area network IP; or the current account is an international server user, then the entity corresponding to the IP will not contribute to the overall risk of the text, Ii,j (E i ,S j = 0. When the text to be identified is in a private chat of an account, then the corresponding entity's behavior of posting a message in the world chat will not contribute to the overall risk of that text. i,j (E i ,S j ) = 0.

[0042] S130, determine the anomaly probability of each entity in the corresponding scenario based on the entity information and the scenario information.

[0043] The anomaly probability of each entity in the corresponding scenario can be calculated using various models / algorithms and manually constructed rules. Algorithms for calculating anomaly probabilities include community detection algorithms and clustering algorithms. In one example, when the entity is a character and the scenario is a tutorial, the time required for each step of the tutorial can be used as a feature. Then, a clustering algorithm can be used to cluster all characters into three clusters, i.e., they are grouped into three classes. Each cluster will contain some characters known to be anomalous. Therefore, we take the proportion of anomalous characters in the cluster of a given character as the anomaly probability of that character in the tutorial scenario, i.e., the anomaly probability of the (character, tutorial) pair. In some examples, the clustering algorithm is the DBSCAN algorithm. DBSCAN is a density-based spatial clustering algorithm based on noise. This algorithm divides regions with sufficient density into clusters and discovers clusters of arbitrary shapes in a noisy spatial database. It defines a cluster as the largest set of density-connected points.

[0044] S140, the risk level of the behavioral data is determined based on the degree of contribution, the abnormal probability of each entity in the corresponding scenario, and the weight of each entity in the overall risk of the text.

[0045] Each entity has a different weight in the overall risk of the text. The risk level of the behavioral data can be obtained by multiplying the weight, contribution level, and anomaly probability of each entity in the corresponding scenario. For example, if entity 1 has a weight of ω1 in the overall risk of the text, its contribution level to the overall risk of the text in the corresponding scenario P1 is 1, and its anomaly probability in the corresponding scenario is P1, then the risk level of entity 1 in the corresponding scenario is ω1*1*P1; and so on, the weight of entity n in the overall risk of the text is ω n The entity in the corresponding scenario P nThe contribution degree of the text overall risk is 1, and the abnormal probability of the entity in the corresponding scene is P1, so the risk degree of the entity in the corresponding scene is ω n *1*P n After calculating the risk degree of all entities in the corresponding scene, the risk degree of the behavior data is obtained by adding ω1*1*P1+…+ω n *1*P n .

[0046] In some embodiments, the weight of each entity in the overall risk of the text is obtained by all positive and negative samples of users. The historical accumulation of positive and negative samples can be obtained from the behavior data of all users. In some examples, the risk of spam text is a positive sample, and the normal text is a negative sample. For example, when calculating the weight of the abnormal probability of the (role, novice guide) pair, according to the historical accumulation of the abnormal probability of the (role, novice guide) pair>0.5, the speech samples are manually annotated according to the preset sample library, or the computer program is annotated, and finally the proportion w of abnormal speech in the speech samples is obtained. As the weight of the abnormal probability of this (entity, scene) pair. Each weight needs to be divided by the sum of all weights to ensure that the overall risk value of the text calculated is between 0 and 1.

[0047] In some embodiments, the abnormal probability of each entity in the corresponding scene can be determined according to the relevance of each entity and different scenes.

[0048] The abnormal risk of different (entity, scene) pairs may be related. For example, a certain device entity is identified as an abnormal risk when creating an account, so the account entity in the device will also have an abnormal risk when logging in. Therefore, when calculating the overall risk of the text, the risk probabilities cannot be directly added or multiplied. In some examples, a f function can be introduced to consider the influence of relevance. The f function can be a normal aggregation function, such as sum, max, avg, etc. In other examples, the f function can be a statistical function such as Principal Component Analysis (PCA) and Ridge Regression.

[0049] S150, determining the abnormal probability of the text to be identified.

[0050] The abnormal probability of the text to be identified can also be calculated by various models / algorithms or manually constructed rules. Algorithms for calculating the abnormal probability of the text include fastText algorithm and XGBoost algorithm. FastText is an efficient learning algorithm for word representation and sentence classification, which is an algorithm mastered by those skilled in the art. The fastText model is a three-layer network structure, including an input layer, a single hidden layer and an output layer. FastText belongs to supervised learning and predicts the category of the text through context. XGBoost (eXtreme Gradient Boosting) algorithm is a gradient boosting decision tree algorithm, which is an algorithm mastered by those skilled in the art. The XGBoost algorithm uses the gradient descent algorithm to minimize the loss when adding a new model, that is, continuously adding trees to fit the residual of the last prediction, and finally adding the scores of each tree to obtain the predicted value of the sample. In some examples, the text data is taken as a sample, the abnormal text in the sample is labeled, the fastText algorithm is used to classify the text, and the probability that the text is abnormal is output. The algorithm for calculating the abnormal probability of the text can identify variant words in the text. In some embodiments, the algorithm for calculating the abnormal probability of the text can identify variant words by establishing a mapping table of variant words or using word vectors to compare the contextual relevance of variant words and context. By establishing a mapping table, i.e., manually constructing rules, game operators define texts with abnormal probability and corresponding probabilities.

[0051] S160, determining the risk degree of the text to be identified based on the abnormal probability of the text to be identified and the weight of the text to be identified in the overall risk of the text.

[0052] The weight of the text to be identified in the overall risk of the text and the abnormal probability of the text to be identified can be multiplied to obtain the risk degree of the text to be identified.

[0053] In some embodiments, the weight of the text to be identified in the overall risk of the text is derived from positive and negative samples of all users. The accumulated positive and negative samples can be derived from the behavior data of all users. In some examples, the risk text is a positive sample and the normal text is a negative sample. In some embodiments, the proportion w of abnormal utterances in the utterance samples can be manually annotated according to the accumulated abnormal probability greater than 0.5, or annotated by a computer program according to a preset sample library, to obtain the weight. Finally, each weight needs to be divided by the sum of all weights to ensure that the overall risk value of the text calculated is between 0 and 1.

[0054] S170, determining the overall risk of the text based on the risk degree of the behavior data and the risk degree of the text to be identified.

[0055] The risk degree of the behavior data calculated in step S140 and the risk degree of the text to be identified calculated in step S160 are added to obtain the overall risk of the text.

[0056] In some embodiments, if only the text to be identified is obtained in the sample but no behavior data of the user publishing the text to be identified, the risk degree of the behavior data can be considered as zero, and the overall risk of the text is directly calculated according to steps S150-S160.

[0057] Figure 2 The overall risk of the text in the formula comprehensively considers the risk degree of the behavior data of the user and the risk degree of the text to be identified to obtain the joint probability of the text anomaly, and expands the filtering of only abnormal text messages to include various features of the game playing behavior (such as the frequency of speaking, online time, progress of various growth lines, etc.), player attributes (such as character level, cumulative payment, etc.), environmental information (such as login device, registration IP, etc.), and specific text content. Not only the abnormal probability of the text itself is considered, but also the behavior data of the user is considered. The personalized game slang of the abnormal player and the seemingly normal dangerous speech of the abnormal player can be identified. The joint probability of a speech is calculated by using the entity information at the time of the speech, the abnormal probability of the entities in various scenes, and the overall risk of the text, which can improve the identification probability of malicious text and improve the retention rate of players.

[0058] In some embodiments, after determining the contribution degree I i,j (E i ,S j ) of each entity to the overall risk of the text in the corresponding scene and the relevance of each entity and different scenes, Figure 2 The overall risk P(X) of the text to be identified in the formula can be represented by the following formula:

[0059]

[0060] X is the sample of the text to be identified, (E i ,S j ), i∈{1,…,n} and j∈{1,…,m} are each entity in the corresponding scene, i.e., the (entity, scene) pair, is the feature set under the (entity, scene) pair, is the abnormal probability of entity i in scene j, f i is the associated abnormal probability of entity i in different scenes, I i,j (E i ,S j) the contribution degree of entity i to the overall risk of the text under scene j, ω i the weight of each entity i in the overall risk of the text, X t the text feature of the text to be identified, i.e., the specific text content of the current utterance, P(X t ) the abnormal probability of the text to be identified, ω t the weight of the text to be identified in the overall risk of the text.

[0061] In formula (2), the text sample X to be identified has a feature set under the (entity, scene) pair and the text feature X of the text to be identified t , i.e., formula (2) comprehensively considers the behavior data of the user corresponding to the current text and the text itself.

[0062] In some embodiments, the text sample X to be identified can include a game ID, a timestamp of the utterance, an IP address of the utterance, an account, a text type, text content, etc. In some embodiments, X can also include one or more of the following: an IP address segment of the utterance, a device ID, a device operating system, a game server ID, a player nickname, a player guild ID, a message object type, a message object ID, a player game level, and a player push progress.

[0063] The (entity, scene) pair (E i ,S j ) refers to the entity E i in the sample X appearing in the scene S j , such as (device, installation), entity E i is a device, and scene S j is installation. In some examples, the (entity, scene) pair can also be (account, login), (role, novice guide), (IP, world chat), etc.

[0064] Each (entity, scene) pair (E i ,S j ) corresponds to a feature set under the (entity, scene) pair For example, under (Device, Installation), the feature set may include: the number of times the device has installed the application in the past 24 hours, and the number of accounts logged in after the application was installed. Under (Account, Login), the feature set may include: the maximum character level under the account, the account creation time, and the account online time. Under (Character, New Player Guide), the feature set may include: the character's latest level, the character's cumulative paid value, the character's login count in the past 24 hours, the time it took to complete the new player guide, and whether the character joined a guild when logging in. Under (IP, World Chat), the feature set may include: the speaking frequency of the character corresponding to the IP in world chat, and the online time of the character corresponding to the IP.

[0065] As mentioned above, (entity, scene) pairs (E) can be determined based on sample X. i ,S j Feature set under ) In some implementations, the feature set can be The input is fed into the corresponding (entity, scene) pair model to obtain the anomaly probability of entity i in scene j. In some implementations, each (entity, scenario) pair model can be trained based on the historical behavior data of all users, which is recorded in the user behavior log.

[0066] As can be seen from formula (2), the overall text risk combines the risk levels of both the user's behavioral data and the text to be identified. It can be determined based on the current speech sample using the indicator function I. i,j (E i ,S j ) Retrieve the effective (entity, scenario) pair (E) i ,S j Then, the feature set under (entity, scene) pairs. Inputting these values ​​into each (entity, scene) pair model yields the anomalous probability of entity i in scene j. Using f i The function determines the probability of entity i's association with anomalies in different scenarios. Specifically, by utilizing the entity's information at the time of the statement and the probability of these entities exhibiting anomalies in various scenarios, it can identify personalized game slang and seemingly normal but dangerous statements from abnormal players. Furthermore, by assessing the risk level of the text to be identified, variant words can be identified, improving the accuracy of the recognition.

[0067] Compared with the filtering method only for text, the overall risk calculation method in formula (2) can reduce the proportion of advertisements in the world chat from 99% to 0 and reduce the proportion of in-game messages from more than 95% to less than 5%. On the other hand, the player chat activity can be improved by 1.5-2.5 times, the 30-day retention rate of the domestic version can be improved by 2% (from 7% to 9%), and the identification recall rate of the in-game message can be improved by 5%.

[0068] In one example, the speech sample X is composed of (role ID, account ID, timestamp, device ID, IP, text), and according to the indicator function, the effective (entity, scenario) pair is (account, login) and (role, novice guide). For (account, login), the feature set is: the maximum role level of the account, and whether the account has social risk. For (role, novice guide), the feature set is: the latest level of the role, the cumulative payment of the role, the login frequency of the role in the past 24 hours, the time of passing the role novice guide, and whether the role joins the guild when logging in. The abnormal risk of the account is p1= The weight of the account in the overall risk of the text is ω1; the abnormal risk of the role is p2= The weight of the role in the overall risk of the text is ω2. The probability of text anomaly is p t , and the weight of the account in the overall risk of the text is ω t . The abnormal risk p1 of the account is obtained by inputting the speech sample into the (account, login) model, and the abnormal risk p2 of the role is obtained by inputting the speech sample into the (role, novice guide) model. The probability P(X t ) of text anomaly is calculated by the text model. The weights ω1, ω2, and ω t can be obtained from all positive and negative samples of users. The overall risk of the speech sample X is:

[0069]

[0070] In some embodiments, the abnormal probability p1 of the speech sample X in the (account, login) scenario pair is 0.6, the weight ω1 is 0.3, the abnormal probability p2 in the (role, novice guide) scenario pair is 0.8, and the weight ω2 is 0.3. The probability P(X t ) of text content being determined as abnormal by the text classification model is 1.0, and the weight ω t is 0.4. Therefore, the overall risk P(X) of the speech sample is: 0.6*0.3+0.8*0.3+1.0*0.4=0.82.

[0071] In some embodiments, subsequent processing measures can be performed on the identified text based on the text overall risk. For example, the calculated text overall risk can be graded. In some examples, the calculated probability * 4 can be rounded to a risk score from 0 to 4, and a score of 3 or above can require human intervention.

[0072] In some embodiments, after grading the text overall risk, specific abnormal reasons can be returned for each (entity, scene). The abnormal reasons can be using simulators, bot (machine-generated) nicknames, etc.

[0073] In some embodiments, the abnormal text can be released or directly intercepted for processing after verification by the verification code according to the grading. In other embodiments, the corresponding specific processing method can also be configured by the development party of the game application in the front end. The processing of the abnormal text only affects the current statement at the timestamp, and the timeliness can be achieved within seconds. Therefore, the timeliness of the data can be effectively utilized, and only the current text is processed, reducing the rate of false injury.

[0074] In some embodiments, for the entity information at the time of the statement, the probability of abnormality in each scene can be calculated or predicted in an offline state, and then written into a label table corresponding to each entity. When calculating the text overall risk of the statement, the label table and additional information such as the statement text at the time of the statement can be combined to calculate the joint probability.

[0075] The text overall risk of long text can be calculated using the same technical architecture. The calculation method of the text overall risk of long text can be applied to risk determination of different games, even different scenes (such as personal mail, private chat, announcement, nickname, alliance schedule, etc.), to identify abnormal long text.

[0076] Reference is now made to Figure 3 , which shows a block diagram of an electronic device 300 according to an embodiment of the present application. The electronic device 300 can include one or more processors 302, a system motherboard 308 connected to at least one of the processors 302, a system memory 304 connected to the system motherboard 308, a non-volatile memory (NVM) 306 connected to the system motherboard 308, and a network interface 310 connected to the system motherboard 308.

[0077] The processor 302 can include one or more single-core or multi-core processors. The processor 302 can include any combination of general-purpose processors and dedicated processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments herein, the processor 302 can be configured to perform operations according to, for example, Figure 2One or more embodiments of the various embodiments shown.

[0078] In some embodiments, system motherboard 308 can include any suitable interface controllers to provide any suitable interface(s) to at least one of processors 302 and / or any suitable device or component in communication with system motherboard 308.

[0079] In some embodiments, system motherboard 308 can include one or more memory controllers to provide an interface to system memory 304. System memory 304 can be used to load and store data and / or instructions. In some embodiments, memory 304 of device 300 can include any suitable volatile memory, such as suitable dynamic random access memory (DRAM).

[0080] NVM / memory 306 can include one or more tangible, non-transitory computer- readable media for storage of data and / or instructions. In some embodiments, NVM / memory 306 can include any suitable non-volatile memory, such as flash memory, and / or any suitable non-volatile storage device, such as at least one of a hard disk drive (HDD), a compact disc (CD) drive, a digital versatile disc (DVD) drive.

[0081] NVM / memory 306 can include a portion of storage resident on a device of device 300, or it can be accessible by the device but not necessarily part of the device. For example, NVM / memory 306 can be accessed over a network via network interface 310.

[0082] In particular, system memory 304 and NVM / memory 306 can include, respectively, a temporary copy and a permanent copy of instructions 320. Instructions 320 can include instructions that, when executed by at least one of processors 302, cause device 300 to implement a method as shown. In some embodiments, instructions 320, hardware, firmware, and / or software components thereof can additionally / alternatively be disposed in system motherboard 308, network interface 310, and / or processors 302. Figure 2

[0083] ​The network interface 310 can include a transceiver to provide a radio interface for the device 300 to communicate with any other suitable device (e.g., a front end module, an antenna, etc.) over one or more networks. In some embodiments, the network interface 310 can be integrated with other components of the device 300. For example, the network interface 310 can be integrated with at least one of the processor 302, the system memory 304, the NVM / memory 306, and a firmware device (not shown) having instructions that, when executed by at least one of the processors 302, enable the device 300 to implement Figure 2 one or more embodiments of the various embodiments shown.

[0084] The network interface 310 can further include any suitable hardware and / or firmware for providing a multiple-input multiple-output radio interface. For example, the network interface 310 can be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0085] In one embodiment, at least one of the processors 302 can be packaged together with logic for one or more controllers of the system motherboard 308 to form a system in a package (SiP). In one embodiment, at least one of the processors 302 can be integrated on the same die with logic for one or more controllers of the system motherboard 308 to form a system on a chip (SoC).

[0086] The device 300 can further include an input / output (I / O) device 312. The I / O device 312 can include a user interface that enables a user to interact with the device 300; a peripheral component interface that enables peripheral components to interact with the device 300.

[0087] In some embodiments, the user interface can include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light emitting diode flash), and a keypad.

[0088] In some embodiments, the peripheral component interface can include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0089] The various method embodiments of the present application can be implemented in software, firmware, and / or hardware.

[0090] The program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a system for processing instructions, including the processor 302, includes any system that has a processor, such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0091] The program code can be implemented in a high-level procedural or object-oriented programming language to communicate with a processing system. The program code can also be implemented in assembly or machine language, if desired. In fact, the mechanisms described herein are not limited in scope to any particular programming language. In any case, the language can be a compiled or interpreted language.

[0092] One or more aspects of at least one embodiment can be implemented by representative instructions stored on a computer-readable storage medium that represents various logic within the processor, which when read by a machine causes the machine to fabricate logic to perform the techniques described herein. Such representations, known as "IP cores" can be stored on a tangible, computer readable storage medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that make the logic or processor.

[0093] Reference Figure 4 , Figure 4 A schematic diagram of a module of a text overall risk identification system in an embodiment of the present application is shown. The system 400 includes:

[0094] An acquisition module 410 is configured to acquire a text to be identified and behavior data of a user publishing the text to be identified; the behavior data includes entity information and scene information;

[0095] A first determination module 420 is configured to determine a contribution degree of each entity to the overall risk of the text in a corresponding scene according to the entity information and the scene information;

[0096] A second determination module 430 is configured to determine an abnormal probability of each entity in a corresponding scene according to the entity information and the scene information;

[0097] A third determination module 440 is configured to determine an abnormal probability of the text to be identified;

[0098] A fourth determination module 450 is configured to determine a weight of an identification probability of each (entity, scene) pair and a weight of an identification probability of the speech text;

[0099] The fifth determining module 460 is configured to determine the overall risk of the text according to the abnormal probability of each entity in the corresponding scene, the weight of each entity in the overall risk of the text, the abnormal probability of the text to be identified, and the weight of the text to be identified in the overall risk of the text.

[0100] The text overall risk identification system 400 in the present application Figure 4 The text overall risk identification system 400 in the present application

[0101] In some embodiments, the text overall risk identification system 400 further comprises a processing module configured to perform subsequent processing measures on the text to be identified based on the overall risk of the text.

[0102] The present application further provides a computer readable medium, wherein the storage medium stores one or more programs, and the one or more programs are executable by one or more processors to implement the text overall risk identification method in one or more embodiments.

[0103] One embodiment of the present application discloses a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the text overall risk identification method in one or more embodiments.

[0104] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it should be understood that various changes in form and detail can be made therein without departing from the spirit and scope of the application.

Claims

1. A method for identifying overall textual risk, characterized in that, The method includes: Acquire the text to be identified and the behavioral data of the user who published the text; the behavioral data includes entity information and scene information. Based on the entity information and the scenario information, the contribution of each entity to the overall risk of the text in the corresponding scenario is determined, and the contribution level is determined based on the following formula: Among them, (E) i ,S j ) represents an entity-scene pair, indicating that each entity is in the corresponding scene, where E i S represents an entity. j Let {(E,S)}κ represent the scenario, which is the set of entities and scenarios that are effective under the application scenario κ. The value range of i is 1 to n, and the value range of j is 1 to m. The anomaly probability of each entity in the corresponding scenario is determined based on the entity information and the scenario information. The risk level of the behavioral data is determined by multiplying the contribution level, the anomaly probability of each entity in the corresponding scenario, and the weight of each entity in the overall risk of the text. The abnormal text in the text to be identified is labeled, and the text to be identified is classified using an artificial intelligence algorithm to determine the probability of abnormality of the text to be identified; wherein, the artificial intelligence algorithm includes the fastText algorithm; The risk level of the text to be identified is determined based on the anomaly probability of the text to be identified and the weight of the text to be identified in the overall risk of the text. The overall risk of the text is determined based on the risk level of the behavioral data and the risk level of the text to be identified, and the overall risk P(X) of the text is determined based on the following formula. in, X is a sample of the text to be recognized. The feature set of the entity scene pair. f represents the probability of entity i being an anomaly in scene j. i Let I be the probability of association anomalies of entity i in different scenarios. i,j (E i ,S j ω represents the contribution of entity i to the overall risk of the text in scenario j. i X represents the weight of each entity in the overall risk of the text. t For the text features of the text to be identified, P(X) t ω represents the anomaly probability of the text to be identified. t The weight of the text to be identified in the overall risk of the text.

2. The method according to claim 1, characterized in that, The contribution of each entity to the overall risk of the text in the corresponding scenario is determined by the indicator function.

3. The method according to claim 2, characterized in that, The method includes determining the probability of an anomaly of each entity in the corresponding scenario based on the correlation between each entity and different scenarios.

4. The method according to claim 1, characterized in that, The weight of each entity in the overall risk of the text and the weight of the text to be identified in the overall risk of the text are derived from positive and negative samples from all users.

5. The method according to claim 1, characterized in that, Based on the overall risk of the text, subsequent processing measures are performed on the text to be identified.

6. A text-based overall risk identification system, characterized in that, The system includes: The acquisition module is used to acquire the text to be identified and the behavioral data of the user who published the text; the behavioral data includes entity information and scene information; The first determining module is used to determine the contribution degree of each entity to the overall risk of the text in the corresponding scenario based on the entity information and the scenario information, and to determine the contribution degree based on the following formula: Among them, (E) i ,S j ) represents an entity-scene pair, indicating that each entity is in the corresponding scene, where E i S represents an entity. j Let {(E,S)}κ represent the scenario, which is the set of entities and scenarios that are effective under the application scenario κ. The value range of i is 1 to n, and the value range of j is 1 to m. The second determining module is used to determine the anomaly probability of each entity in the corresponding scenario based on the entity information and the scenario information. The third determining module is used to label the abnormal text in the text to be identified, and to classify the text to be identified using an artificial intelligence algorithm to determine the probability of abnormality of the text to be identified; wherein, the artificial intelligence algorithm includes the fastText algorithm; The fourth determining module is used to determine the weights of the recognition probability of the entity scene pair and the recognition probability of the spoken text. The fifth determining module is used to determine the overall text risk based on the anomaly probability of each entity in the corresponding scenario, the weight of each entity in the overall text risk, the anomaly probability of the text to be identified, and the weight of the text to be identified in the overall text risk, and to determine the overall text risk P(X) based on the following formula. in, X is a sample of the text to be recognized. The feature set of the entity scene pair. f represents the probability of entity i being an anomaly in scene j. i Let I be the probability of association anomalies of entity i in different scenarios. i,j (E i ,S j ω represents the contribution of entity i to the overall risk of the text in scenario j. i X represents the weight of each entity in the overall risk of the text. t For the text features of the text to be identified, P(X) t ω represents the anomaly probability of the text to be identified. t The weight of the text to be identified in the overall risk of the text.

7. The system according to claim 6, characterized in that, The system also includes a processing module for performing subsequent processing measures on the text to be identified based on the overall risk of the text.

8. An electronic device, characterized in that, The device includes a memory storing computer-executable instructions and a processor; when the instructions are executed by the processor, the device performs the text-based risk identification method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the text-based risk identification method according to any one of claims 1 to 5.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the text-based risk identification method as described in any one of claims 1 to 5.

Images