A method, medium, equipment, and program product for detecting illegal content.

Abstract

This application provides a method, medium, device, and program product for detecting illegal content, relating to the field of data detection. The method includes: acquiring currently published content of a target user account in a target application and an initial violation detection result for the currently published content; the initial violation detection result is obtained by performing a first content violation detection on the currently published content; determining the account violation risk level of the target user account; the account violation risk level is determined based on historical application data generated by the target user account in the target application; and performing a second content violation detection based on the account violation risk level and the initial violation detection result to obtain a content violation detection result for the currently published content. This application avoids the adversarial detection challenges faced when only performing content violation detection on currently published content, thus improving the accurate recall rate of illegal content.

Description

Technical Field

[0001] This application relates to the field of data detection, and in particular to a method, medium, device, and program product for detecting illegal content. Background Technology

[0002] Currently, content detection methods are primarily based on vertical category algorithms, simply identifying content violations from a spatial perspective. However, with increasingly sophisticated countermeasures against violating content, these vertical category algorithms are facing growing challenges in identification, leading to a bottleneck in accuracy and low overall performance. Improvements in performance are not significant. Summary of the Invention

[0003] The purpose of this application is to provide a method for detecting illegal content, a computer-readable storage medium, an electronic device, and a computer program product that can perform comprehensive content detection by combining the account violation risk level of a user account with the content violation detection results, thereby improving the accuracy of content detection and solving the problem of low accuracy in content detection caused by applying a single vertical category algorithm.

[0004] To address the aforementioned technical problems, this application provides a method for detecting illegal content, the specific technical solution of which is as follows:

[0005] Obtain the current published content of the target user account in the target application and the initial violation detection result of the current published content; the initial violation detection result is obtained by performing a first content violation detection on the current published content;

[0006] Determine the account violation risk level of the target user account; the account violation risk level is determined based on the historical application data generated by the target user account in the target application;

[0007] A second content violation detection is performed based on the account violation risk level and the initial violation detection result to obtain the content violation detection result for the currently published content.

[0008] Optionally, determining the account violation risk level of the target user account includes:

[0009] The first decision model is applied to determine the account violation risk level of the target user account;

[0010] The generation process of the first decision model includes:

[0011] Collect historical application data of several user accounts in the target application, and construct sample data containing the mapping relationship between the user accounts and the corresponding historical application data.

[0012] The sample data is divided into a first training set and a first test set according to a first predetermined ratio; wherein the first training set includes a training set and a validation set.

[0013] A first loss function is set for the first decision model, and an initial model is obtained by minimizing the first loss function in multiple rounds of training using the training set and the validation set; wherein the validation set is changed in turn in different rounds of training.

[0014] The initial model is validated using the validation set.

[0015] If the verification is successful, the initial model will be used as the first decision model.

[0016] Optionally, after collecting the account history application data of several user accounts in the target application, the method further includes:

[0017] Set default values ​​for missing content in the account's historical application data;

[0018] For outliers in the account's historical application data: if the outlier is a numerical outlier, delete the corresponding sample data; if the outlier is a non-numerical outlier, convert the account's historical application data into a vector and perform normalization.

[0019] Optionally, after collecting the account history application data of several user accounts in the target application, the method further includes:

[0020] Calculate the information entropy gain of a single feature in the account's historical application data; the single feature is the account's historical application data that can affect the detection results of illegal content.

[0021] Discarding single features whose information entropy gain does not meet the set value, and using the remaining single features as historical application data in the sample data to train the first decision model.

[0022] Optionally, calculating the information entropy gain of a single feature in the account's historical application data includes:

[0023] For any single target feature in the account's historical application data, the historical application data is used as the first sample dataset, and the account's historical application data after discarding the single target feature is used as the second sample dataset.

[0024] Calculate the first violation probability of the first sample dataset and the second violation probability of the second sample dataset, respectively;

[0025] The information entropy gain of the target single feature is calculated based on the difference between the first violation probability and the second violation probability.

[0026] Optionally, if the historical application data includes scenario behavior data, user historical content review data, and backend device environment security data, the scenario behavior data is used to determine user behavior profile data, the user historical content review data is used to determine user content profile data, and the backend device environment security data is used to determine device environment profile data. Determining the account violation risk level of the target user account includes:

[0027] If the user behavior profile data, the user content profile data, and the device environment profile data are all normal, the account violation risk level of the target user account is confirmed to be a white user, so that the risk level parameter corresponding to the white user can be used to participate in the second content violation detection.

[0028] If the user behavior profile data, the user content profile data, and the device environment profile data contain illegal content, and the frequency of violations is less than a set threshold, the account violation risk level of the target user account is confirmed as a regular black user, so that the risk level parameter corresponding to the regular black user can be used to participate in the second content violation detection.

[0029] If the user behavior profile data, the user content profile data, and the device environment profile data contain illegal content, and the frequency of violations is not less than the set threshold, the account violation risk level of the target user account is confirmed as a severely blacklisted user, so that the risk level parameter corresponding to the severely blacklisted user can be used to participate in the second content violation detection.

[0030] Optionally, a second content violation detection is performed based on the account violation risk level and the initial violation detection result to obtain the content violation detection result for the currently published content, including:

[0031] The account violation risk level and the initial violation detection result are input into the second decision model, and the output of the second decision model is used as the content violation detection result for the currently published content.

[0032] The training process of the second decision model includes:

[0033] Construct structured data containing the account history behavior feature levels and violation detection results of user accounts; wherein, the violation detection results are the violation rule hit results corresponding to the historical application data generated by the user account in the target application;

[0034] Set a second loss function for the second decision model, and divide the structured data into a second training set and a second test set according to a second set ratio;

[0035] Using the second training set and the second test set, the second decision model is trained with the goal of minimizing the second loss function.

[0036] Optionally, after performing a second content violation detection based on the account violation risk level and the initial violation detection result to obtain the content violation detection result for the currently published content, the method further includes:

[0037] The account violation risk level of the target user account is updated based on the content violation detection results of the currently published content.

[0038] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0039] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described above when it invokes the computer program in the memory.

[0040] This application also provides a computer program product, including a computer program that, when executed, implements the steps of the illegal content detection method described above.

[0041] This application provides a method for detecting illegal content, comprising: obtaining the currently published content of a target user account in a target application and the initial illegal detection result of the currently published content; the initial illegal detection result is obtained by performing a first content illegal detection on the currently published content; determining the account illegal risk level of the target user account; the account illegal risk level is determined based on the historical application data generated by the target user account in the target application; and performing a second content illegal detection based on the account illegal risk level and the initial illegal detection result to obtain a content illegal detection result for the currently published content.

[0042] When performing violation content detection, this application first obtains the current published content of the target user account in the target application and the initial violation detection results of the current published content. Then, it performs comprehensive content violation detection based on the account violation risk level of the target user account. Based on the initial violation detection results obtained from the preliminary content detection of the current published content, the account violation risk level reflects the historical violation situation of the target user account. The account violation risk level and the initial violation detection results are combined for comprehensive content violation detection, avoiding the violation content countermeasures faced when only performing content violation detection on the current published content, and improving the accurate recall rate of violation content.

[0043] This application also provides a computer-readable storage medium, an electronic device, and a computer program product, which have the above-mentioned beneficial effects, and will not be repeated here. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 A flowchart illustrating a method for detecting illegal content provided in an embodiment of this application;

[0046] Figure 2 A flowchart illustrating the generation process of the first decision model provided in this application embodiment;

[0047] Figure 3 A flowchart illustrating the generation process of the second decision model provided in this application embodiment;

[0048] Figure 4 This is a complete flowchart of the illegal content detection process provided in the embodiments of this application;

[0049] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] Current content detection solutions are mostly based on vertical category algorithms, simply identifying content violations from a spatial dimension. However, as the challenges of combating violations become more severe and the data volume grows larger, these vertical category algorithms are reaching their limits. Furthermore, these algorithms ignore common features such as user profiles, device profiles, and recent penalties associated with the content, resulting in less than ideal accuracy. To address these technical shortcomings, see [link to relevant documentation]. Figure 1 , Figure 1 A flowchart of a method for detecting illegal content provided in this application embodiment, the method including:

[0052] S101: Obtain the current published content of the target user account in the target application and the initial violation detection results of the current published content;

[0053] S102: Determine the account violation risk level of the target user account;

[0054] S103: Based on the account's violation risk level and the initial violation detection results, a second content violation detection is performed to obtain the content violation detection results for the currently published content.

[0055] First, it is necessary to obtain the initial violation detection results of the target user account's currently published content in the target application. The target user account can be the account through which the user interacts in the target application, such as the user publishing content through their account. The initial violation detection results are obtained by performing a first content violation detection on the currently published content. The method for performing this first content violation detection is not limited here; typically, a vertical algorithm can be used. A vertical algorithm refers to an algorithm model that provides accurate solutions through deep learning and training with a large amount of industry data. The specific vertical algorithm used is not limited here. In this application, the vertical algorithm refers to a professional algorithm developed for identifying various types of violations in samples of various types of content (images, text, audio, video), including but not limited to text multi-classification, clustering, audio and video pornography detection, image target detection, etc. The initial violation detection results can employ algorithms including multi-classification detection algorithms, pornography detection algorithms, abusive language detection algorithms, vulgarity detection algorithms, etc. Furthermore, it can also include rule hit information, such as regular expression matching, statistics of low-frequency alphanumeric symbols, keyword hit count, keyword hit degree, and other hard rule hit information.

[0056] The initial violation detection result obtained from performing the first content violation detection can be the already determined result or the possible determination of the currently published content, expressed as a percentage or identification level.

[0057] When performing the first content violation detection, a corresponding first content violation detection model can be invoked, such as a vertical category algorithm model. This embodiment assumes that the first content violation detection model has already been acquired or its generation process has been completed before performing the first content violation detection. In practical applications of this application, the first content violation detection model can be generated offline.

[0058] Next, the account violation risk level of the target user account is determined. The account violation risk level is determined based on the historical application data generated by the target user account within the target application. The account violation risk level can be categorized by setting different risk levels for each user account. This account violation risk level reflects the user account's historical application data performance within the target application. The risk level can be determined based on factors such as the nature of violations, the number of violations, and the frequency of violations in the historical application data, or it can be combined with user activity or other characteristics.

[0059] One feasible method for classifying account violation risk levels is to categorize platform users into three groups: white users, ordinary black users, and heavily black users. Specifically, the following classification method can be used:

[0060] If the user behavior profile data, user content profile data, and device environment profile data are all normal, confirm that the target user account's account violation risk level is a white user, so that the risk level parameters corresponding to white users can be used to participate in the second content violation detection.

[0061] If there is illegal content in the user behavior profile data, user content profile data, and device environment profile data, and the frequency of violations is less than the set threshold, the account violation risk level of the target user account is confirmed as a regular black user, so that the risk level parameters corresponding to regular black users can be used to participate in the second content violation detection.

[0062] If the user behavior profile data, user content profile data, and device environment profile data contain illegal content, and the frequency of violations is not less than the set threshold, the target user account is confirmed to be a heavily blacklisted user, so that the risk level parameter corresponding to the heavily blacklisted user can be used to participate in the second content violation detection.

[0063] As can be seen, "white users" are users who do not post any prohibited content, "ordinary black users" refer to users who post prohibited content, but with fewer violations or lower frequency, and "heavy black users" are users who post prohibited content multiple times. Different account violation risk levels can correspond to different risk level parameters, thus allowing for different content violation detection results based on the account's violation risk level when participating in the second content violation detection. In other account violation risk level classification methods in this application, user activity can also be added as a criterion, classifying users who post content multiple times, have high user activity, and no prohibited content as "excellent white users."

[0064] It should also be noted that different account violation risk levels have corresponding risk level parameters, which are used for secondary content violation detection. To facilitate secondary content violation detection and quickly output violation results, account violation risk levels can be represented by numbers or letters, for example, 0, 1, and 2 can be used to represent white users, ordinary black users, and heavily black users, respectively.

[0065] After obtaining the account's violation risk level, a second content violation detection is performed based on the account's violation risk level and the initial violation detection results to obtain the content violation detection results for the currently published content.

[0066] In one feasible implementation, a second decision model can be applied to perform a second content violation detection to obtain the violation result of the currently published content. That is, the account violation risk level and the initial violation detection result are input into the second decision model, and the output of the second decision model is used as the content violation detection result for the currently published content.

[0067] This second decision model can also be generated in advance, for example, it can be a binary classification model trained based on the account violation risk levels and violation detection results of several user accounts included in the application. The violation detection result is the result of the violation rule hit corresponding to the historical application data generated by the user account in the target application;

[0068] The training results of the second decision model are used to indicate whether the current sample content violates regulations. Typically, the second decision model can directly output a violation determination. However, if the violation result is a predicted value, the binary classification result corresponding to the predicted value can be determined first, and this binary classification result can be used as the violation result. That is, the second decision model can also output the probability that the current sample content violates regulations, and then determine whether the current sample content violates regulations based on the probability and the corresponding violation determination threshold.

[0069] When performing violation content detection, this application first obtains the current published content of the target user account in the target application and the initial violation detection results of the current published content. Then, it performs comprehensive content violation detection based on the account violation risk level of the target user account. Based on the initial violation detection results obtained from the preliminary content detection of the currently published content, the account violation risk level reflects the historical violation situation of the target user account. By comprehensively combining the account violation risk level and the initial violation detection results, the application avoids the violation content countermeasures faced when only performing content violation detection on the currently published content, thereby improving the accurate recall rate of violation content.

[0070] The generation process of the first decision model is further explained below; see [link to relevant documentation]. Figure 2 , Figure 2A flowchart illustrating the generation process of the first decision model provided in this application embodiment, the process including:

[0071] S201: Collect historical application data of several user accounts in the target application, and construct sample data containing the mapping relationship between user accounts and corresponding historical application data.

[0072] S202: Divide the sample data into a first training set and a first test set according to a first predetermined ratio; wherein the first training set includes a training set and a validation set;

[0073] S203: Define the first loss function of the first decision model, and use the training set and validation set to minimize the first loss function in multiple rounds of training to obtain the initial model; wherein, the validation set is changed in turn in different rounds of training.

[0074] S204: Validate the initial model using the validation set;

[0075] S205: If the verification is successful, the initial model will be used as the first decision model.

[0076] First, historical application data of platform users is collected. This historical application data includes data that can reflect the user's historical operations, such as the user's private messages, comments, searches and follows, the user's historical content, and the hardware status of the device used by the user, etc.

[0077] The first decision model pertains to a regression task, and therefore, the first loss function applied is not limited. One feasible implementation is to use the mean squared error as the loss function. The sample data is divided into a first training set and a first test set. The first training set may contain a training set and at least one validation set. The ratio of the training set to the validation set is not limited; for example, if the first training set comprises N parts, N-1 parts are the training set and 1 part is the validation set. Furthermore, other ratios can be used, which can be determined by those skilled in the art based on validation requirements. For example, the first training set D and the first test set T can be split at the user granularity in a 7:3 ratio. The first training set D is divided into 5 equal parts, with 4 parts used for training and the remaining part used for validation. Through multiple rounds of cross-validation—that is, performing 5 rounds of training, each using a different set of data as the validation set—better model results can be obtained. Finally, the model's accuracy and generalization are verified by comparing the results with the first test set T.

[0078] Furthermore, if the initial model fails to pass validation using a validation set, iterative training can be performed until validation is successful.

[0079] By adopting the first decision model, after obtaining the historical application data of the target user account, the data can be input into the first decision model to directly output the account violation risk level of the target user account. There is no need for manual judgment of the account violation risk level of the target user account. At the same time, the determined account violation risk level of the target user account is more accurate, which helps to improve the accuracy of content detection for currently published content.

[0080] To further improve the training efficiency of the first decision model and avoid interference from invalid or abnormal data, one feasible implementation involves optimizing the abnormal and missing data in the historical application data collected from platform users. Specifically, a default value, such as 0, can be set for the missing content in the historical application data.

[0081] For outliers in historical application data: If the historical application data is numerical outlier, delete the corresponding sample data; if the historical application data is non-numerical outlier, convert the historical application data into a vector and perform normalization.

[0082] For example, ages greater than 150 are considered invalid and can be deleted, set to unknown or default data. One-hot encoding can be used for non-numerical outlier data. Normalization can be performed on continuous variables with large numerical ranges, or log transformation or bucketing can be applied.

[0083] Normalization uses maxmin, and the corresponding normalization formula is as follows:

[0084] x1 = (x-min) / (max-min);

[0085] The formula for taking the logarithmic transformation is as follows:

[0086] x1 = log(x+1);

[0087] Where x represents the original data value, x1 represents the value after normalization or log transformation, and min and max represent the minimum and maximum values ​​of the data in this dimension, respectively. Bucketing is used to group data with the same characteristics into one bucket. For example, if the age characteristic is 0-80 years old, it can be divided into four buckets: minors, teenagers, middle-aged, and elderly.

[0088] Furthermore, based on the above embodiments, in order to improve the accuracy of the account violation risk levels obtained by the first decision model and to improve the interpretability of the first decision model, in one feasible implementation, after constructing sample data containing the mapping relationship between user accounts and their corresponding historical application data, feature filtering can be performed on the sample data. Specifically, the acquired historical application data can be processed and dimensionality reduced.

[0089] There are no restrictions on how to perform the dimensionality reduction processing. We can calculate the information entropy gain of a single feature in the account's historical application data, and then discard single features whose information entropy gain does not meet a set value. The remaining single features are used as historical application data in the sample data to train the first decision model. This single feature is the account's historical application data that can influence the results of detecting illegal content.

[0090] Specifically, when calculating the information entropy gain for each individual feature, for any target single feature in the account's historical application data, the historical application data can be used as the first sample dataset, and the historical application data after discarding the target single feature can be used as the second sample dataset. Then, the first violation probability of the first sample dataset and the second violation probability of the second sample dataset are calculated respectively. Finally, the information entropy gain of the target single feature can be calculated based on the difference between the first and second violation probabilities. If the violation probabilities of the two sample datasets are small, it indicates that the single feature has a small information entropy gain on the sample data. If the violation probability of the first sample dataset is significantly higher than that of the second sample dataset, it indicates that the single feature has a high information entropy gain on the sample data, i.e., the historical application data. In this case, features with larger information entropy gains can be selected, while features with small or no information entropy gains can be discarded. For example, user comments, user likes, and user views are all user behavior profile data, but user likes and user views are less likely to cause violations, while user comments may contain violating content. Therefore, user comments generally have a larger information entropy gain, meaning they have a greater impact on the detection of violating content.

[0091] Those skilled in the art can determine that a single feature has a high information entropy gain and retain it if the difference between the violation probability of the first sample dataset and the violation probability of the second sample dataset is greater than the violation probability difference threshold; otherwise, the single feature is discarded. The specific value of this violation probability difference threshold is not limited here and can be set by those skilled in the art based on the amount of training sample data of the first decision model.

[0092] In one optional implementation, if the historical application data includes scenario behavior data, user historical content review data, and backend device environment security data, it can be executed as follows:

[0093] The first step is to determine user behavior profile data based on scenario behavior data;

[0094] The second step is to determine user content profile data based on users' historical content review data.

[0095] The third step is to determine the device environment profile data based on the background device environment security data.

[0096] The fourth step is to determine the information entropy differences between the user behavior profile data, user content profile data, and device environment profile data and the target dataset.

[0097] This embodiment divides historical application data into three main categories. The first category is user scenario behavior data, which can include behaviors such as private messaging, commenting, searching, following, logging in, registering, playing songs, visiting live streams, earning points through activities, and other core scenarios. Based on this, user behavior profile data can be generated. Abnormal tags for black and gray market activities can be added to the user behavior profile data, such as those for inflated playback numbers, follower manipulation, boosting live stream popularity, fake followers, private message fraud, private message advertising, group bulk registration / login, and exploiting promotional offers.

[0098] The second category is user historical content review data, which mainly targets four categories of data: images, text, audio, and video submitted by users, as well as the results of machine review and human review. Based on user historical content review data, historical characteristics such as the number of content sent, the number of violations, the number of black market advertisements, and the number of false recalls can be added as tags.

[0099] The third category is background device environment security data, including device hardware parameters, device status parameters, device environment data, etc. Based on the background device environment security data, abnormal device environment tags can be added, such as whether the device is rooted, whether it is an emulator, whether it is an MSGISK attack environment, whether it is a virtual location, whether it is a virtual click, etc.

[0100] This embodiment classifies historical application data and adds corresponding labels, which helps to quickly evaluate the information entropy gain of each single feature in each type of data, so as to quickly filter out single features with high information entropy gain for training the first decision model, effectively improving the training efficiency of the first decision model.

[0101] The following section describes the generation process of the second decision model in the above embodiments. See [link to documentation]. Figure 3 , Figure 3 The flowchart of the generation process of the second decision model provided in the embodiments of this application includes:

[0102] S301: Construct structured data containing the account history behavior characteristic levels and violation detection results of user accounts; wherein, the violation detection results are the violation rule hit results corresponding to the historical application data generated by the user account in the target application;

[0103] S302: Set a second loss function for the second decision model, and divide the structured data into a second training set and a second test set according to a second set ratio; wherein the second training set includes two parts, one of which is the training set and the other is the validation set;

[0104] S303: Using the second training set and the second test set, with the goal of minimizing the second loss function, train the second decision model.

[0105] The second decision model is used to determine whether the current sample content violates regulations. There are no restrictions on the type of second loss function used; for example, cross-entropy can be used. During training, the second training set and the second test set can be split at the user granularity according to a predetermined ratio. Then, the second training set is evenly divided for cross-validation to obtain better model results. Finally, the second test set is used to verify the model's accuracy and generalization.

[0106] It should be noted that when training the second decision model, the initial violation detection results can be further refined to determine the result score corresponding to the initial violation detection results.

[0107] The first and second set ratios in the above embodiments can be the same or different ratios, and can be set by those skilled in the art.

[0108] The choice of which model to use for the first and second decision models is not limited here. In a preferred implementation method, the XGBoost algorithm can be employed. XGBoost is a type of boosting algorithm, where the idea is to integrate many weak learners together to form a strong learner. Because XGBoost is a boosting tree model, it integrates many tree models to form a very strong learner. When using the XGBoost algorithm, it supports distributed computing, facilitating the detection of illegal content on massive datasets. In other embodiments of this application, the second decision model can also employ the logistic regression algorithm.

[0109] See Figure 4 , Figure 4 This is a complete flowchart of the illegal content detection process provided in the embodiments of this application, by... Figure 4 As can be seen, the first decision model can be generated offline first. The specific model generation process will not be repeated here.

[0110] After obtaining the target user's current posted content, the first decision model outputs the account violation risk level. It's easy to understand that if the target user has no historical application data, the target user is used as the default user, meaning the corresponding account violation risk level belongs to the user level of those who have not posted any violation content. Otherwise, the account violation risk level is determined based on their historical application data.

[0111] Then, the initial violation detection results of the currently published content are input into the second decision model to obtain the violation detection results of the currently published content.

[0112] In the practical application of this application, the first decision model can be an offline model, trained based on historical application data. The second decision model can be trained based on content posted by users whose violations have been determined. For example, the violation results determined by the second decision model in the previous time period and the historical behavioral characteristic levels of the users who posted that content can be used as sample data for iterative training. For instance, the sample data for the first decision model can be set as historical application data of platform users within the previous week, while the second decision model can include account violation risk levels and initial violation detection results from all time periods before and yesterday as training data.

[0113] This application also provides a computer program product, including a computer program, which, when executed, implements the steps of the illegal content detection method as described in the above embodiments.

[0114] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0115] This application also provides an electronic device, see [link to document]. Figure 5 The present application provides a structural diagram of an electronic device, such as... Figure 5 As shown, it may include a processor 510 and a memory 520.

[0116] The processor 510 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 510 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 510 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 510 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 510 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0117] The memory 520 may include one or more computer-readable storage media, which may be non-transitory. The memory 520 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 520 is used to store at least the following computer program 521, which, after being loaded and executed by the processor 510, is capable of implementing the relevant steps in the methods executed by the electronic device side as disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 520 may also include an operating system 522 and data 523, etc., and the storage method may be temporary storage or permanent storage. The operating system 522 may include Windows, Linux, Android, etc.

[0118] In some embodiments, the electronic device may further include a display screen 530, an input / output interface 540, a communication interface 550, a sensor 560, a power supply 570, and a communication bus 580.

[0119] certainly, Figure 5 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than [other components]. Figure 5 More or fewer components as shown, or combinations of certain components.

[0120] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. As the system provided in the embodiments corresponds to the method provided in the embodiments, the description is relatively simple; relevant parts can be found in the method section.

[0121] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

[0122] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for detecting illegal content, characterized in that, include: Obtain the current published content of the target user account in the target application and the initial violation detection results of the currently published content; The initial violation detection result is obtained by performing a first content violation detection on the currently published content; Determine the account violation risk level of the target user account; The account violation risk level is determined based on the historical application data generated by the target user account in the target application. A second content violation detection is performed based on the account violation risk level and the initial violation detection result to obtain the content violation detection result for the currently published content; The determination of the account violation risk level of the target user account includes: The first decision model is applied to determine the account violation risk level of the target user account; The generation process of the first decision model includes: Collect historical application data of several user accounts in the target application, and construct sample data containing the mapping relationship between the user accounts and the corresponding historical application data. The sample data is divided into a first training set and a first test set according to a first predetermined ratio; wherein the first training set includes a training set and a validation set. A first loss function is set for the first decision model, and an initial model is obtained by minimizing the first loss function in multiple rounds of training using the training set and the validation set; wherein the validation set is changed in turn in different rounds of training. The initial model is validated using the validation set. If the verification is successful, the initial model will be used as the first decision model; After collecting historical application data of several user accounts in the target application, the process also includes: Calculate the information entropy gain of a single feature in the account's historical application data; the single feature is the account's historical application data that can affect the detection results of illegal content. Discarding single features whose information entropy gain does not meet the set value, and using the remaining single features as historical application data in the sample data to train the first decision model.

2. The method for detecting illegal content according to claim 1, characterized in that, After collecting historical application data of several user accounts in the target application, the process also includes: Set default values ​​for missing content in the account's historical application data; For outliers in the account's historical application data: if the outlier is a numerical outlier, delete the corresponding sample data; if the outlier is a non-numerical outlier, convert the account's historical application data into a vector and perform normalization.

3. The method for detecting illegal content according to claim 1, characterized in that, Calculating the information entropy gain of a single feature in the account's historical application data includes: For any single target feature in the account's historical application data, the historical application data is used as the first sample dataset, and the account's historical application data after discarding the single target feature is used as the second sample dataset. Calculate the first violation probability of the first sample dataset and the second violation probability of the second sample dataset, respectively; The information entropy gain of the target single feature is calculated based on the difference between the first violation probability and the second violation probability.

4. The method for detecting illegal content according to claim 1, characterized in that, If the historical application data includes scenario behavior data, user historical content review data, and backend device environment security data, the scenario behavior data is used to determine user behavior profile data, the user historical content review data is used to determine user content profile data, and the backend device environment security data is used to determine device environment profile data. Determining the account violation risk level of the target user account includes: If the user behavior profile data, the user content profile data, and the device environment profile data are all normal, the account violation risk level of the target user account is confirmed to be a white user, so that the risk level parameter corresponding to the white user can be used to participate in the second content violation detection. If the user behavior profile data, the user content profile data, and the device environment profile data contain illegal content, and the frequency of violations is less than a set threshold, the account violation risk level of the target user account is confirmed as a regular black user, so that the risk level parameter corresponding to the regular black user can be used to participate in the second content violation detection. If the user behavior profile data, the user content profile data, and the device environment profile data contain illegal content, and the frequency of violations is not less than the set threshold, the account violation risk level of the target user account is confirmed as a severely blacklisted user, so that the risk level parameter corresponding to the severely blacklisted user can be used to participate in the second content violation detection.

5. The method for detecting illegal content according to any one of claims 1 to 4, characterized in that, Based on the account violation risk level and the initial violation detection result, a second content violation detection is performed to obtain the content violation detection result for the currently published content, including: The account violation risk level and the initial violation detection result are input into the second decision model, and the output of the second decision model is used as the content violation detection result for the currently published content. The training process of the second decision model includes: Construct structured data containing the account history behavior feature levels and violation detection results of user accounts; wherein, the violation detection results are the violation rule hit results corresponding to the historical application data generated by the user account in the target application; Set a second loss function for the second decision model, and divide the structured data into a second training set and a second test set according to a second set ratio; Using the second training set and the second test set, the second decision model is trained with the goal of minimizing the second loss function.

6. The method for detecting illegal content according to claim 5, characterized in that, After performing a second content violation detection based on the account violation risk level and the initial violation detection result, and obtaining the content violation detection result for the currently published content, the method further includes: The account violation risk level of the target user account is updated based on the content violation detection results of the currently published content.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the illegal content detection method as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and the processor, when calling the computer program in the memory, implements the steps of the illegal content detection method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, It includes a computer program that, when executed, implements the steps of the illegal content detection method as described in any one of claims 1 to 6.

Images