Illegal content detection method, medium, equipment and program product

Abstract

The invention provides an illegal content detection method, a medium, equipment and a program product, and relates to the field of data detection, and the method comprises the steps: obtaining the current published content of a target user account in a target application and an initial illegal detection result of the current published content; the initial violation detection result is obtained by performing first content violation detection on the current published content; determining an account violation risk level of the target user account; the account violation risk level is determined based on historical application data generated by the target user account in the target application; and performing second content violation detection according to the account violation risk level and the initial violation detection result to obtain a content violation detection result for the currently published content. According to the method and the device, illegal content confrontation detection confronted by only carrying out content illegal detection on the current published content can be avoided, and the accurate recall rate of the illegal content is improved.

Description

Technical Field

[0001] The present application relates to the field of data detection, and in particular to a method, medium, device and program product for detecting illegal content. Background Art

[0002] Currently, content detection is basically based on vertical algorithm optimization, which simply identifies whether the content is illegal from the spatial dimension. As the confrontation with illegal content becomes more and more serious, the identification of vertical algorithms becomes more and more difficult, and the accuracy rate reaches a bottleneck, and its accuracy rate is low. The effect optimization improvement is not obvious. Summary of the invention

[0003] The purpose of this application is to provide a method for detecting illegal content, a computer-readable storage medium, an electronic device and a computer program product, which can perform comprehensive content detection based on the account violation risk level of the user account and the content violation detection results, improve the content detection accuracy, and solve the problem of low content detection accuracy generated by a single application vertical algorithm.

[0004] In order to solve the above technical problems, this application provides a method for detecting illegal content. The specific technical solution is as follows:

[0005] Acquire the current published content of the target user account in the target application and the initial violation detection result of the current published content; the initial violation detection result is obtained by performing a first content violation detection on the current published content;

[0006] Determining an account violation risk level of the target user account; the account violation risk level is determined based on historical application data generated by the target user account in the target application;

[0007] A second content violation detection is performed according to the account violation risk level and the initial violation detection result to obtain a content violation detection result for the currently published content.

[0008] Optionally, determining the account violation risk level of the target user account includes:

[0009] Applying a first decision model to determine an account violation risk level of the target user account;

[0010] The generation process of the first decision model includes:

[0011] Collecting account history application data of several user accounts in the target application, and constructing sample data including mapping relationships between the user accounts and corresponding account history application data;

[0012] Dividing the sample data into a first training set and a first test set according to a first set ratio; wherein the first training set includes a training set and a validation set;

[0013] Setting a first loss function of the first decision model, using the training set and the validation set to minimize the first loss function in multiple rounds of training to obtain an initial model; wherein the validation set is changed in turn in different rounds of training;

[0014] Validating the initial model using the validation set;

[0015] If the verification is successful, the initial model is used as the first decision model.

[0016] Optionally, after collecting the account history application data of several user accounts in the target application, the method further includes:

[0017] Setting default values ​​for missing content in the historical application data of the account;

[0018] For abnormal values ​​in the historical application data of the account: if the abnormal value is numerical abnormal data, delete the corresponding sample data; if the abnormal value is non-numerical abnormal data, convert the historical application data of the account into a vector and perform normalization processing.

[0019] Optionally, after collecting the account history application data of several user accounts in the target application, the method further includes:

[0020] Calculate the information entropy gain of a single feature in the historical application data of the account; the single feature is the historical application data of the account that can affect the detection result of illegal content;

[0021] Single features whose information entropy gain does not meet the set value are discarded, and the remaining single features are used as historical application data in the sample data to train the first decision model.

[0022] Optionally, calculating the information entropy gain of a single feature in the historical application data of the account includes:

[0023] For any target single feature in the historical application data of the account, the historical application data is used as a first sample data set, and the historical application data of the account after discarding the target single feature is used as a second sample data set;

[0024] respectively calculating a first violation probability of the first sample data set and a second violation probability of the second sample data set;

[0025] The information entropy gain of the target single feature is calculated according to the difference between the first violation probability and the second violation probability.

[0026] Optionally, if the historical application data includes scenario behavior data, user historical content review data, and background device environment security data, the scenario behavior data is used to determine user behavior portrait data, the user historical content review data is used to determine user content portrait data, and the background device environment security data is used to determine device environment portrait data, determining the account violation risk level of the target user account includes:

[0027] If the user behavior portrait data, the user content portrait data, and the device environment portrait data are all normal, confirm that the account violation risk level of the target user account is a white user, so as to apply the risk level parameter corresponding to the white user to participate in the second content violation detection;

[0028] If there are illegal contents in the user behavior portrait data, the user content portrait data and the device environment portrait data, and the violation frequency is less than the set threshold, confirm that the account violation risk level of the target user account is an ordinary black user, so as to apply the risk level parameters corresponding to the ordinary black user to participate in the second content violation detection;

[0029] If the user behavior portrait data, the user content portrait data and the device environment portrait data contain illegal content, and the violation frequency is not less than the set threshold, confirm that the account violation risk level of the target user account is a severe black user, so as to apply the risk level parameters corresponding to the severe black user to participate in the second content violation detection.

[0030] Optionally, performing a second content violation detection according to the account violation risk level and the initial violation detection result, and obtaining a content violation detection result for the currently published content includes:

[0031] Inputting the account violation risk level and the initial violation detection result into a second decision model, and using the output result of the second decision model as a content violation detection result for the currently published content;

[0032] The training process of the second decision model includes:

[0033] Constructing structured data including the account history behavior feature level and violation detection result of the user account; wherein the violation detection result is the violation rule hit result corresponding to the historical application data generated by the user account in the target application;

[0034] Setting a second loss function of the second decision model, and dividing the structured data into a second training set and a second test set according to a second set ratio;

[0035] The second decision model is obtained by training by applying the second training set and the second test set and taking minimizing the second loss function as a training goal.

[0036] Optionally, after performing a second content violation detection according to the account violation risk level and the initial violation detection result to obtain the content violation detection result for the currently published content, the method further includes:

[0037] The account violation risk level of the target user account is updated according to the content violation detection result of the currently published content.

[0038] The present application also provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above-mentioned method when executed by a processor.

[0039] The present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned method when calling the computer program in the memory.

[0040] The present application also provides a computer program product, including a computer program, which, when executed, implements the steps of the above-mentioned illegal content detection method.

[0041] The present application provides a method for detecting illegal content, including: obtaining current published content of a target user account in a target application and an initial illegal content detection result of the current published content; the initial illegal content detection result is obtained by performing a first content illegal content detection on the current published content; determining an account illegal content risk level of the target user account; the account illegal content risk level is determined based on historical application data generated by the target user account in the target application; performing a second content illegal content detection according to the account illegal content risk level and the initial illegal content detection result to obtain a content illegal content detection result for the current published content.

[0042] When performing illegal content detection, the present application first obtains the current published content of the target user account in the target application and the initial illegal content detection result of the current published content, and then performs a comprehensive content illegal content detection based on the account illegal content risk level of the target user account. On the basis of performing a preliminary content detection on the current published content to obtain the initial illegal content detection result, the historical illegal content of the target user account is reflected by the account illegal content risk level, and the account illegal content risk level and the initial illegal content detection result are comprehensively used to perform content illegal content detection, thereby avoiding the illegal content confrontation detection faced by performing content illegal content detection only on the current published content, and improving the accurate recall rate of illegal content.

[0043] The present application also provides a computer-readable storage medium, an electronic device, and a computer program product, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 A flowchart of a method for detecting illegal content provided by an embodiment of the present application;

[0046] Figure 2 A flow chart of the generation process of the first decision model provided in the embodiment of the present application;

[0047] Figure 3 A flow chart of the generation process of the second decision model provided in the embodiment of the present application;

[0048] Figure 4 A complete flowchart of the illegal content detection process provided in the embodiment of the present application;

[0049] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] Current content detection solutions are basically based on vertical algorithm optimization, which simply identifies whether the content is illegal from the spatial dimension. As the confrontation problem of illegal content becomes more and more serious and the amount of data becomes larger and larger, the vertical algorithm has reached a certain bottleneck. In addition, the vertical algorithm ignores the common features such as the user portrait and device portrait of the content, as well as the recent punishment situation, resulting in less than ideal accuracy. To solve the above technical defects, see Figure 1 , Figure 1 A flowchart of a method for detecting illegal content provided by an embodiment of the present application, the method comprising:

[0052] S101: Obtaining the current published content of the target user account in the target application and the initial violation detection result of the current published content;

[0053] S102: Determine the account violation risk level of the target user account;

[0054] S103: Perform a second content violation detection according to the account violation risk level and the initial violation detection result to obtain a content violation detection result for the currently published content.

[0055] First, it is necessary to obtain the initial violation detection result of the current published content of the target user account in the target application. The target user account can be an account for the user to interact in the target application, such as the user can publish content in the target application through the account held. The initial violation detection result is obtained by performing a first content violation detection on the current published content. There is no limitation on how to perform the first content violation detection here, and a vertical classification algorithm can usually be used. A vertical classification algorithm refers to an algorithm model that provides an accurate solution through deep learning and a large amount of industry data training. There is no limitation on what kind of content violation detection vertical classification algorithm is used here. The vertical classification algorithm of the embodiment of the present application refers to a professional algorithm for identifying various types of violations of samples developed for various contents (pictures, texts, audio, and videos), including but not limited to text multi-classification, clustering, audio and video pornography, image target detection, etc. The initial violation detection result can use algorithms including multi-classification detection algorithms, pornography detection algorithms, insult detection algorithms, and vulgarity detection algorithms. In addition, it can also include rule hits, such as regular expression matching, alphanumeric low-frequency symbol statistics, keyword hit counts, keyword hit degree, and other hard rule hits.

[0056] The initial violation detection result obtained by executing the first content violation detection may be a determined result, or may be a determination possibility of the current published content, expressed in a percentage or identification level.

[0057] When performing the first content violation detection, the corresponding first content violation detection model, such as a vertical classification algorithm model, may be called. In this embodiment, it is assumed that before performing the first content violation detection, the first content violation detection model has been acquired or the model generation process has been completed. In the actual application process of this application, the first content violation detection model may be generated offline.

[0058] Thereafter, the account violation risk level of the target user account is determined. The account violation risk level is determined based on the historical application data generated by the target user account in the target application. The account violation risk level of the target user account can be differentiated by setting a level to distinguish different risk levels of the user account. The account violation risk level is used to feedback the performance of the historical application data generated by the user account in the target application. The account violation risk level can be divided according to the violation phenomenon, number of violations, frequency of violations, etc. in the historical application data, and the account violation risk level can also be divided based on user activity or other characteristics.

[0059] One feasible way to divide account violation risk levels is to divide platform users into white users, ordinary black users, and severe black users. Specifically, the following level division method can be used:

[0060] If the user behavior profile data, user content profile data, and device environment profile data are all normal, confirm that the target user account's account violation risk level is a white user, so that the risk level parameters corresponding to the white user can be applied to participate in the second content violation detection;

[0061] If there are illegal contents in the user behavior profile data, user content profile data and device environment profile data, and the violation frequency is less than the set threshold, the account violation risk level of the target user account is confirmed to be a common black user, so as to apply the risk level parameters corresponding to the common black user to participate in the second content violation detection;

[0062] If there is illegal content in the user behavior portrait data, user content portrait data and device environment portrait data, and the violation frequency is not less than the set threshold, confirm that the account violation risk level of the target user account is a severe black user, so that the risk level parameters corresponding to the severe black user can be applied to participate in the second content violation detection.

[0063] It can be seen that white users are users who have not posted any illegal content, ordinary black users are users who have posted illegal content but have fewer violations or have a lower frequency of violations, and severe black users are users who have posted illegal content many times. Different account violation risk levels can correspond to different risk level parameters, so that when participating in the second content violation detection, different content violation detection results can be obtained based on the account violation risk level. In the division method of other account violation risk levels of this application, user activity can also be added as a judgment criterion, and users who have posted content many times and have high user activity and no illegal content can be regarded as excellent white users.

[0064] It should also be noted that different account violation risk levels have corresponding risk level parameters, which are used to participate in the second content violation detection. In order to facilitate the second content violation detection and quickly output the violation results, the account violation risk level can be represented by numbers or letters, for example, 0, 1, and 2 are used to represent white users, ordinary black users, and severe black users, respectively.

[0065] After obtaining the account violation risk level, a second content violation detection is performed according to the account violation risk level and the initial violation detection result to obtain a content violation detection result for the currently published content.

[0066] In a feasible implementation, the second decision model can be applied to perform second content violation detection to obtain the violation result of the currently published content. That is, the account violation risk level and the initial violation detection result are input into the second decision model, and the output result of the second decision model is used as the content violation detection result for the currently published content.

[0067] The second decision model can also be generated in advance, for example, it can be a binary classification model trained based on the account violation risk levels of several user accounts contained in the application and the violation detection results. The violation detection result is the violation rule hit result corresponding to the historical application data generated by the user account in the target application;

[0068] The training result of the second decision model is used to indicate whether the current sample content violates the rules. Usually, the second decision model can directly output the judgment result of whether it violates the rules. However, if the violation result is a predicted value, the binary classification result corresponding to the predicted value can be determined first, and the binary classification result can be used as the violation result. That is, the second decision model can also output the possibility of the current sample content violating the rules, and then determine whether the current sample content violates the rules based on the possibility and the corresponding violation judgment threshold.

[0069] When performing illegal content detection, this application first obtains the current published content of the target user account in the target application and the initial violation detection result of the current published content, and then performs comprehensive content violation detection based on the account violation risk level of the target user account. On the basis of performing preliminary content detection on the current published content to obtain the initial violation detection result, the historical violation situation of the target user account is reflected by the account violation risk level, and the account violation risk level and the initial violation detection result are comprehensively used to perform content violation detection, thereby avoiding the confrontation detection of illegal content faced by performing content violation detection only on the current published content, and improving the accurate recall rate of illegal content.

[0070] The generation process of the first decision model is further explained below. Figure 2 , Figure 2A flow chart of a generation process of a first decision model provided in an embodiment of the present application, the process comprising:

[0071] S201: Collecting account history application data of several user accounts in a target application, and constructing sample data including a mapping relationship between user accounts and corresponding account history application data;

[0072] S202: Dividing the sample data into a first training set and a first test set according to a first set ratio; wherein the first training set includes a training set and a validation set;

[0073] S203: setting a first loss function of the first decision model, using the training set and the validation set to minimize the first loss function in multiple rounds of training to obtain an initial model; wherein the validation set is changed in turn in different rounds of training;

[0074] S204: Validate the initial model using the validation set;

[0075] S205: If the verification is successful, the initial model is used as the first decision model.

[0076] First, the historical application data of platform users is collected. The historical application data includes relevant data that can feedback the historical operations of platform users, such as users' private messages, comments, searches and follows, historical content posted by users, and the hardware status of the devices used by users, etc.

[0077] The first decision model belongs to a regression task, and the first loss function used for it is not limited here. In a feasible implementation, the mean square error can be used as the loss function. The sample data is divided into a first training set and a first test set. And the first training set can include a training set and at least one validation set. The ratio of the training set and the validation set is not limited here. For example, if the first training set includes N parts, N-1 parts are training sets and 1 part is a validation set. In addition, the data ratio of the training set and the validation set can also adopt other ratios, which can be set by technicians in this field according to the verification requirements. For example, it can be divided into the first training set D and the first test set T at the user granularity in a ratio of 7:3. Divide the first training set D into 5 parts, take 4 of them for training, and the other part is used to verify the effect. Through multiple rounds of cross-validation, that is, perform 5 rounds of training, and use a different piece of data as a validation set in each round, a better model result can be obtained. Finally, the accuracy and generalization of the model are compared with the results of the first test set T.

[0078] In addition, if the initial model fails to pass the validation set, it can be trained iteratively until it passes the validation.

[0079] By adopting the first decision model, after obtaining the corresponding account historical application data of the target user account, it can be input into the first decision model to directly output the account violation risk level of the target user account. There is no need to manually determine the account violation risk level of the target user account. At the same time, the determined account violation risk level of the target user account is more accurate, which is conducive to improving the content detection accuracy of the currently published content.

[0080] In order to further improve the training efficiency of the first decision model and avoid the interference of invalid data or abnormal data on the training process, in a feasible implementation, after collecting the historical application data of the platform users, the abnormal data and missing data in the historical application data can be optimized. Specifically, a default value can be set for the missing content in the historical application data, for example, it can be set to 0.

[0081] For abnormal values ​​in historical application data: if the historical application data is numerical abnormal data, delete the corresponding sample data; if the historical application data is non-numerical abnormal data, convert the historical application data into a vector and perform normalization.

[0082] For example, if the age is over 150, it is considered invalid. The age data can be deleted and set as unknown or default data. One-hot encoding can be used for non-numeric abnormal data, and continuous variables with a large numerical range can be normalized. Log transformation or bucketing can also be used.

[0083] Normalization uses maxmin, and the corresponding normalization formula is as follows:

[0084] x1 = (x-min) / (max-min);

[0085] The formula for taking log transformation can be as follows:

[0086] x1 = log(x+1);

[0087] Among them, x represents the original value of the data, x1 represents the value after normalization or log transformation, min and max represent the minimum and maximum values ​​of the dimension data respectively. Bucket processing is used to group data with the same characteristics into one bucket. For example, if the age feature is 0 to 80 years old, it can be divided into four buckets: minors, teenagers, middle-aged, and elderly.

[0088] In addition, based on the above embodiment, in order to improve the accuracy of the account violation risk level obtained by the first decision model and to improve the interpretability of the first decision model, in a feasible implementation, after constructing sample data including the mapping relationship between the user account and the corresponding account historical application data, feature screening can be performed on the sample data. Specifically, the acquired historical application data can be processed and reduced in dimension.

[0089] There is no limitation on how to perform the dimensionality reduction processing. The information entropy gain of a single feature in the historical application data of the account can be calculated, so as to discard the single feature whose information entropy gain does not meet the set value, and use the remaining single feature as the historical application data in the sample data to train the first decision model. The single feature is the historical application data of the account that can affect the detection result of the illegal content.

[0090] Specifically, when calculating the information entropy gain for each single feature, for any target single feature in the historical application data of the account, the historical application data can be used as the first sample data set, and the historical application data of the account after discarding the target single feature can be used as the second sample data set. Thereafter, the first violation probability of the first sample data set and the second violation probability of the second sample data set are calculated respectively, and finally the information entropy gain of the target single feature can be calculated according to the difference between the first violation probability and the second violation probability. If the difference between the violation probabilities of the two sample data sets is small, it indicates that the information entropy gain of the single feature for the sample data is small. If the violation probability of the first sample data set is significantly higher than the violation probability of the second sample data set, it indicates that the information entropy gain of the single feature for the sample data, i.e., the historical application data, is high. At this time, features with large information entropy gain can be selected, and features with small or no information entropy gain can be discarded. For example, user comments, user likes, and user browsing are all user behavior portrait data, but user likes and user browsing are less likely to cause violations, while user comments may contain illegal content. Therefore, generally speaking, the information entropy gain corresponding to user comments is large, that is, it has a greater impact on the detection of illegal content.

[0091] Those skilled in the art can determine that the information entropy gain of the single feature is high by setting a violation probability difference threshold, that is, when the probability difference between the violation probability of the first sample data set and the violation probability of the second sample data set is greater than the violation probability difference threshold, and retain the single feature, otherwise discard the single feature. The specific value of the violation probability difference threshold is not limited here, and can be set accordingly by those skilled in the art according to the amount of training sample data of the first decision model.

[0092] In an optional implementation, if the historical application data includes scenario behavior data, user historical content review data, and background device environment security data, it can be executed as follows:

[0093] The first step is to determine the user behavior profile data based on the scenario behavior data;

[0094] Step 2: Determine user content profile data based on user historical content review data;

[0095] Step 3: Determine the device environment portrait data based on the background device environment security data;

[0096] Step 4: Determine the information entropy differences of user behavior portrait data, user content portrait data, and device environment portrait data for the data set target one by one.

[0097] This embodiment divides historical application data into three categories of data. The first category of data is user scenario behavior data, which can include private messages, comments, searches, attention, logins, registrations, song playback, live broadcast room visits, activity points, user visits and other core scenario flow and other behaviors, based on which user behavior profile data can be generated. In the user behavior profile data, you can add abnormal labels such as playback volume, attention diversion, live broadcast room popularity, zombie fans, private message fraud, private message advertising, group batch registration / login, activity wool grabbing, etc.

[0098] The second category is the user's historical content review data, which mainly targets the four major types of data sent by users: pictures, texts, audios, and videos, as well as the results of machine and human review. Based on the user's historical content review data, historical criminal record features such as the number of content sent, the number of violations, the number of black market advertisements, and the number of false recalls can be added as labels.

[0099] The third category is the background device environment security data, including device hardware parameters, device status parameters, device environment data, etc. Based on the background device environment security data, you can add abnormal device environment labels including whether the device is rooted, whether it is an emulator, whether it is a msgisk attack environment, whether it is virtual positioning, whether it is virtual click, etc.

[0100] This embodiment can also add corresponding labels by classifying historical application data, which helps to quickly evaluate the information entropy gain of each single feature in each type of data, so as to quickly screen single features with higher information entropy gain for training to obtain the first decision model, thereby effectively improving the training efficiency of the first decision model.

[0101] The following is an explanation of the generation process of the second decision model in the above embodiment, see Figure 3 , Figure 3 A flow chart of a generation process of a second decision model provided in an embodiment of the present application, the process comprising:

[0102] S301: Constructing structured data including the account historical behavior feature level and violation detection result of the user account; wherein the violation detection result is the violation rule hit result corresponding to the historical application data generated by the user account in the target application;

[0103] S302: setting a second loss function of a second decision model, dividing the structured data into a second training set and a second test set according to a second set ratio; wherein the second training set includes 2 parts, one of which is a training set and the other is a verification set;

[0104] S303: Apply the second training set and the second test set, take minimizing the second loss function as the training goal, and train to obtain a second decision model.

[0105] The second decision model is used to determine whether the current sample content violates the rules. There is no limitation on the second loss function used. For example, cross entropy can be used as the loss function. During training, the second training set and the second test set can be first divided into the second set ratio at the user granularity, and then the second training set can be evenly divided for cross-validation to obtain a better model result. Finally, the second test set can be used to verify the accuracy and generalization of the model.

[0106] It should be noted that when the second decision model is obtained through training, the initial violation detection result may be further refined to determine the result score corresponding to the initial violation detection result.

[0107] The first setting ratio and the second setting ratio in the above embodiments may be the same ratio or different ratios, which may be specifically set by those skilled in the art.

[0108] There is no limitation on what model is used for the first decision model and the second decision model. In a preferred implementation mode, the XGBoost algorithm can be used. XGBoost is one of the boosting algorithms. The idea of ​​the Boosting algorithm is to integrate many weak learners together to form a strong learner. Because XGBoost is a boosting tree model, it integrates many tree models together to form a very strong learner. When the XGBoost algorithm is used, it supports distributed computing, which is convenient for supporting the detection of illegal content in massive data. In other embodiments of the present application, the second decision model can also use a logistic regression algorithm.

[0109] See also Figure 4 , Figure 4 A complete flow chart of the illegal content detection process provided in the embodiment of the present application is provided by Figure 4 It can be seen that the first decision model can be generated offline first, and the specific model generation process will not be repeated here.

[0110] After obtaining the current published content of the target user, the first decision model outputs the account violation risk level. It is easy to understand that if the target user does not have historical application data, the target user is used as the default user, that is, the corresponding account violation risk level belongs to the user level that has not published any illegal content. Otherwise, the account violation risk level is determined based on its historical application data.

[0111] Thereafter, the initial violation detection result of the currently published content is input into the second decision model to obtain the violation detection result of the currently published content.

[0112] In the actual application process of this application, the first decision model can be an offline model, which is trained based on historical application data. The second decision model can be trained based on the content published by users who have been determined to have violated the rules. For example, the violation results determined by the second decision model in the previous time period and the historical behavioral feature level of the user who published the content can be used as sample data for iterative training. For example, the sample data of the first decision model can be set to the historical application data of platform users in the previous week, while the second decision model can include the account violation risk level and initial violation detection results in all time periods yesterday and before as training data.

[0113] The present application also provides a computer program product, including a computer program, which, when executed, implements the steps of the illegal content detection method of the above embodiment.

[0114] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0115] The present application also provides an electronic device, see Figure 5 , a structural diagram of an electronic device provided in an embodiment of the present application, such as Figure 5 As shown, a processor 510 and a memory 520 may be included.

[0116] Among them, the processor 510 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 510 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 510 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 510 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 510 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0117] The memory 520 may include one or more computer-readable storage media, which may be non-transitory. The memory 520 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 520 is at least used to store the following computer program 521, wherein, after the computer program is loaded and executed by the processor 510, it can implement the relevant steps in the method performed by the electronic device side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 520 may also include an operating system 522 and data 523, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 522 may include Windows, Linux, Android, etc.

[0118] In some embodiments, the electronic device may further include a display screen 530 , an input / output interface 540 , a communication interface 550 , a sensor 560 , a power source 570 , and a communication bus 580 .

[0119] certainly, Figure 5 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of the present application. In actual applications, the electronic device may include Figure 5 More or fewer components than shown, or combinations of certain components.

[0120] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system provided in the embodiment, since it corresponds to the method provided in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0121] Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

[0122] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for detecting illegal content, It is characterized in that include: Obtaining the current published content of the target user account in the target application and the initial violation detection result of the current published content; The initial violation detection result is obtained by performing a first content violation detection on the currently published content; Determining an account violation risk level of the target user account; The account violation risk level is determined based on historical application data generated by the target user account in the target application; A second content violation detection is performed according to the account violation risk level and the initial violation detection result to obtain a content violation detection result for the currently published content.

2. The method for detecting illegal content according to claim 1, It is characterized in that Determining the account violation risk level of the target user account includes: Applying a first decision model to determine an account violation risk level of the target user account; The generation process of the first decision model includes: Collecting account history application data of several user accounts in the target application, and constructing sample data including mapping relationships between the user accounts and corresponding account history application data; Dividing the sample data into a first training set and a first test set according to a first set ratio; wherein the first training set includes a training set and a validation set; Setting a first loss function of the first decision model, using the training set and the validation set to minimize the first loss function in multiple rounds of training to obtain an initial model; wherein the validation set is changed in turn in different rounds of training; Validating the initial model using the validation set; If the verification is successful, the initial model is used as the first decision model.

3. The method for detecting illegal content according to claim 2, It is characterized in that After collecting the account history application data of several user accounts in the target application, it also includes: Setting default values ​​for missing content in the historical application data of the account; For abnormal values ​​in the historical application data of the account: if the abnormal value is numerical abnormal data, delete the corresponding sample data; if the abnormal value is non-numerical abnormal data, convert the historical application data of the account into a vector and perform normalization processing.

4. The method for detecting illegal content according to claim 2, It is characterized in that After collecting the account history application data of several user accounts in the target application, it also includes: Calculate the information entropy gain of a single feature in the historical application data of the account; the single feature is the historical application data of the account that can affect the detection result of illegal content; Single features whose information entropy gain does not meet the set value are discarded, and the remaining single features are used as historical application data in the sample data to train the first decision model.

5. The method for detecting illegal content according to claim 4, It is characterized in that Calculating the information entropy gain of a single feature in the historical application data of the account includes: For any target single feature in the historical application data of the account, the historical application data is used as a first sample data set, and the historical application data of the account after discarding the target single feature is used as a second sample data set; respectively calculating a first violation probability of the first sample data set and a second violation probability of the second sample data set; The information entropy gain of the target single feature is calculated according to the difference between the first violation probability and the second violation probability.

6. The method for detecting illegal content according to claim 1, It is characterized in that If the historical application data includes scenario behavior data, user historical content review data, and background device environment security data, the scenario behavior data is used to determine user behavior portrait data, the user historical content review data is used to determine user content portrait data, and the background device environment security data is used to determine device environment portrait data, determining the account violation risk level of the target user account includes: If the user behavior portrait data, the user content portrait data, and the device environment portrait data are all normal, confirm that the account violation risk level of the target user account is a white user, so as to apply the risk level parameter corresponding to the white user to participate in the second content violation detection; If there are illegal contents in the user behavior portrait data, the user content portrait data and the device environment portrait data, and the violation frequency is less than the set threshold, confirm that the account violation risk level of the target user account is an ordinary black user, so as to apply the risk level parameters corresponding to the ordinary black user to participate in the second content violation detection; If the user behavior portrait data, the user content portrait data and the device environment portrait data contain illegal content, and the violation frequency is not less than the set threshold, confirm that the account violation risk level of the target user account is a severe black user, so as to apply the risk level parameters corresponding to the severe black user to participate in the second content violation detection.

7. The method for detecting illegal content according to any one of claims 1 to 6, It is characterized in that A second content violation detection is performed according to the account violation risk level and the initial violation detection result, and the content violation detection result for the currently published content is obtained, including: Inputting the account violation risk level and the initial violation detection result into a second decision model, and using the output result of the second decision model as a content violation detection result for the currently published content; The training process of the second decision model includes: Constructing structured data including the account history behavior feature level and violation detection result of the user account; wherein the violation detection result is the violation rule hit result corresponding to the historical application data generated by the user account in the target application; Setting a second loss function of the second decision model, and dividing the structured data into a second training set and a second test set according to a second set ratio; The second decision model is obtained by training by applying the second training set and the second test set and taking minimizing the second loss function as a training goal.

8. The method for detecting illegal content according to claim 7, It is characterized in that After performing a second content violation detection according to the account violation risk level and the initial violation detection result, and obtaining a content violation detection result for the currently published content, the method further includes: The account violation risk level of the target user account is updated according to the content violation detection result of the currently published content.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the illegal content detection method according to any one of claims 1 to 8 are implemented.

10. An electronic device, It is characterized in that The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the illegal content detection method as claimed in any one of claims 1 to 8 are implemented.

11. A computer program product, It is characterized in that It comprises a computer program, which, when executed, implements the steps of the illegal content detection method according to any one of claims 1 to 8.

Images