Methods, systems, and media for identifying abusive content items

Abstract

Methods, systems, and media for identifying abusive content items are provided. In some embodiments, the method includes: receiving, at a server, a request to provide user-generated content on a media content platform from a user device associated with a user; in response to receiving the request, determining whether the user-generated content is to be reviewed by one or more review users based on one or more bypass criteria; in response to determining that the user-generated content is to be reviewed based on the one or more bypass criteria, adding a review request to a queue of review users, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and determining whether a decision in response to the review request indicates that the user-generated content will be provided on the media content platform, wherein: in response to determining that the decision indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is transmitted to a user of the user device, the first message indicating at least one violated policy and a penalty to be applied to a user account corresponding to the user, and (ii) a corresponding action is taken to prohibit the user-generated content from being provided on the media content platform; and in response to determining that the decision indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is transmitted to the user of the user device, the second message indicating that the user-generated content is eligible to be provided on the media content platform.

Description

Technical Field

[0001] The disclosed subject matter relates to methods, systems, and media for identifying abusive content items. More specifically, the disclosed subject matter relates to determining whether user-generated content should be manually reviewed by one or more reviewers or whether manual review of user-generated content can be bypassed based on one or more criteria, and determining whether user-generated content is abusive content so that corresponding actions can be taken on the user-generated content. Background Art

[0002] Media content platforms have become the primary mechanism for users to discover and share digital media content. For example, a content creator may upload a video or other content item that the content creator has created to a media content platform, which may then make the uploaded content item available to viewers (e.g., by streaming the content item, by downloading the content item, etc.).

[0003] While many user-generated channels on a media content platform contain original, non-abusive content, there may be abusive content that violates the media content platform's policies, such as misleading, repetitive, sad, pornographic, infringing, and / or "clickbait" content, and that targets unsuspecting users. For example, particular content items may include objectionable content that a media content platform may want to remove or otherwise prevent its users from accessing, such as violent content, offensive language, adult content, drug-related content, gambling content, etc. As another example, a media content platform may want to prevent particular features of a service (e.g., features that allow content creators to collect revenue and / or any other appropriate features) from being used with content items that violate these particular policies.

[0004] However, it can be difficult to identify content items that violate policies. For example, while a service can use human reviewers to manually review uploaded content items, reviewing each and every uploaded content to identify policy violations can be time intensive. In this case, a content item can be available on a media content platform for any number of days or weeks before the manual review process indicates that the content item should be blocked or removed due to a policy violation of the media content platform.

[0005] Therefore, it is desirable to provide new methods, systems, and media for identifying abusive content items. Summary of the invention

[0006] Methods, systems, and media are provided for identifying abusive content items.

[0007] According to some embodiments of the disclosed subject matter, a method for identifying abusive content items is provided, the method comprising: receiving a request to provide user-generated content on a media content platform from a user device associated with a user; in response to receiving the request, determining whether the user-generated content is to be reviewed by one or more review users based on one or more bypass criteria; in response to determining that the user-generated content is to be reviewed based on the one or more bypass criteria, adding a review request to a queue of the review users, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and determining whether a decision in response to the review request indicates that the user-generated content is to be provided on the media content platform, wherein: in response to determining that the decision indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is transmitted to a user of the user device, the first message indicating at least one violated policy and a penalty to be applied to the user account corresponding to the user, and (ii) a corresponding action is taken to prohibit the user-generated content from being provided on the media content platform; and in response to determining that the decision indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is transmitted to the user of the user device, the second message indicating that the user-generated content is eligible to be provided on the media content platform.

[0008] In some embodiments, the user is a content creator, and wherein the request to provide the user-generated content comprises an upload request to store the user-generated content on a storage device associated with the media content platform.

[0009] In some embodiments, the user is a content creator, and wherein the request to provide the user-generated content comprises an initial request to start a live stream using the media content platform.

[0010] In some embodiments, the user is a viewer, and wherein the request to provide the user-generated content includes a content feature for interaction with a content creator.

[0011] In some embodiments, the method also includes determining whether the user is eligible to provide user-generated content on the media content platform by: calculating a risk score indicating a risk associated with the user account of the user based on other user-generated content that has previously been provided to the media content platform; and comparing the risk score to a threshold risk score corresponding to the content type of the user-generated content, wherein based on the comparison of the risk score to the threshold risk score, the user is deemed eligible to provide the user-generated content.

[0012] In some embodiments, the method further includes determining whether the user is eligible to provide user-generated content on the media content platform based on audience information associated with the user account of the user.

[0013] In some embodiments, the method further includes determining whether the user is eligible to provide user-generated content on the media content platform based on location information associated with the user account of the user.

[0014] In some embodiments, the method further includes determining whether the user is eligible to provide user-generated content on the media content platform based on enforcement actions of previous policy violations associated with the user account.

[0015] In some embodiments, the method also includes determining whether the user is eligible to provide user-generated content on the media content platform by transmitting a second review request to a queue of review users, wherein the second review request includes information associated with a user account corresponding to the user and prompts the review user to determine whether the information associated with the user account violates at least one policy associated with the media content platform.

[0016] In some embodiments, the method further comprises, in response to receiving a request to provide the user-generated content on the media content platform, causing a timing indication to be presented indicating when a decision responsive to the review request will be provided.

[0017] In some embodiments, the method further includes: receiving a flag from a viewer of the user-generated content on the media content platform, wherein the flag indicates a potential policy violation; and in response to receiving a subsequent review request, adding the subsequent review request to a queue of review users who previously reviewed the user-generated content, wherein the subsequent review request includes the user-generated content and the flag from the viewer, and wherein the user-generated content is restricted from being provided on the media content platform until a decision in response to the subsequent review request is received.

[0018] In some embodiments, determining whether the user-generated content is to be reviewed by one or more reviewing users based on one or more bypass criteria further includes: identifying second user-generated content that is similar to the user-generated content, wherein the second user-generated content has been previously reviewed; and determining that the user-generated content will not be reviewed by the one or more reviewing users based on the similarity between the user-generated content and the second user-generated content. In some embodiments, the method further includes: determining that the second user-generated content was previously restricted from being made available on the media content platform; and in response to determining that the second user-generated content was previously restricted from being made available on the media content platform, performing a corresponding action to prohibit the user-generated content from being made available on the media content platform, wherein the first message indicates the restriction based on the second user-generated content.

[0019] In some embodiments, determining whether the user-generated content is to be reviewed by one or more reviewing users further comprises determining whether at least one of the user-generated content and content related to the user-generated content has been previously flagged by a threshold number of viewers.

[0020] In some embodiments, the method further includes updating a user account associated with the user to indicate at least one policy associated with the media content platform that is violated by the content generated by the user.

[0021] In some embodiments, the penalty to be applied to the user account corresponding to the user is based on the number of previous policy violations by the content creator.

[0022] In some embodiments, the penalty to be applied to the user account corresponding to the user is based on the severity of at least one policy associated with the media content platform that is violated by the user-generated content.

[0023] In some embodiments, the method also includes: receiving an appeal request for a decision from a user device, wherein the appeal request includes appeal text; adding the appeal request including the appeal text and the user-generated content to a queue of a second review user, the second review user being different from the review user who provided the decision indicating that the user-generated content will be restricted from being provided on the media content platform; receiving an appeal decision corresponding to the appeal request from a user device associated with the second review user; and determining at least one corresponding action to be performed on the user-generated content based on the decision and the appeal decision.

[0024] In some embodiments, in response to determining that the decision indicates that the user-generated content item does not violate at least one policy associated with the media content platform, the second message indicates a time when the user-generated content is automatically made available on the media content platform.

[0025] According to some embodiments of the disclosed subject matter, a system for identifying abusive content items is provided, the system comprising a memory and a hardware processor, the hardware processor being configured, when executing computer-executable instructions stored in the memory, to: receive a request from a user device associated with a user to provide user-generated content on a media content platform; in response to receiving the request, determine whether the user-generated content is to be reviewed by one or more review users based on one or more bypass criteria; in response to determining that the user-generated content is to be reviewed based on the one or more bypass criteria, add a review request to a queue of the review users, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and determine in response to the request, Review whether the decision of the request indicates that the user-generated content is to be provided on the media content platform, wherein: in response to determining that the decision indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is transmitted to a user of the user device, the first message indicating at least one violated policy and a penalty to be applied to a user account corresponding to the user, and (ii) a corresponding action is taken to prohibit the user-generated content from being provided on the media content platform; and in response to determining that the decision indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is transmitted to the user of the user device, the second message indicating that the user-generated content is eligible to be provided on the media content platform.

[0026] According to some embodiments of the disclosed subject matter, a non-transitory computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for identifying abusive content items, the method comprising: receiving a request from a user device associated with a user to provide user-generated content on a media content platform; in response to receiving the request, determining whether the user-generated content is to be reviewed by one or more review users based on one or more bypass criteria; in response to determining that the user-generated content is to be reviewed based on the one or more bypass criteria, adding the review request to a queue of the review users, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and determining in response to the request that the user-generated content be reviewed. Review whether the decision of the request indicates that the user-generated content will be provided on the media content platform, wherein: in response to determining that the decision indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is transmitted to a user of the user device, the first message indicating at least one violated policy and a penalty to be applied to a user account corresponding to the user, and (ii) a corresponding action is taken to prohibit the user-generated content from being provided on the media content platform; and in response to determining that the decision indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is transmitted to the user of the user device, the second message indicating that the user-generated content is eligible to be provided on the media content platform.

[0027] According to some embodiments of the disclosed subject matter, a system for identifying abusive content items is provided, the system comprising: a device for receiving a request for providing user-generated content on a media content platform from a user device associated with a user; a device for determining, in response to receiving the request, whether the user-generated content is to be reviewed by one or more reviewing users based on one or more bypass criteria; a device for adding a review request to a queue of reviewing users in response to determining that the user-generated content is to be reviewed based on the one or more bypass criteria, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and a device for determining whether a decision in response to the review request indicates that the user-generated content is to be provided on the media content platform, wherein: in response to determining that the decision indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is transmitted to a user of the user device, the first message indicating at least one violated policy and a penalty to be applied to the user account corresponding to the user, and (ii) a corresponding action is taken to prohibit the user-generated content from being provided on the media content platform; and in response to determining that the decision indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is transmitted to the user of the user device, the second message indicating that the user-generated content is eligible to be provided on the media content platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Various objects, features, and advantages of the disclosed subject matter may be more fully understood by reference to the following detailed description of the disclosed subject matter when considered in conjunction with the following drawings, in which like reference numerals identify like elements.

[0029] Figure 1 An illustrative example of an information flow diagram for identifying abusive content items is shown in accordance with some embodiments of the disclosed subject matter.

[0030] Figure 2 An illustrative example of a process for identifying abusive content items in accordance with some embodiments of the disclosed subject matter is shown.

[0031] Figure 3 A schematic diagram showing an illustrative system suitable for implementing the mechanisms described herein for identifying abusive content items in accordance with some embodiments of the disclosed subject matter.

[0032] Figure 4 Showing that some embodiments of the disclosed subject matter can be used for Figure 3 Detailed examples of hardware in servers and / or user devices. DETAILED DESCRIPTION

[0033] According to various embodiments, mechanisms (which may include methods, systems, and media) are provided for identifying abusive content items. More specifically, the disclosed subject matter relates to determining whether user-generated content should be manually reviewed by one or more reviewers or whether manual review of user-generated content can be bypassed based on one or more criteria, and determining whether user-generated content is abusive content so that corresponding actions can be taken on the user-generated content.

[0034] Existing abusive content detection systems typically allow user-generated content to be published or otherwise made available before it is reviewed, which introduces a number of risks. The mechanisms disclosed herein can provide a multi-stage abuse mitigation approach that, for example, prohibits user-generated content from being made available on a media content platform without requiring a human to review each piece of content, while ensuring manual review of user-generated content that does not meet one or more bypass criteria.

[0035] In some embodiments, the mechanisms described herein can receive content items as user-generated content items from the creator of the content items. The user-generated content items can include, for example, any appropriate type of user-generated content, such as video content items or audio content items generated by the content creator, comments entered by viewers of the media content in conjunction with an online chat or comment section, the use of content features to generate user-generated content, and / or any other appropriate type of user-generated content. It should be noted that user-generated content can include any appropriate content, such as video content, audio content, text content, live stream video content, etc.

[0036] In some embodiments, the mechanism can determine whether the content item will be manually examined by one or more examiners. In some embodiments, in response to determining that the user-generated content will be manually examined, the mechanism can add the user-generated content to the content item queue to be examined by an examiner (e.g., an examiner at an appropriate computing device). In some embodiments, the examiner can then receive the user-generated content from the content item queue, and can examine the user-generated content to determine whether the user-generated content violates the policy of the media content platform. For example, in some embodiments, the examiner can determine whether the user-generated content includes the objectionable content (e.g., violent content, objectionable language, disgusting voice, adult content and / or the objectionable content of any other appropriate type) of any particular type. In some embodiments, the examiner can then transmit the decision-making indicating whether the user-generated content violates any policy of the media content platform. In some embodiments, the mechanism can then determine whether the content item will be posted to the media content platform in conjunction with features.

[0037] It should be noted that in some embodiments, the mechanism can, in response to receiving a decision from a first reviewer indicating that the user-generated content violates the policies of the media content platform, add the user-generated content to another queue of content items for review by a different reviewer (e.g., a verifying reviewer, a reviewer with more experience than the initial reviewer, etc.). Continuing with this example, in response to receiving a decision from a first reviewer and a different second reviewer indicating that the user-generated content violates the policies of the media content platform, a corresponding action can be taken, such as removing the user-generated content from the media content platform or otherwise prohibiting the user-generated content from being made available on the media content platform.

[0038] In some embodiments, the mechanism can determine whether to manually review content items based on any appropriate information. For example, in some embodiments, the mechanism can determine whether to manually review content items based on information associated with the content creator, such as whether the content creator has previously uploaded content items that have been approved, whether the content creator has previously uploaded content items that have violated the policy of the media content platform, and / or any other appropriate information associated with the content creator. As another example, in some embodiments, the mechanism can determine whether to manually review content items based on the similarity of content items and previously reviewed content items. As a more specific example, in some embodiments, the mechanism can determine that content items are similar to previously rejected content items, and can determine that the uploaded content items will also be rejected. In some embodiments, the mechanism can determine the criteria for determining whether content items are to be added to the item queue to be manually reviewed based on the number of received content items, thereby allowing the number of assigned manual reviews to be adjusted.

[0039] In some embodiments, in the case where the review of the content item indicates that the content item violates the policy of the media content platform, the mechanism can cause a message indicating any appropriate information to be transmitted to the content creator. For example, in some embodiments, the message can indicate the violated policy, the timestamp information of the part of the violation policy indicating the content item, the penalty for the policy violation and / or any other appropriate information. Note that in some embodiments, the mechanism can determine the penalty based on any appropriate information, such as the number of previous violations of the content creator, the severity of the violation and / or any other appropriate information. In addition, in some embodiments, the mechanism can prevent the content item from entering the media content platform and / or removing it from the media content platform.

[0040] In some embodiments, in response to determining that user-generated content or use of features for generating user-generated content is permissible (e.g., in response to a reviewer determining that the content does not violate the policies of the media content platform, in response to determining that the content may not violate the policies of the media content platform, etc.), the mechanism can communicate a specific time at which the user-generated content is posted or otherwise made available on the media content platform (e.g., rather than posting or posting the user-generated content prior to any review for violation of one or more policies of the media content platform, rather than a delay after posting or posting the content in which the content was submitted for review, etc.). Alternatively, the mechanism can communicate a specific time at which the user-generated content may be posted or published under the direction of the content creator.

[0041] In some embodiments, the mechanism can receive the appeal of decision from the content creator. In some such embodiments, the mechanism can add the appeal to the appeal queue, and the appeal can be assigned to the examiner. In some embodiments, any appropriate information can be provided to the examiner of the appeal, such as content items, information associated with the original decision appealed (e.g., information indicating a violation of policy, timing information indicating the part of the content item violating policy and / or any other appropriate information), text from the content creator (e.g., argument from the content creator and / or any other appropriate text) and / or any other appropriate information. In some embodiments, the decision relevant to the appeal can then enable the original decision to be supported, or on the contrary, the original decision can be overturned. In the case where the original decision is overturned and the content item has been previously blocked, the mechanism can make the content item available on the media content platform.

[0042] Note that in some embodiments, content items can be received from content creators, where the content items are to be provided in conjunction with the content creator's use of the feature. For example, in some embodiments, a content item can be a media content item received by a media content platform for provision to viewers in conjunction with a particular feature. As a more specific example, in some embodiments, the feature can include providing a membership service to viewers of a content channel associated with the media content platform, where the membership can allow members to access any appropriate privileges associated with the membership (e.g., access to special content, access to chat rooms or forums with other members, access to customized emoticons or badges, and / or any other appropriate privileges). As another more specific example, in some embodiments, the feature can include allowing content creators to provide merchandise (e.g., branded clothing and / or any other appropriate branded merchandise) associated with a content item and / or associated with a content channel to which the content item belongs. In conjunction with Figure 2 Detailed description of additional features. In some such embodiments, the mechanism can determine whether to manually review a content item to determine whether the content item is eligible to be provided to a user in conjunction with use of the feature. For example, in some embodiments, the mechanism can determine whether a content creator is eligible to use the feature, determine whether a content item is to be manually reviewed, determine whether a content item is to be blocked from being provided in conjunction with the feature, etc., as described below in conjunction with Figure 2 Described in more detail.

[0043] In a more specific example, before making user-generated content available on a media content platform, the mechanism can use human reviewers and machine reviewers to determine whether a user account is eligible to access media content platform features to generate user-generated content. This can, for example, limit access to media content platform features to user accounts that have been evaluated as clean or non-abusive user accounts.

[0044] Note that in some embodiments, content items as referred to herein can refer to any appropriate type of user-generated content, such as media content generated and uploaded by a content creator, an offer of membership privileges provided by a content creator in exchange for purchasing a membership associated with a channel of content produced by the content creator, comments entered by viewers of the content in a comment section or a social network post, and / or any other appropriate type of content. That is, in some embodiments, the mechanisms described herein can be used to determine whether to manually review any appropriate type of user-generated content before posting user-generated content in, for example, a specific page, forum, comment section, etc. For example, the mechanisms described herein can be used to review comments (e.g., comments submitted on a comment section associated with a content item, comments submitted in a chat forum, comments submitted on a social network post, and / or any other appropriate comments). As a more specific example, in some embodiments, the mechanisms described herein can be used to determine whether a particular comment is to be added to a comment queue to be manually reviewed, or to determine whether a comment is to be deleted. As another example, in some embodiments, the mechanisms described herein can be used to review privileges associated with membership of a channel of content provided by a content creator. As a more specific example, in some embodiments, the mechanisms described herein can be used to determine whether particular privileges provided by a content creator are to be added to a queue of privileges to be manually reviewed, and / or to determine whether particular privileges are to be revoked (e.g., due to violation of any particular policy associated with the media content platform).

[0045] Steering Figure 1 , an illustrative example 100 of an information flow diagram for identifying abusive content items is shown according to some embodiments of the disclosed subject matter. As shown, in some embodiments, the blocks of process 100 can be executed on a content creator device 308, a server 302, and / or a reviewer device 310, as described below in conjunction with Figure 3 As shown and described.

[0046] At 102, the content creator device 308 can transmit the user-generated content items to the server 302. In some embodiments, the user-generated content items can include any appropriate type of content, such as a video, a playlist of videos, a link to a live stream of a video, an image, an animation, a photo, a slideshow of photos, a document, and / or any other appropriate type of content. In a more specific example, the content creator device 308 can upload the user-generated content items for posting using a channel associated with a user account of a video sharing service. As another example, in some embodiments, the user-generated content items can include comments submitted by the user of the content creator device 308 to be posted in a chat room, a live chat, a comment section associated with a specific page, a media content item, or a social network post, and / or any other appropriate type of comments. In some embodiments, the content creator device 308 can transmit the user-generated content items in conjunction with a user account associated with the content creator device 308 (e.g., a user account on a media content platform associated with the server 302, and / or any other appropriate user account). Note that in some embodiments, server 302 can be associated with a media content platform that hosts uploaded content and streams or otherwise provides the content hosted by the media content platform to one or more user devices.

[0047] It should be noted that in some embodiments, in response to receiving a user-generated content item, the server 302 can determine whether the user-generated content item is to be reviewed by one or more review users. The server 302 can then cause a notification or other appropriate interface to be presented on the content creator device 308 indicating that the user-generated content item or access to features for generating user-generated content is being reviewed. For example, the notification can indicate a specific amount of time before a decision from a reviewer is received. In another example, the notification can indicate that an updated notification will be presented when the user-generated content is approved, thereby providing the content creator at the content creator device 308 with control over publishing or otherwise making the user-generated content available on the media content platform.

[0048] At 104, the server 302 can add the content item to a queue of content items to be reviewed. In some embodiments, the queue of content items can correspond to a queue of content items to be manually reviewed by a human reviewer and / or reviewed in any other appropriate manner. In some embodiments, the server 302 can maintain the queue in any appropriate manner. For example, in some embodiments, the content item can be added to the queue based on the time at which the content item is received by the server 302 (e.g., so that items received later are placed at a later point in the queue, and / or placed in any other manner). As another example, in some embodiments, the content item can be added to the queue based on information associated with the content creator associated with the content creator device 308. As a more specific example, in some embodiments, the server 302 can determine the popularity of the content creator (e.g., based on the number of subscribers to the content channel associated with the content creator, based on the number of views of the content generated by the content creator, and / or based on any other appropriate information), and can place the content item in the queue based on the popularity of the content creator (e.g., content items received from relatively more popular content creators can be inserted at a relatively higher position in the queue, and / or inserted in any other appropriate manner). In some embodiments, server 302 can maintain a queue of content items based on any suitable combination of factors.

[0049] Note that, if combined with Figure 2 As described in more detail at block 208 of FIG. 1 , in some embodiments, the server 302 can determine whether the content item is to be added to a queue. In some embodiments, the server 302 can add the content item to a queue of content items to be reviewed in response to determining that the content item is to be reviewed. Alternatively, in some embodiments, the server 302 can determine that the user-generated content item satisfies one or more bypass criteria such that the user-generated content item will not be reviewed and will not be added to the queue of content items to be reviewed at 104. In some such cases, the server 302 can determine whether the user-generated content item is to be posted or removed (or otherwise prohibited from being available on the media content platform) using any suitable technique or combination of techniques, such as described below in conjunction with Figure 2208 and 210 of . For example, in some embodiments, the server 302 can determine that the user-generated content is substantially similar to previously approved user-generated content (e.g., a threshold amount of similarity), and in response, can make the user-generated content available on the media content platform. In another example, in some embodiments, the server 302 can determine that the user-generated content is substantially similar to previously removed user-generated content (e.g., a threshold amount of similarity), and in response, can automatically mark, remove, or otherwise prohibit the user-generated content from being provided on the media content platform without transmitting the user-generated content to one or more reviewers.

[0050] It should be noted that in some embodiments, server 302 can receive an indication, such as a flag, from any appropriate user of the media content platform that the user-generated content may violate the policy of the media content platform (e.g., for containing objectionable content within the user-generated content). In response to receiving such an indication, server 302 can add the user-generated content to a queue of content items to be reviewed by one or more reviewers (e.g., reviewers at appropriate computing devices). This can, for example, ensure that the user-generated content is reviewed by one or more reviewers.

[0051] At 106, the reviewer device 310 can receive an indication of the content item from the queue. In some embodiments, the reviewer device 310 can receive the indication in any suitable manner. For example, in some embodiments, the reviewer device 310 can receive the indication via a user interface of the reviewer device 310 through which a user views the content item, and transmit a decision related to the content item (e.g., a decision indicating that the content item violates a specific policy associated with the media content platform, a decision indicating that the content item does not violate any policy associated with the media content platform, and / or any other suitable decision). As a more specific example, in some embodiments, the reviewer device 310 can receive a notification indicating that the content item has reached the top of the queue of items to be reviewed and / or that the content item is being assigned to the user of the reviewer device 310 for review. In some such embodiments, the notification can include any appropriate information, such as a link to the content item, the name of the content creator associated with the content creator device 308, a subject matter associated with the content item, information associated with the content creator (e.g., the number of previously uploaded and / or approved content items created by the content creator, the number of times content previously uploaded by the content creator has been associated with policy violations, and / or any other appropriate information), and / or any other appropriate information.

[0052] Note that in some embodiments, a user of reviewer device 310 can select a content item based on the indication in any appropriate manner (e.g., by selecting a link presented in the indication, and / or in any other appropriate manner), which can cause the content item to be presented in a video player window on reviewer device 310 or in any other appropriate user interface.

[0053] At 108, the reviewer device 310 can transmit a decision related to the content item. For example, in some embodiments, the decision can indicate any appropriate information related to the content item, such as whether the content item includes content that violates one or more policies associated with the media content platform to which the content item is uploaded. As a more specific example, in some embodiments, the policies can include any appropriate policies, such as whether the content item includes violent content, sexual content, abusive language, offensive speech, and / or any other appropriate type of objectionable content.

[0054] In some embodiments, the transmitted decision can include any appropriate information, such as an indication of the type of objectionable content included in the content item, an indication of one or more portions of the content item that are objectionable (e.g., based on timestamps of the video, frames of the video, and / or in any other appropriate manner), and / or any other appropriate information. Note that in some embodiments, the decision can indicate that the content item does not violate any policy.

[0055] At 110, the server 302 can update an account associated with the content creator device based on the decision received from the reviewer device 310. For example, where the decision indicates that the content item violates a particular policy, the server 302 can update a count indicating the number of times the content creator has uploaded content that violates a policy associated with the media content platform. As another example, where the decision indicates that the content item does not violate a particular policy, the server 302 can update a count indicating the number of times the content creator has uploaded content that does not violate any policy associated with the media content platform (i.e., the number of times the content creator has uploaded content that has been approved by the reviewer).

[0056] In some embodiments, the decision can then be viewed by the user of the content creator device 308. For example, in some embodiments, the user of the content creator device 308 can log into a user account associated with the content creator device 308 and the media content platform and can view the decision in a user interface associated with the user account. In another example, in response to a decision indicating that the user-generated content or the use of a feature for generating the user-generated content is permissible (e.g., in response to a reviewer determining that the content does not violate the policy of the media content platform, in response to a determination that the content may not violate the policy of the media content platform, etc.), the decision can be presented on the content creator device 308 with a specific time at which the user-generated content is posted or otherwise available on the media content platform (e.g., rather than posting or posting the user-generated content before any review of violating one or more policies of the media content platform, rather than a delay after posting or posting the content in which the content is submitted for review, etc.). Alternatively, in some embodiments, the decision can be presented with a specific time at which the user-generated content can be posted or published under the guidance of the content creator.

[0057] It should be noted that in some embodiments, in response to a reviewer providing a decision in which the user-generated content is considered to violate the policy of the media content platform, the server 302 can add the content to a queue of content items for review by another reviewer. It should also be noted that in some embodiments, the additional reviewer can be a different reviewer with more experience than the reviewer who provided the initial decision. In such embodiments, in response to receiving two negative decisions from two different reviewers indicating that the user-generated content violates the policy of the media content platform, the server 302 can remove or otherwise prohibit the user-generated content from being provided by the media content platform.

[0058] In some embodiments, the decision can be appealed by the user of the content creator device 308. For example, in some embodiments, the decision can be appealed if the user of the content creator device 308 disagrees with the decision (e.g., if the decision indicates that the content item includes particular types of objectionable content and / or violates particular policies, and if the user disagrees, and / or in any other appropriate circumstances). Figure 1 The blocks 112-118 shown in and described below can be performed in the case where the decision is appealed. In some embodiments, blocks 112-118 can be omitted in the instance where the decision is not appealed.

[0059] At 112, the content creator device 308 can transmit an indication that the decision is being appealed. In some embodiments, the content creator device 308 can transmit the indication in any suitable manner. For example, in some embodiments, a user of the content creator device 308 can select any suitable button or user interface control associated with the decision, which, when selected, causes a user interface to be used to enter any suitable text associated with an appeal of the decision (e.g., any suitable reasons for contesting the decision and / or any other suitable text). In some such embodiments, the text associated with the appeal of the decision can be transmitted to the server 302 in conjunction with the indication that the decision is being appealed.

[0060] At 114, the server 302 can add the content item to the appeal queue. In some embodiments, the content item can be added to the appeal queue in any suitable manner. For example, in some embodiments, the content item can be added to the appeal queue based on the time at which the content creator device 308 transmits an indication that the decision is being appealed at 112. As another example, in some embodiments, the content item can be added to the appeal queue based on the relative popularity of the content creator associated with the content item, as described above in conjunction with block 104.

[0061] At 116, reviewer device 310 can receive an indication of a content item from the appeal queue. In some embodiments, reviewer device 110 can receive an indication of a content item in any suitable manner. For example, in some embodiments, reviewer device 110 can present a notification indicating that a decision associated with the content item is being appealed by a content creator associated with the content item.

[0062] In some embodiments, the user of reviewer device 310 can select the indication in any suitable manner, and in some embodiments, selection of the indication can cause the content item to be presented on reviewer device 310 in conjunction with any suitable information associated with the decision and / or any suitable information associated with the appeal. For example, in some embodiments, the information can include text associated with the appeal (e.g., as described above in conjunction with block 110), information associated with the appealed decision (e.g., information indicating the objectionable content or policy violation indicated in the decision, information indicating a timestamp when the particular objectionable content was indicated as being presented in the decision, and / or any other suitable information).

[0063] Note that although Figure 1The same device (i.e., reviewer device 310) is shown as both reviewing a content item at 106 and reviewing an appeal of the content item at 116, but in some embodiments, two different reviewer devices can be used. That is, in some embodiments, a reviewer different from the reviewer who delivered the decision being appealed can review the appeal of the decision. For example, different reviewers at different reviewer devices can be used to review user-generated content in an appeal queue, where the different reviewers are designated as having more experience than an initial reviewer who provided a negative decision, and where actions can be taken in response to both the initial reviewer and the different appeal reviewer who provided the negative decision, such as suspending a user account or disabling presentation of the user-generated content.

[0064] At 118, reviewer device 310 can transmit a decision related to the appeal of the content item. In some embodiments, the decision can indicate any suitable information, such as that the user of reviewer device 310 agrees with the appealed decision, that the user of reviewer device 310 disagrees with the appealed decision, and / or any other suitable information.

[0065] At 120, the server 302 can update an account associated with the content creator based on the decision associated with the appeal. For example, in the event that the appealed decision indicates that the content item violates a particular policy and in which the decision associated with the appeal is inconsistent with the appealed decision, the server 302 can update the account to reflect the inconsistency between the two decisions. As a more specific example, in some embodiments, the server 302 can decrement a count of the number of times the content creator has uploaded content items that violate policies associated with the media content platform. As another example, in the event that the decision associated with the appeal confirms that the decision is appealed as a content item violating a particular policy, the server 302 can perform any appropriate action, such as blocking the content item, prohibiting the content creator device 308 from uploading the content item for a predetermined duration (e.g., for one month, ninety days, and / or any other appropriate duration), and / or any other appropriate action. Additional details regarding the actions that can be performed by the server 302 will be described below in conjunction with the description of the actions that can be performed by the server 302. Figure 2 224 is described below.

[0066] Steering Figure 2 , an illustrative example 200 of a process for identifying abusive content is shown according to some embodiments of the disclosed subject matter. In some embodiments, blocks of process 200 can be performed on a server, such as a server associated with a media content platform that hosts content items uploaded by one or more content creators.

[0067] Process 200 can start at 202 by receiving content items to be uploaded or posted in conjunction with a media content platform from a content creator. For example, in some embodiments, a content item can be a video content item or an audio content item created by a content creator, which content items are to be uploaded to a media content platform and are made available for viewing by other users of the media content platform. As another example, in some embodiments, a content item can be a comment submitted by a viewer of a content item hosted by a media content platform. As a more specific example, in some embodiments, a comment can be a comment submitted by a viewer in conjunction with a chat room or live chat associated with a specific content channel, a comment submitted by a viewer to post in a specific comment section (e.g., a comment section associated with a specific media content item hosted by a media content platform and / or any other suitable comment section), and / or any other suitable comment. As yet another example, in some embodiments, a content item can be a privilege provided by a content creator via a media content platform in conjunction with a content channel created by a content creator. As a more specific example, a privilege can be the provision of a specific type of content (e.g., exclusive video content, behind-the-scenes content, clothing or other merchandise, and / or any other suitable type of content) to be provided by a content creator in exchange for a user purchasing a membership or subscription to a content channel.

[0068] In some embodiments, the uploaded content item can be uploaded in conjunction with specific features of the media content platform, such as features that the content creator is interested in for generating user-generated content. In some embodiments, the features can include features associated with comments or messages that can be transmitted by viewers of the content item. For example, in some embodiments, the features can be related to comments in the comment section associated with the comment item and / or comments posted in the live chat associated with the content item. As a more specific example, the features can include allowing viewers of the content item to purchase a "super chat" that can allow viewers to pin comments associated with the purchased "super chat" to a specific location in the chat or comment section (e.g., at the top of the live chat, in the top position of the comment section, and / or in any other suitable location). Continuing with this example, in some embodiments, the features can include determining the duration for which the "super chat" comments are to be pinned to a specific location based on the payment amount paid by the viewer who purchased the "super chat". As another more specific example, in some embodiments, a feature can include allowing viewers of a content item to purchase a "super sticker" that allows the viewer to include an animated image or icon in a comment section or live chat associated with the content item. Continuing with this example, in some embodiments, a feature can include pinning a specific "super sticker" image or icon to a specific location in a live chat or comment section in response to determining that any specific criteria have been met (e.g., the viewer has paid more than a predetermined amount for the "super sticker" and / or any other suitable criteria). Note that in some embodiments, the animated image or icon available for selection by viewers of the content item can be curated in any suitable manner, for example, generated and made available by a content creator, by a media content platform, and / or by any other suitable entity. As yet another more specific example, in some embodiments, a feature can include allowing viewers of a content item to purchase a "super thank you." In some such embodiments, a viewer's purchase of a "Super Thanks" can cause a selected animated image or icon (e.g., selected by a viewer of the content item in conjunction with the purchase of the "Super Thanks") to be presented in conjunction with the content item and / or can cause any suitable proportion of the viewer's payment to be shared with the content creator.

[0069] Note that in some embodiments, the feature can allow the content creator to receive any suitable percentage of revenue from purchased Super Chats, Super Stickers, and / or Super Thanks. For example, in some embodiments, the content creator can receive a percentage of revenue from the purchased options, while the media content platform can receive the remainder of that revenue. Additionally or alternatively, in some embodiments, the feature can allow the content creator to designate any suitable charitable organization to receive a percentage of revenue.

[0070] In some embodiments, features can allow content creators to provide membership services to viewers of content created by the content creator. For example, in some embodiments, the membership service can allow viewers who purchase a membership to receive any suitable privileges related to content created by the content creator. In some embodiments, the privileges can include any suitable special offers available to viewers who purchase a membership, such as access to special or exclusive content created by the content creator (e.g., available only to viewers who purchase a membership), access to exclusive emoticons or other animated icons that can be included in chat messages or comment sections associated with content created by the content creator, access to an exclusive chat forum where viewers who have purchased a membership can chat with other viewers who have also purchased a membership, and / or any other suitable special offers available to members.

[0071] In some embodiments, a feature can allow a content creator to offer merchandise through a media content platform. For example, in some embodiments, a feature can allow a content creator to include a link to a merchandise (e.g., clothing, mugs or cups, artwork, and / or any other suitable merchandise) in a page presenting a content item or other content provided by the content creator through a media content platform. As a more specific example, a feature can allow a content creator to include a link to one or more goods available for purchase in a video watch page presenting a content item. As another more specific example, a feature can allow a content creator to include a link to one or more goods available for purchase in a page associated with a channel associated with the content creator's content.

[0072] In some embodiments, a feature can allow a content creator to provide one or more links for purchasing tickets to live performances associated with the content creator. For example, where the content item is a music video associated with a particular band or other musical group, the feature can allow a link for purchasing concert tickets associated with the band or musical group to be presented in the page where the content item is presented (e.g., below the video player window where the content item is presented and / or in any other suitable location). Note that in some embodiments, a feature can allow a link to purchase tickets to be presented in conjunction with any other suitable page or content (such as a page associated with a content channel created by the content creator and / or any other suitable content).

[0073] At 204, the process 200 can determine whether the content creator is eligible to use the feature for generating user-generated content. In some embodiments, the process 200 can determine whether the content creator is eligible to use the feature based on any suitable information and using any suitable (multiple) techniques. For example, in some embodiments, the process 200 can determine whether the content creator is eligible to use the feature based on the content items that the content creator has previously uploaded to the media content platform. As a more specific example, in some embodiments, the process 200 can use any suitable classifier or machine learning technology to determine the risk score indicating the possibility that the content creator is likely to upload video content including one or more types of offensive content based on previously uploaded content items. As a specific example, the process 200 can determine that a content creator with a risk score below a predetermined threshold is not eligible to use the feature. As another specific example, in some embodiments, the process 200 can determine that a content creator with a risk score above a predetermined threshold is eligible to use the feature. As another more specific example, in some embodiments, the process 200 can determine the number of times the content creator has previously violated one or more policies associated with the media content platform (e.g., a policy prohibiting specific types of offensive content, and / or any other suitable policy). As a specific example, in some embodiments, process 200 can determine that a content creator who has violated one or more policies of the media content platform more than a predetermined threshold number of times (e.g., more than once, more than twice, more than once in the past month, and / or any other suitable number of times) is ineligible to use the feature.

[0074] As another example, in some embodiments, process 200 can determine whether a content creator qualifies for the feature based on information associated with the content creator, such as the size of the content creator's audience and / or a location associated with the content creator. As a more specific example, in some embodiments, process 200 can determine that a content creator with a relatively small audience is not eligible to use the feature. As another more specific example, in some embodiments, process 200 can determine that a content creator with a relatively large number of viewers qualifies for the feature. In some embodiments, audience size can be determined based on any suitable information such as the total number of views of content items previously uploaded by the content creator, the average number of views of content items previously uploaded by the content creator, the number of subscribers to a content channel created by the content creator, and / or any other suitable metric.

[0075] As yet another example, in some embodiments, process 200 can determine whether a content creator is eligible to use the feature based on a manual review of content posted by the content creator. As a more specific example, in some embodiments, a human reviewer can review one or more content items previously posted by the content creator and / or content items uploaded by the content creator as described above in conjunction with block 202 and can determine whether the content items violate one or more policies of the media content platform. In some embodiments, in response to determining that the content item violates one or more policies of the media content platform, process 200 can determine that the content creator is not eligible to use the feature.

[0076] As yet another example, in some embodiments, process 200 can determine whether the content creator is eligible to use the feature based on any other suitable information associated with the content creator and / or associated with the content item associated with the content creator. As a more specific example, in some embodiments, process 200 can determine whether the content creator has previously been approved to join a program associated with a media content platform, which allows the content creator to use specific features associated with membership in the program. As another more specific example, in some embodiments, process 200 can determine whether the age of the content creator is greater than a predetermined age (e.g., over 18 years old, and / or over any other suitable age). As yet another more specific example, process 200 can determine whether the content item associated with the content creator is associated with a specific genre (e.g., age-restricted content, content for children, music video content, and / or content of any other suitable genre). As a specific example, in some embodiments, if the content item associated with the content item generally belongs to a specific genre, such as age-restricted content, content for children, music video content, etc., process 200 can determine that the content creator is not eligible to use the feature.

[0077] As another example, in some embodiments, process 200 can determine whether a content creator is eligible to use the feature based on a determination that viewers of content created by the content creator are satisfied or dissatisfied with the content creator's use of the feature. As a more specific example, in some embodiments, where the feature includes providing privileges to viewers who purchase memberships associated with content produced by the content creator, process 200 can determine whether users who have purchased memberships are generally satisfied or dissatisfied with their purchases. As another more specific example, in some embodiments, where the feature includes allowing viewers to purchase pinned comments, pinned animated images, or other messages or chat content, process 200 can determine whether users who have purchased pinned messages or chat content are generally satisfied or dissatisfied with their purchases. In some embodiments, process 200 can determine a viewer satisfaction metric in any suitable manner. For example, in some embodiments, process 200 can determine the number of viewers who have canceled memberships or the proportion of viewers, the number of viewers who have left negative comments on the privileges provided or the proportion of viewers, the number of viewers who have unsubscribed from content channels produced by content creators or the proportion of viewers, and / or any other suitable satisfaction amount. In some embodiments, process 200 can determine that the content creator is ineligible for the feature in response to determining that the calculated satisfaction amount is below a predetermined threshold. As another example, in some embodiments, process 200 can determine that the content creator is ineligible for the feature in response to determining that more than a predetermined percentage of viewers (e.g., more than 10%, more than 20%, and / or any other suitable percentage) have canceled a purchased membership or left a negative review related to the offered privileges or purchase options.

[0078] Note that in the case where the content item corresponds to a submitted comment, process 200 can determine whether the user who submitted the comment is eligible to post the comment to a particular chat or a particular comment section. For example, in the case where the comment is submitted in conjunction with a chat or comment section that requires subscription to a particular content channel of a media content platform or requires satisfying any other suitable criteria, process 200 can determine whether the user who submitted the comment is a subscriber to the content channel or satisfies any other suitable criteria. Additionally or alternatively, in some embodiments, in the case where the content item corresponds to a submitted comment, box 204 can be omitted.

[0079] Additionally, note that in some embodiments, process 200 can determine whether a content creator qualifies for the feature based on any suitable combination of information. For example, in some embodiments, process 200 can determine whether a content creator qualifies for the feature based on a combination of a risk score associated with the content creator, information indicating the size of the content creator's audience, information indicating previous violations of policy by the content creator, and / or manual review of content items. In some embodiments, process 200 can combine information in any suitable manner. For example, in some embodiments, process 200 can assign a score to each information item and can determine a total score associated with the content creator based on the score of each information item. In some embodiments, process 200 can then determine whether the content creator qualifies for the feature based on a determination of whether the total score associated with the content creator exceeds a predetermined threshold.

[0080] If at 204, process 200 determines that the content creator is not eligible to use the feature ("No" at 204), process 200 can proceed to 206 and can reject the content item. In some embodiments, process 200 can reject the content item in any suitable manner. For example, in some embodiments, process 200 can prevent the content item from being uploaded to a server associated with a media content platform. As another example, in some embodiments, process 200 can transmit a message to the content creator and / or cause a notification indicating that the content item has been rejected to be presented on a user device associated with the content creator. In some such embodiments, the message and / or notification can include any suitable information, such as the reason why the content item has been rejected (for example, due to a previous violation of one or more policies associated with a media content platform by the content creator, and / or any other suitable reason). As another example, in some embodiments, in the case where the content item corresponds to a comment to be posted to a chat or comment section, process 200 can forbid the comment to be posted to a chat or comment section.

[0081] Note that in some embodiments, in response to determining that the content creator is not eligible to use the feature, process 200 can allow the content item to be uploaded but can deny the content creator from using the feature. For example, where process 200 determines that the content creator has not previously violated any policy associated with the media content platform but the content creator is not eligible to use the feature for other reasons (e.g., because the content creator does not have a large enough audience, because the content creator has not previously uploaded more than a predetermined number of content items, and / or for any other suitable reason), process 200 can allow the content item to be uploaded to the media content platform but can prevent or prohibit the content creator from using the feature.

[0082] If at 204, process 200 determines that the content creator is eligible to use the feature ("yes" at 204), process 200 can proceed to 208 and can determine whether to selectively bypass review of the user-generated content item. In some embodiments, process 200 can determine whether to bypass review of the content item based on any suitable information.

[0083] For example, in some embodiments, if the content item includes objectionable content, the process 200 can determine whether to bypass the review of the content item based on the relative risk of the release of the content item. In some embodiments, the relative risk of the release of the content item can be determined based on any suitable information such as the expected audience size of the content item (e.g., the predicted number of people who are likely to watch the content item), whether the content creator will receive any revenue from viewers of the content item, the expected revenue that the content creator will receive from viewers of the content item, and / or any other suitable information. Note that in some embodiments, the process 200 can determine that the content item will be assigned a relatively high risk score in response to determining that the content item meets any specific criteria. For example, in some embodiments, the process 200 can determine that the content item will be assigned a relatively high risk score in response to determining that the content item is provided in exchange for payment by the viewer (e.g., in the case where the content item is exclusive content provided to users who pay for the content, in the case where the content item is a privilege provided to users who pay for membership to a content channel and / or any other suitable type of content).

[0084] In some embodiments, process 200 can determine the relative risk in any suitable manner. For example, in some embodiments, process 200 can identify multiple risk scores (e.g., a first risk score corresponding to an estimated audience size, a second risk score corresponding to the expected revenue of the content creator from viewers of the content item, and / or any other suitable risk score), and can combine the multiple risk scores in any suitable manner (e.g., as a weighted average and / or in any other suitable manner). As a more specific example, in the case where the content item corresponds to a content channel to be posted to a relatively large expected audience (e.g., based on the channel having more than a predetermined number of subscribers, based on the channel having more than a predetermined number of previous views of content associated with the channel, and / or based on any other suitable metric), process 200 can determine that the relative risk of the release of the content item is high. As another more specific example, in the case where the content item corresponds to a video content item that has been monetized in any suitable manner (e.g., due to the insertion of advertisements, because the content item is provided to users who have paid to receive the content item, and / or in any other suitable manner), process 200 can determine that the relative risk of the release of the content item is high. As yet another more specific example, where the content item corresponds to a comment to be posted in the comments section, the process 200 can determine that the relative risk of publication of the content item is low. In some embodiments, the process 200 can determine that a content item associated with a relative risk of publication exceeding a predetermined threshold is to be manually reviewed (i.e., not bypassing manual review, or "No" at 208). Note that in some embodiments, where the process 200 determines that a content item is associated with a relative risk of publication exceeding a certain predetermined threshold, the process 200 can determine that the content item is to be automatically rejected.

[0085] As another example, in some embodiments, process 200 can determine that review of a content item is to be bypassed in response to determining that the content item is similar to or matches a previously uploaded content item that has been previously reviewed. As a more specific example, in some embodiments, process 200 can determine that the content item is similar to or matches a content item that has been previously reviewed and approved. As another more specific example, in some embodiments, process 200 can determine that the content item is similar to or matches a content item that has been previously reviewed and rejected.

[0086] Note that in some embodiments, process 200 can use any suitable technique or combination of techniques to determine whether an uploaded content item is similar to or matches a previously reviewed content item. For example, in some embodiments, process 200 can compare one or more audio and / or video fingerprints of an uploaded content item to a database of previously uploaded content items to identify a match to a previously uploaded and reviewed content item.

[0087] In addition, note that in some embodiments, process 200 can determine the degree of similarity between the uploaded content item and at least one previously reviewed content item to determine that the review is to be bypassed. For example, in some embodiments, process 200 can determine that the similarity of the uploaded content item to the previously reviewed content item has at least a predetermined threshold (e.g., at least 70% similarity, at least 80% similarity, and / or any other suitable threshold) to determine that the review is to be bypassed. In some embodiments, the predetermined threshold of similarity can be determined or set in any suitable manner. For example, in some embodiments, the predetermined threshold of similarity can be determined by process 200 so that when the predetermined threshold of similarity is applied, a predetermined range of content items is sent for manual review. That is, in some embodiments, process 200 can adjust the predetermined threshold of similarity over time based on the number of content items uploaded to the media content platform, thereby allowing adjustment of the amount of manual review performed.

[0088] Additionally, note that in some embodiments, process 200 can determine whether manual review is to be bypassed based on any suitable combination of the relative risk of publication of the content item and the similarity between the uploaded content item and previously uploaded content items. For example, in some embodiments, process 200 can calculate a risk score indicating the relative risk of publication of the content item and a similarity score indicating the similarity between the content item and previously uploaded content items, and can determine whether manual review is to be bypassed based on the risk score and the similarity score. As a more specific example, in some embodiments, process 200 can determine that manual review is not to be bypassed ("No" at 208) in response to determining that the risk score exceeds a first predetermined threshold and the similarity score is below a second predetermined threshold (i.e., not sufficiently similar to previously uploaded content items). As another more specific example, in some embodiments, process 200 can determine that manual review is to be bypassed ("Yes" at 208) in response to determining that the risk score is below a first predetermined threshold and the similarity score is above a second predetermined threshold.

[0089] In some embodiments, process 200 can determine that certain types of content items are to be manually reviewed regardless of the risk score. For example, in some embodiments, process 200 can determine that content items that are offered in exchange for payment by viewers of the content item (e.g., exclusive content offered to paying users and / or any other suitable type of paid content) are always to be manually reviewed. As another example, in some embodiments, process 200 can determine that content items generated by content creators offering paid content for the first time are to be manually reviewed. Note that in some embodiments, paid content items can include video content, audio content, "super stickers," "super reviews" (as described above in conjunction with box 202), and / or any other suitable type of content offered to viewers in exchange for payment.

[0090] If at 208, process 200 determines that the review of the content item is to be bypassed ("yes" at 208), process 200 can proceed to 210 and can determine whether the content item is rejected due to being abusive content. In some embodiments, process 200 can determine whether the content item is to be rejected based on whether it is determined at box 208 that the uploaded content item is similar to or matches a previously rejected content item. For example, in some embodiments, process 200 can determine that the uploaded content item is similar to or matches a previously rejected content item that was rejected because the content item includes one or more types of objectionable content, and in some such embodiments, process 200 can determine that the content item is to be rejected. As another example, in some embodiments, process 200 can determine that the uploaded content item is similar to or matches a previously approved content item, and in some such embodiments, process 200 can determine that the content item will not be rejected. As yet another example, in some embodiments, process 200 can use any suitable (multiple) techniques to determine the likelihood that the content item includes any particular type of objectionable content. As a more specific example, in some embodiments, process 200 can use any suitable classifier or combination of classifiers to determine the likelihood that a content item includes objectionable content. Note that where a content item corresponds to video content or audio content, process 200 can use a classifier that takes as input any suitable audio or video fingerprint of the content item. In some embodiments, where a content item corresponds to a submitted review, process 200 can use a classifier that takes as input any suitable text or link included in the review to determine the likelihood that the text includes objectionable content or that the link is to a website associated with objectionable content. Continuing with this example, in some embodiments, process 200 can determine that a content item is to be rejected in response to a determined likelihood that the content item includes objectionable content exceeding a predetermined threshold (e.g., greater than 70%, greater than 80%, and / or any other suitable threshold).

[0091] If, at 210, process 200 determines that the content item is to be rejected ("YES" at 210), process 200 can reject the content item at 206, as described above.

[0092] If at 210, process 200 determines that the content item will not be rejected ("No" at 210), process 200 can proceed to 212 and can post the content item to the media content platform. In some embodiments, process 200 can post the content item to the media content platform in any suitable manner. For example, in some embodiments, process 200 can upload the content item to a specific channel associated with the content creator. As another example, in some embodiments, process 200 can provide a link to the content item to other user devices to stream or download the content item. As another example, in some embodiments, process 200 can transmit a notification indicating that the content item has been approved to the content creator. In some such embodiments, the content creator can then initiate posting of the content item in any suitable manner, for example, by selecting any suitable selectable input that makes the content item available to other viewers. As yet another example, in the case where the content item corresponds to a live stream of the content, process 200 can provide the live stream to the viewer after any suitable time delay (e.g., after thirty seconds, after one minute, and / or after any other suitable time delay). As yet another example, where the content item corresponds to a comment to be posted to a comments section or chat, process 200 enables the comment to be added to the chat or comments section in any suitable manner (e.g., at the bottom of the chat, at the bottom of the comments section, at a pinned location of the chat or comments section, and / or in any other suitable manner).

[0093] Referring back to block 208, if at 208, process 200 determines that review of the content item is not to be bypassed ("No" at 208), process 200 can proceed to 214 and can add the content item to a queue of content items to be reviewed. Figure 1As described in block 104 of FIG. 20 , in some embodiments, process 200 can add content items to a queue of content items to be reviewed in any suitable manner. For example, in some embodiments, content items can be added to a queue based on the date or time at which the content items are received at block 202 (e.g., such that items received earlier are placed earlier in the queue, and / or placed in any other suitable manner). As another example, in some embodiments, content items can be added to a queue based on the popularity of other content items associated with the content creator. As a more specific example, in some embodiments, content items associated with a content creator who has previously uploaded content items that have received more than a predetermined number of views, shares, comments, etc. can be placed higher in the queue. As yet another example, in some embodiments, where the posting of a content item is time-sensitive (e.g., where the content item is live stream content, where the content item is a comment in a live chat and / or any other suitable time-sensitive context), process 200 can add the content item to a relatively high position within the queue (e.g., to the front of the queue and / or in any other suitable position), thereby allowing the content item to be manually reviewed more quickly.

[0094] As above combined Figure 1 As described, in response to a content item being moved to the head of a queue of content items to be reviewed, the content item can be assigned to a reviewer. Figure 1 As described, the content item can be presented on a user device associated with the reviewer. In some embodiments, the reviewer can then use the user device associated with the reviewer to enter any information associated with the decision related to the content item, such as whether the content item is determined to include one or more types of objectionable content, the frame or timestamp of the content item that includes the objectionable content if the content item corresponds to video or audio content, one or more policies that the content item violates, and / or any other suitable information.

[0095] At 216, process 200 can receive a decision associated with the content item. In some embodiments, process 200 can receive the decision in any suitable manner. For example, in some embodiments, a server performing process 200 can receive a message from a user device associated with a reviewer who reviewed the content item. In some such embodiments, the message can include any suitable information, such as an indication that the content item does not violate any policy associated with the media content platform, an indication that the content item does violate one or more policies associated with the media content platform, an indication of one or more types of objectionable content included in the content item, a timestamp of the content item including one or more types of objectionable content, and / or any other suitable information.

[0096] At 218, process 200 can determine whether the decision indicates a policy violation.

[0097] If, at 218 , process 200 determines that the decision does not indicate a policy violation (“NO” at 218 ), process 200 can post the content item to the media content platform at 212 as described above.

[0098] If at 218, process 200 determines that the decision does indicate a policy violation ("yes" at 218), process 200 can proceed to 220 and can transmit a message indicating a policy violation to the content creator. In some embodiments, process 200 can transmit a message to the content creator in any suitable manner. For example, in some embodiments, process 200 can transmit a message to an email address associated with the content creator. As another example, in some embodiments, process 200 can transmit a message in conjunction with a user account associated with the content creator and a media content platform. As a more specific example, in some embodiments, the message can be a notification presented on the homepage of a user account associated with the content creator in conjunction with the media content platform. For example, this can provide precise feedback to the content creator about policy violations and other reasons for taking actions to prohibit user-generated content from being made available on the media content platform.

[0099] In some embodiments, the message can include any suitable information. For example, in some embodiments, the message can indicate one or more policies that the content item violates. As a more specific example, in some embodiments, the message can indicate that the content item includes one or more types of offensive content that violate one or more policies of the media content platform. As another example, in some embodiments, the message can use any suitable timing information, such as the timestamp of the content item, the frame of the content item and / or any other suitable timing information, to indicate one or more parts of the content item that violates the policy of the media content platform. As another example, in some embodiments, the message can indicate a penalty. As a more specific example, in some embodiments, the message can indicate that the content item has been blocked from being uploaded to the media content platform. As another more specific example, in some embodiments, the message can indicate that the content creator will be prohibited from uploading the content item for a predetermined duration (e.g., for one month, 90 days and / or at any other suitable duration). As another more specific example, in some embodiments, the message can indicate that the account associated with the content creator will be terminated. As another more specific example, in some embodiments, the message can indicate that the content creator will not be allowed to use the feature for a predetermined duration (e.g., thirty days, ninety days and / or any other suitable duration). As yet another more specific example, in some embodiments, the message can indicate that the content creator will not be allowed to stream live content for a predetermined duration (e.g., thirty days, ninety days, and / or any other suitable duration). As yet another more specific example, where the content item corresponds to a comment to be posted to a chat or comment section, the message can indicate that the user will not be allowed to post the comment to the comment section associated with the rejected comment or to any comment section associated with the media content platform for a certain duration.

[0100] Note that in some embodiments, process 200 can determine a penalty based on the policy violated by the content item. For example, in some embodiments, process 200 can determine a penalty based on the severity of the violation. As a more specific example, in some embodiments, process 200 can determine a first penalty (e.g., the content item will be blocked) in response to determining that a policy violation corresponds to including a first type of objectionable content in the content item and a second penalty (e.g., a user account associated with the content creator will be terminated) in response to determining that a policy violation corresponds to including a second type of objectionable content in the content item. In addition, note that in some embodiments, process 200 can determine to apply a specific penalty (e.g., account termination, prohibition of uploading content for a predetermined duration, and / or any other suitable penalty) in response to determining that a content creator has violated one or more policies more than a predetermined number of times (e.g., more than three violations, more than five violations, and / or any other suitable number of times). For example, this can discourage policy violations with increasingly severe enforcement actions. In addition, note that in some embodiments, process 200 can determine a penalty based on the type of content item. For example, in some embodiments, where the content item corresponds to an upload of video content or audio content available to other viewers, the process 200 can identify a penalty associated with a temporary or permanent block on uploading other video or audio content to the media content platform. As another example, in some embodiments, where the content item corresponds to a comment submitted to a chat or comment section, the process 200 can identify a penalty associated with a temporary or permanent block on the submission of the comment.

[0101] Note that in some embodiments, the process 200 can add the content item to the second queue so that the content item is then reviewed a second time by a second reviewer before proceeding to block 220. That is, in some embodiments, in response to a determination that the decision indicates a policy violation, the process 200 can cause the content item to be reviewed a second time by a second reviewer different from the first reviewer to confirm that the content item does violate the policy of the media content platform.

[0102] At 222, process 200 can update an account associated with the content creator based on the policy violation. For example, in some embodiments, process 200 can update an account to indicate that the uploaded content item violated a particular policy. As another example, in some embodiments, process 200 can update an account to indicate that the uploaded content item was determined to include a particular type of objectionable content. As yet another example, in some embodiments, process 200 can update a count indicating the number of times the content creator has uploaded content items that have violated a policy associated with the media content platform. Note that in some embodiments, process 200 can use information associated with the content creator's account to determine a penalty for the content creator as described above in conjunction with block 220.

[0103] At 224, process 200 can prevent the content item from being uploaded to the media content platform. Note that in the event that the content item is a live stream of content, process 200 can stop or prohibit the live stream from being streamed to any user device.

[0104] Note that although processes 100 and 200 are described above as being associated with reviewing content items (e.g., videos, live streamed video content, and / or any other suitable content items), in some embodiments, processes 100 and 200 can be used to review any other suitable type of content, such as comments in a comment section, comments in a social network post, messages in a live chat, and / or any other suitable comments. For example, in some embodiments, process 200 can determine whether to add a particular comment to a queue of comments to be reviewed (e.g., at blocks 208 and 214) based on a classifier applied to a set of comments. As a more specific example, in some embodiments, process 200 can add a comment to a queue of comments to be manually reviewed in response to determining, using any suitable machine learning classifier, that the comment is likely to contain one or more types of objectionable content (e.g., hate speech, objectionable language, objectionable images or links, and / or any other suitable type of objectionable content). Continuing with the example, in some embodiments, the process 200 can add the comment to a queue of comments to be reviewed (e.g., at block 214), and can proceed through the blocks of the process 200 to determine whether the comment will be removed from the live chat or comment section to which it was posted based on the manual review. As another example, in some embodiments, the processes 100 and 200 can be used to review privileges provided by a content creator in conjunction with the purchase of a membership, as described above in conjunction with block 202. As a more specific example, in some embodiments, the process 200 can determine whether a particular provided privilege will be added to a queue of privileges to be reviewed (e.g., at blocks 208 and 214) based on any suitable information such as the following and / or based on any other suitable criteria: whether a viewer who has purchased a membership has received a complaint, whether a viewer exceeding a predetermined threshold has cancelled the purchased membership. Continuing with the example, the process 200 can then proceed through the blocks of the process 200 to determine whether the provision of the privilege is to be canceled or suspended based on the manual review.

[0105] Steering Figure 3 , shows an illustrative example 300 of hardware for identifying abusive content items that can be used according to some embodiments of the disclosed subject matter. As shown, the hardware 300 can include a server 302, a communication network 304, and / or one or more user devices 306, such as a content creator device 308 and / or a reviewer device 310.

[0106] Server 302 can be any suitable server for storing information, data, media content, and / or any other suitable type of content. For example, in some embodiments, server 302 can store user-generated media content uploaded by one or more content creator devices 308. As a more specific example, in some embodiments, user-generated media content can include videos, movies, photos, slides, animations, documents, and / or any other suitable user-generated media content. As another example, in some embodiments, server 302 can store information about one or more content creators who have uploaded user-generated content to server 302, such as whether a particular content creator has previously uploaded content that violates a particular policy of a media content platform, and / or any other suitable information. In some embodiments, server 302 can implement any suitable process to queue uploaded content items for review, store decisions related to uploaded content items (e.g., decisions indicating whether a content item contains objectionable content and / or any other suitable decisions), and / or perform any other suitable functions. For example, in some embodiments, server 302 can implement any suitable box of process 200, such as in Figure 2 In and above combined Figure 2 Described.

[0107] In some embodiments, the communication network 304 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 304 can include any one or more of the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 306 can be connected to the communication network 304 by one or more communication links (e.g., communication link 312), which can be linked to the server 302 via one or more communication links (e.g., communication link 314). The communication link can be any communication link suitable for conveying data among the user device 306 and the server 302, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0108] The user device 306 can include any one or more user devices. For example, in some embodiments, the user device 306 can include a content creator device 308. In some embodiments, the content creator device 308 can be suitable for capturing user-generated content, uploading user-generated content, receiving notifications indicating decisions related to the uploaded content items, and / or any user device for performing any other suitable function. As another example, in some embodiments, the user device 306 can include a reviewer device 310. In some embodiments, the reviewer device 310 can be suitable for viewing uploaded content items, transmitting decisions related to the uploaded content items, and / or any user device for performing any other suitable function. In some embodiments, the user device 306 can include any suitable type of user device. For example, in some embodiments, the user device 306 can include a mobile phone, a wearable computer, a tablet computer, a desktop computer, a laptop computer, a vehicle entertainment system, a game console, a television, and / or any other suitable user device.

[0109] Although server 302 is illustrated as one device, in some embodiments, the functions performed by server 302 can be performed using any suitable number of devices. For example, in some embodiments, multiple devices can be used to implement the functions performed by server 302.

[0110] Although in Figure 3 Two user devices 308 and 310 are shown in FIG. 3 to avoid overcomplicating the diagram, but in some embodiments any suitable number of user devices and / or any suitable type of user devices can be used.

[0111] In some embodiments, server 302 and user device 306 can be implemented using any suitable hardware. For example, in some embodiments, devices 302 and 306 can be implemented using any suitable general-purpose computer or special-purpose computer. For example, a mobile phone can be implemented using a special-purpose computer. Any such general-purpose computer or special-purpose computer can include any suitable hardware. For example, Figure 4 As illustrated in example hardware 400 of , such hardware can include a hardware processor 402, memory and / or storage 404, an input device controller 406, an input device 408, a display / audio driver 410, display and audio output circuitry 412, a communication interface 414, an antenna 416, and a bus 418.

[0112] In some embodiments, hardware processor 402 can include any suitable hardware processor, such as a microprocessor, a microcontroller, a digital signal processor, dedicated logic, and / or any other suitable circuit system for controlling the functions of a general purpose computer or a special purpose computer. In some embodiments, hardware processor 402 can be controlled by a server program stored in a memory and / or storage device of a server such as server 302. In some embodiments, hardware processor 402 can be controlled by a computer program stored in a memory and / or storage device 404 of user device 306.

[0113] In some embodiments, the memory and / or storage device 404 can be any suitable memory and / or storage device for storing programs, data and / or any other suitable information. For example, the memory and / or storage device 404 can include random access memory, read-only memory, flash memory, hard disk storage, optical media and / or any other suitable memory.

[0114] In some embodiments, input device controller 406 can be any suitable circuitry for controlling and receiving input from one or more input devices 408. For example, input device controller 406 can be circuitry for receiving input from a touch screen, from a keyboard, from one or more buttons, from a voice recognition circuit, from a microphone, from a camera, from an optical sensor, from an accelerometer, from a temperature sensor, from a near-field sensor, from a pressure sensor, from an encoder, and / or any other type of input device.

[0115] In some embodiments, display / audio driver 410 can be any suitable circuitry for controlling and driving outputs to one or more display / audio output devices 412. For example, display / audio driver 410 can be circuitry for driving a touch screen, a flat panel display, a cathode ray tube display, a projector, one or more speakers, and / or any other suitable display and / or presentation device.

[0116] Communication interface 414 can be any suitable circuitry for interfacing with one or more communication networks, such as computer network 304. For example, interface 414 can include network interface card circuitry, wireless communication circuitry, and / or any other suitable type of communication network circuitry.

[0117] In some embodiments, antenna 416 can be any suitable antenna or antennas for wirelessly communicating with a communication network, such as communication network 304. In some embodiments, antenna 416 can be omitted.

[0118] In some embodiments, bus 418 can be any suitable mechanism for communicating between two or more components 402 , 404 , 406 , 410 , and 414 .

[0119] According to some embodiments, any other suitable components can be included in hardware 400 .

[0120] In some embodiments, the present invention can be performed or executed in any order or sequence not limited to the order and sequence shown and described in connection with the figures. Figure 1 and Figure 2 In addition, it is possible to implement or execute substantially simultaneously or in parallel as appropriate. Figure 1 and Figure 2 Some of the above boxes can be omitted to reduce waiting time and processing time. Figure 1 and Figure 2 Some of the above boxes of the process.

[0121] In some embodiments, any suitable computer-readable medium can be used to store instructions for performing the functions and / or processes herein. For example, in some embodiments, the computer-readable medium can be temporary or non-temporary. For example, the non-temporary computer-readable medium can include media such as the following: non-temporary forms of magnetic media (such as hard disks, floppy disks, and / or any other suitable magnetic media), non-temporary forms of optical media (such as optical disks, digital video disks, blue-ray disks, and / or any other suitable optical media), non-temporary forms of semiconductor media (such as flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and / or any other suitable semiconductor media), any suitable medium that does not fly or have any permanent appearance during transmission, and / or any suitable tangible medium. As another example, a temporary computer-readable medium can include a signal on a network, in a wire, a conductor, an optical fiber, a circuit, any suitable medium that flies during transmission and does not have any permanent appearance, and / or any suitable intangible medium.

[0122] Thus, methods, systems, and media for identifying abusive content items are provided.

[0123] Although the present invention has been described and illustrated in the foregoing illustrative embodiments, it should be understood that the present disclosure has been made by way of example only and that many changes can be made in the details of the embodiments of the present invention without departing from the spirit and scope of the present invention, which is limited only by the following claims. The features of the disclosed embodiments can be combined and rearranged in various ways.

Claims

1. A method for identifying abusive content items, the method include: receiving, at a server, from a user device associated with a user, a request to provide user-generated content on a media content platform; determining a combined risk of a release of the user-generated content if the user-generated content includes objectionable content, wherein the combined risk of a release is a combination of a plurality of released relative risk values, wherein a first value of the plurality of released relative risk values ​​corresponds to an estimated audience size for the user-generated content, and wherein a second value of the plurality of released relative risk values ​​corresponds to an expected amount of subscription revenue from viewers of the user-generated content; in response to receiving the request, determining whether the user-generated content is to be reviewed by one or more review users based on one or more bypass criteria indicating that a combined risk of the publication of the user-generated content exceeds a threshold; In response to determining that the user-generated content is to be reviewed based on the one or more bypass criteria, adding a review request to a queue for a review user, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and determining whether a decision in response to the review request indicates that the user-generated content is to be provided on the media content platform, wherein: In response to determining that the decision indicates that the user-generated content violates at least one policy associated with the media content platform, (i) a first message is transmitted to the user of the user device, the first message indicating the at least one violated policy and a penalty to be applied to the user account corresponding to the user, wherein the penalty is determined by determining a type of the objectionable content included in the user-generated content and determining whether to prohibit the user-generated content from being provided on the media content platform or to terminate the user account of the user based on the type of the objectionable content included in the user-generated content, and (ii) based on the determination to prohibit the user-generated content from being provided on the media content platform or to terminate the user account of the user, a corresponding action is taken; and In response to determining that the decision indicates that the user-generated content does not violate the at least one policy associated with the media content platform, a second message is transmitted to the user of the user device, the second message indicating that the user-generated content is eligible to be provided on the media content platform.

2. The method according to claim 1, in, The user is a content creator, and wherein the request to provide user-generated content comprises an upload request to store the user-generated content on a storage device associated with the media content platform.

3. The method according to claim 1, in, The user is a content creator, and wherein the request to provide user-generated content comprises an initial request to start a live stream using the media content platform.

4. The method according to claim 1, in, The user is a viewer, and wherein the request to provide user-generated content includes interacting with a content creator's content features.

5. The method according to claim 1, further comprising determining whether the user is eligible to provide the user-generated content on the media content platform by: calculating a risk score indicative of risk associated with the user account of the user based on other user-generated content that has previously been provided to the media content platform; and comparing the risk score to a threshold risk score corresponding to a content type of the user-generated content, in, Based on the comparison of the risk score to the threshold risk score, the user is deemed eligible to provide the user-generated content.

6. The method of claim 1, further comprising determining whether the user is eligible to provide the user-generated content on the media content platform based on audience information associated with the user account of the user.

7. The method of claim 1, further comprising determining whether the user is eligible to provide the user-generated content on the media content platform based on location information associated with the user account of the user.

8. The method of claim 1, further comprising determining whether the user is eligible to provide the user-generated content on the media content platform based on enforcement actions of previous policy violations associated with the user account.

9. The method of claim 1 , further comprising determining whether the user is eligible to provide the user-generated content on the media content platform by transmitting a second review request to the queue of review users, in, The second review request includes information associated with the user account corresponding to the user and prompts the review user to determine whether the information associated with the user account violates at least one policy associated with the media content platform.

10. The method of claim 1 further comprising, in response to receiving the request to provide the user-generated content on the media content platform, causing a timing indication to be presented indicating when the decision in response to the review request will be provided.

11. The method according to claim 1, further comprising: include: receiving an indicia from a viewer of the user-generated content on the media content platform, wherein the indicia indicates a potential policy violation; and In response to receiving the flag, adding a subsequent review request to the queue of the reviewing user who previously reviewed the user-generated content, wherein the subsequent review request includes the user-generated content and the flag from the viewer, and wherein the user-generated content is restricted from being provided on the media content platform until a decision is received in response to the subsequent review request.

12. The method according to claim 1, in, Determining whether the user-generated content is to be reviewed by one or more review users based on the one or more bypass criteria further comprises: identifying second user-generated content that is similar to the user-generated content, wherein the second user-generated content has been previously reviewed; and Based on a similarity between the user-generated content and the second user-generated content, it is determined that the user-generated content is not to be reviewed by the one or more reviewing users.

13. The method according to claim 12, further comprising: include: determining that the second user-generated content was previously restricted from being made available on the media content platform; as well as In response to determining that the second user-generated content was previously restricted from being made available on the media content platform, performing the corresponding action of prohibiting the user-generated content from being made available on the media content platform, wherein the first message indicates the restriction based on the second user-generated content.

14. The method according to claim 1, in, Determining whether the user-generated content is to be reviewed by the one or more reviewing users also includes determining whether at least one of the user-generated content and content related to the user-generated content has been previously flagged by a threshold number of viewers.

15. The method of claim 1, further comprising updating the user account associated with the user to indicate the at least one policy associated with the media content platform that is violated by the user-generated content.

16. The method according to claim 1, in, The penalty to be applied to the user account corresponding to the user is based on a number of previous policy violations by the content creator.

17. The method according to claim 1, in, The penalty to be applied to the user account corresponding to the user is based on a severity of the at least one policy associated with the media content platform that is violated by the user-generated content.

18. The method according to claim 1, further comprising: include: Receiving an appeal request for the decision from the user equipment, wherein the appeal request includes an appeal text; adding the appeal request including the appeal text and the user-generated content to a second queue of a second review user, the second review user being different from the review user that provided the decision indicating that the user-generated content is to be restricted from being provided on the media content platform; receiving, from a user device associated with the second reviewing user, an appeal decision corresponding to the appeal request; and At least one corresponding action to be performed on the user-generated content is determined based on the decision and the appeal decision.

19. The method according to claim 1, in, In response to determining that the decision indicates that the user-generated content does not violate the at least one policy associated with the media content platform, the second message indicates a time when the user-generated content is automatically made available on the media content platform.

20. A system for identifying abusive content items, the system include: Memory; as well as a hardware processor, which, when executing computer-executable instructions stored in the memory, is configured to: receiving, from a user device associated with a user, a request to provide user-generated content on a media content platform; determining a combined risk of a release of the user-generated content if the user-generated content includes objectionable content, wherein the combined risk of a release is a combination of a plurality of released relative risk values, wherein a first value of the plurality of released relative risk values ​​corresponds to an estimated audience size for the user-generated content, and wherein a second value of the plurality of released relative risk values ​​corresponds to an expected amount of subscription revenue from viewers of the user-generated content; in response to receiving the request, determining whether the user-generated content is to be reviewed by one or more review users based on one or more bypass criteria indicating that a combined risk of the publication of the user-generated content exceeds a threshold; In response to determining that the user-generated content is to be reviewed based on the one or more bypass criteria, adding a review request to a queue for a review user, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and determining whether a decision in response to the review request indicates that the user-generated content is to be provided on the media content platform, wherein: In response to determining that the decision indicates that the user-generated content violates at least one policy associated with the media content platform, (i) a first message is transmitted to the user of the user device, the first message indicating the at least one violated policy and a penalty to be applied to the user account corresponding to the user, wherein the penalty is determined by determining a type of the objectionable content included in the user-generated content and determining whether to prohibit the user-generated content from being provided on the media content platform or to terminate the user account of the user based on the type of the objectionable content included in the user-generated content, and (ii) based on the determination to prohibit the user-generated content from being provided on the media content platform or to terminate the user account of the user, a corresponding action is taken; and In response to determining that the decision indicates that the user-generated content does not violate the at least one policy associated with the media content platform, a second message is transmitted to the user of the user device, the second message indicating that the user-generated content is eligible to be provided on the media content platform.

21. A non-transitory computer readable medium containing computer executable instructions which, when executed by a processor, cause the processor to perform a method for identifying abusive content items, the method include: receiving, from a user device associated with a user, a request to provide user-generated content on a media content platform; determining a combined risk of a release of the user-generated content if the user-generated content includes objectionable content, wherein the combined risk of a release is a combination of a plurality of released relative risk values, wherein a first value of the plurality of released relative risk values ​​corresponds to an estimated audience size for the user-generated content, and wherein a second value of the plurality of released relative risk values ​​corresponds to an expected amount of subscription revenue from viewers of the user-generated content; in response to receiving the request, determining whether the user-generated content is to be reviewed by one or more review users based on one or more bypass criteria indicating that a combined risk of the publication of the user-generated content exceeds a threshold; In response to determining that the user-generated content is to be reviewed based on the one or more bypass criteria, adding a review request to a queue for a review user, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and determining whether a decision in response to the review request indicates that the user-generated content is to be provided on the media content platform, wherein: In response to determining that the decision indicates that the user-generated content violates at least one policy associated with the media content platform, (i) a first message is transmitted to the user of the user device, the first message indicating the at least one violated policy and a penalty to be applied to the user account corresponding to the user, wherein the penalty is determined by determining a type of the objectionable content included in the user-generated content and determining whether to prohibit the user-generated content from being provided on the media content platform or to terminate the user account of the user based on the type of the objectionable content included in the user-generated content, and (ii) based on the determination to prohibit the user-generated content from being provided on the media content platform or to terminate the user account of the user, a corresponding action is taken; and In response to determining that the decision indicates that the user-generated content does not violate the at least one policy associated with the media content platform, a second message is transmitted to the user of the user device, the second message indicating that the user-generated content is eligible to be provided on the media content platform.

Images