Method, system, and medium for identifying inappropriate content item

Abstract

To provide a method, a system and a medium for identifying an inappropriate content item.SOLUTION: A method includes the steps of: receiving a request for providing a user-generated content on a media content platform (PF) from a user device; determining whether or not the user-generated content should be examined by an examining user on the basis of an avoidance criterion; adding an examination request to a queue of the examination user; determining whether or not the content has to be provided on the PF; in the case of determining that the content infringes a policy of the PF, transmitting to the user device a message indicating the infringed policy and a penalty to be applied to a user; and transmitting the message indicating that the content is eligible for being provided on the PF to the user device in a case where an action to prohibit provision of the content on the PF is taken and no infringement is determined.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The disclosed subject matter relates to methods, systems, and media for identifying inappropriate content items. More specifically, the disclosed subject matter relates to determining whether user-generated content should be manually reviewed by one or more reviewers or whether the user-generated content can be bypassed from manual review based on one or more criteria, and determining whether the user-generated content is inappropriate content so that corresponding action can be taken on the user-generated content. [Background technology]

[0002] Media content platforms have become the primary mechanism through which users discover and share digital media content. For example, content creators may upload videos or other content items that they have created to the media content platform, which then makes the uploaded content items available to viewers (e.g., by streaming the content item, by downloading the content item, etc.).

[0003] While many user-generated channels on a media content platform contain original, non-inappropriate content, there may be inappropriate content that violates the media content platform's policies and targets unsuspecting users, such as misleading, repetitive, suggestive, obscene, illegal, and / or "click bait" content. For example, certain content items may contain objectionable content, such as violent content, offensive language, adult content, drug-related content, gambling content, etc., that the media content platform may want to remove or otherwise prevent users from accessing. As another example, the media content platform may want to prevent certain features of the service (e.g., features that enable content creators to collect revenue, and / or any other appropriate features) from being used in conjunction with content items that violate these particular policies.

[0004] However, identifying content items that violate policies can be difficult. For example, some services may use human reviewers to manually review uploaded content items, but reviewing each uploaded piece of content to identify policy violations can be time consuming. In such cases, a content item may be available on the media content platform for any number of days or weeks until the manual review process indicates that the content item should be blocked or removed due to a violation of the media content platform's policies. Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, it would be desirable to provide new methods, systems, and mediums for identifying inappropriate content items. [Means for solving the problem]

[0006] SUMMARY OF THE DISCLOSURE Methods, systems, and media are provided for identifying inappropriate content items.

[0007] According to some embodiments of the disclosed subject matter, a method for identifying inappropriate content items is provided, the method including the steps of receiving a request for providing user-generated content on a media content platform from a user device associated with a user; in response to receiving the request, determining whether the user-generated content should be reviewed by one or more screening users based on one or more avoidance criteria; and in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, adding a screening request to a queue of screening users, the screening request including the user-generated content and information associated with a user account corresponding to the user; and notifying the user of the screening request that the user-generated content should be provided on the media content platform. and determining whether the judgment responsive to the determination indicates that the user-generated content item violates at least one policy associated with the media content platform, wherein in response to determining that the determination indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is sent to a user of the user device indicating the at least one violated policy and a penalty to be imposed on a user account corresponding to the user, and (ii) corresponding action is taken to prevent the user-generated content from being provided on the media content platform, and in response to determining that the determination indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is sent to the user of the user device indicating that the user-generated content is eligible for provision on the media content platform.

[0008] In some embodiments, the user is a content creator and the request to provide the user-generated content comprises an upload request for storing the user-generated content on a storage device associated with the media content platform.

[0009] In some embodiments, the user is a content creator and the request to provide the user-generated content comprises a start request to begin a live stream using the media content platform.

[0010] In some embodiments, the user is a reviewer and the request to provide user-generated content includes a content function that interacts with the content creator.

[0011] In some embodiments, the method further comprises determining whether a user is eligible to provide user-generated content on the media content platform by calculating a risk score indicative of a risk associated with the user's user account based on other user-generated content previously provided to the media content platform and comparing the risk score to a threshold risk score corresponding to a content type of the user-generated content, and the user is deemed eligible to provide user-generated content based on comparison of the risk score to the threshold risk score.

[0012] In some embodiments, the method further comprises determining whether the user is eligible to provide user-generated content on the media content platform based on audience information associated with the user's user account.

[0013] In some embodiments, the method further comprises determining whether the user is eligible to provide user-generated content on the media content platform based on location information associated with the user's user account.

[0014] In some embodiments, the method further comprises determining whether a user is eligible to provide user-generated content on the media content platform based on enforcement actions for previous policy violations associated with the user account.

[0015] In some embodiments, the method further comprises determining whether the user is eligible to provide user-generated content on the media content platform by sending a second review request to a queue of the review user, the second review request including information associated with a user account corresponding to the user and prompting the review user to determine whether the information associated with the user account violates at least one policy associated with the media content platform.

[0016] In some embodiments, the method further comprises causing, in response to receiving a request to provide user-generated content on the media content platform, a timing indication to be presented indicating when a decision responsive to the review request should be provided.

[0017] In some embodiments, the method further comprises receiving a flag from a viewer of the user-generated content on the media content platform, the flag indicating a possible policy violation, and in response to receiving the subsequent review request, adding the subsequent review request to a queue of review users that previously reviewed the user-generated content, the subsequent review request including the user-generated content and the flag from the viewer, and the user-generated content is restricted from being provided on the media content platform until a decision responsive to the subsequent review request is received.

[0018] In some embodiments, determining whether the user-generated content should be reviewed by the one or more review users based on the one or more avoidance criteria further comprises identifying a second user-generated content similar to the user-generated content, where the second user-generated content has been previously reviewed, and determining that the user-generated content should not be reviewed by the one or more review users based on the similarity of the user-generated content to the second user-generated content. In some embodiments, the method further comprises determining that the second user-generated content was previously restricted from being made available on the media content platform, and in response to determining that the second user-generated content was previously restricted from being made available on the media content platform, performing a corresponding action that prevents the user-generated content from being made available on the media content platform, wherein the first message indicates the restriction based on the second user-generated content.

[0019] In some embodiments, determining whether the user-generated content should be reviewed by one or more review users further comprises determining whether at least one of the user-generated content and content related to the user-generated content has previously been flagged by a threshold number of viewers.

[0020] In some embodiments, the method further comprises updating a user account associated with the user to indicate at least one policy associated with the media content platform that the user-generated content violates.

[0021] In some embodiments, the penalty to be imposed on the user account corresponding to the user is based on the number of previous policy violations by the content creator.

[0022] In some embodiments, the penalty to be assessed against a user account corresponding to a user is based on the severity of at least one policy associated with the media content platform that the user-generated content violates.

[0023] In some embodiments, the method further comprises receiving a request to appeal the judgment from a user device, the appeal request including appeal text; adding the appeal request including the appeal text and the user-generated content to a queue of a second review user different from the review user that provided the judgment indicating that the user-generated content should be restricted from being provided on the media content platform; receiving an appeal decision corresponding to the appeal request from a user device associated with the second review user; and determining at least one corresponding action to be taken on the user-generated content based on the judgment and the appeal decision.

[0024] In some embodiments, in response to determining that the judgment indicates that the user-generated content item does not violate at least one policy associated with the media content platform, the second message indicates a time during which the user-generated content will be automatically made available on the media content platform.

[0025] According to some embodiments of the disclosed subject matter, a system for identifying inappropriate content items is provided, the system comprising: a memory; and a hardware processor, the hardware processor, when executing computer-executable instructions stored in the memory, receives a request for providing user-generated content on a media content platform from a user device associated with a user; in response to receiving the request, determines whether the user-generated content should be reviewed by one or more screening users based on one or more avoidance criteria; and in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, adds a review request to a queue of screening users, the review request including the user-generated content and information associated with a user account corresponding to the user, and the user-generated content is reviewed by the media content platform. and (ii) a corresponding action is taken to prevent the user-generated content from being provided on the media content platform, and in response to determining that the determination indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is sent to the user of the user device indicating that the user-generated content item is eligible for provision on the media content platform.

[0026] In accordance with certain embodiments of the disclosed subject matter, a non-transitory computer-readable medium includes computer-executable instructions that, when executed by a processor, cause the processor to perform a method for identifying inappropriate content items, the method including the steps of receiving a request for providing user-generated content on a media content platform from a user device associated with a user; in response to receiving the request, determining whether the user-generated content should be reviewed by one or more screening users based on one or more avoidance criteria; and in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, adding a screening request to a queue of screening users, the screening request including the user-generated content and information associated with a user account corresponding to the user; and, in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, adding a screening request to a queue of screening users, the screening request including the user-generated content and information associated with a user account corresponding to the user. and determining whether a judgment responsive to the review request indicates that the user-generated content item should be provided on the platform, wherein in response to determining that the judgment indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is sent to a user of the user device indicating the at least one violated policy and a penalty to be imposed on a user account corresponding to the user, and (ii) corresponding action is taken to prevent the user-generated content from being provided on the media content platform, and in response to determining that the judgment indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is sent to the user of the user device indicating that the user-generated content is eligible for provision on the media content platform.

[0027] In accordance with some embodiments of the disclosed subject matter, a system for identifying inappropriate content items is provided, the system including: means for receiving a request to provide user-generated content on a media content platform from a user device associated with a user; means for determining, in response to receiving the request, whether the user-generated content should be reviewed by one or more screening users based on one or more avoidance criteria; means for adding a screening request to a screening user's queue in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, the screening request including the user-generated content and information associated with a user account corresponding to the user; and means for determining that the user-generated content should be provided on the media content platform by a screening user. and means for determining whether a judgment responsive to the request indicates that in response to determining that the judgment indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is sent to a user of the user device indicating the at least one violated policy and a penalty to be imposed on a user account corresponding to the user, and (ii) corresponding action is taken to prevent the user-generated content from being provided on the media content platform, and in response to determining that the judgment indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is sent to the user of the user device indicating that the user-generated content is eligible for provision on the media content platform.

[0028] Various objects, features, and advantages of the disclosed subject matter may be more fully understood by reference to the following detailed description of the disclosed subject matter when considered in conjunction with the following drawings, in which like reference numerals identify like elements and in which: [Brief description of the drawings]

[0029] [Figure 1] 1 shows an illustrative example of an information flow diagram for identifying inappropriate content items in accordance with some embodiments of the disclosed subject matter. [Diagram 2] 1 shows an illustrative example of a process for identifying inappropriate content items in accordance with some embodiments of the disclosed subject matter. [Diagram 3] FIG. 1 is a schematic diagram of an illustrative system suitable for implementation of mechanisms described herein for identifying inappropriate content items, in accordance with some embodiments of the disclosed subject matter. [Figure 4] A diagram showing a detailed example of hardware that may be used in the server and / or user device of FIG. 3 in accordance with some embodiments of the disclosed subject matter. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] According to various embodiments, mechanisms (which may include methods, systems, and media) are provided for identifying inappropriate content items. More specifically, the disclosed subject matter relates to determining whether user-generated content should be manually reviewed by one or more reviewers or whether the user-generated content can be bypassed from manual review based on one or more criteria, and determining whether the user-generated content is inappropriate content so that corresponding action can be taken on the user-generated content.

[0031] Previous inappropriate content detection systems typically allow user-generated content to be published or otherwise made available until such content has been reviewed, which poses significant risks. The mechanisms disclosed herein can provide a multi-stage inappropriateness mitigation approach that ensures that user-generated content that does not meet one or more avoidance criteria is manually reviewed, while preventing user-generated content from being made available on a media content platform, for example, without requiring human review of each piece of content.

[0032] In some embodiments, the mechanisms described herein can receive a content item from a content item creator that is a user-generated content item. A user-generated content item may include any suitable type of user-generated content, such as, for example, a video or audio content item generated by a content creator, a comment entered by a viewer of media content in connection with an online chat or comment section, use of a content feature to generate user-generated content, and / or any other suitable type of user-generated content. It should be noted that user-generated content may include any suitable content, such as video content, audio content, text content, live streamed video content, etc.

[0033] In some embodiments, the facility may determine whether the content item should be manually reviewed by one or more reviewers. In some embodiments, in response to determining that the user-generated content should be manually reviewed, the facility may add the user-generated content to a queue of content items to be reviewed by a reviewer (e.g., a reviewing user at an appropriate computing device). In some embodiments, the reviewer may then receive the user-generated content from the queue of content items and review the user-generated content to determine whether the user-generated content violates a policy of the media content platform. For example, in some embodiments, the reviewer may determine whether the user-generated content includes any particular type of objectionable content (e.g., violent content, offensive language, hate speech, adult content, and / or any other suitable type of objectionable content). In some embodiments, the reviewer may then transmit a decision indicating whether the user-generated content violates any policy of the media content platform. In some embodiments, the facility may then determine whether the content item should be posted to the media content platform in association with the feature.

[0034] Note that in some embodiments, in response to receiving a determination from a first reviewer indicating that the user-generated content violates a policy of the media content platform, the facility may add the user-generated content to another queue of content items to be reviewed by a different reviewer (e.g., a validating reviewer, a reviewer with more experience than the first reviewer, etc.) Continuing with this example, in response to receiving determinations from the first reviewer and a second, different reviewer that both indicate that the user-generated content violates a policy of the media content platform, a corresponding action may be taken, such as removing the user-generated content from the media content platform or otherwise preventing the user-generated content from being made available on the media content platform.

[0035] In some embodiments, the mechanism can determine whether a content item should be manually reviewed based on any appropriate information. For example, in some embodiments, the mechanism can determine whether a content item should be manually reviewed based on information associated with the content creator, such as whether the content creator has previously uploaded an approved content item, whether the content creator has previously uploaded a content item that violates the media content platform's policies, and / or any other appropriate information associated with the content creator. As another example, in some embodiments, the mechanism can determine whether a content item should be manually reviewed based on the similarity of the content item to previously reviewed content items. As a more specific example, in some embodiments, the mechanism can determine that a content item is similar to a previously rejected content item and determine that the uploaded content item should also be rejected. In some embodiments, the mechanism can determine the criteria used to determine whether a content item should be added to a queue of items to be manually reviewed based on the number of content items received, thereby allowing the amount of manual review allocated to be adjusted.

[0036] In some embodiments, in instances where a review of the content item indicates that the content item violates a policy of the media content platform, the mechanism may cause a message to be sent to the content creator indicating any appropriate information. For example, in some embodiments, the message may indicate the policy that was violated, timestamp information indicating the portion of the content item that violates the policy, the penalty for the policy violation, and / or any other appropriate information. In some embodiments, the mechanism may determine the penalty based on any appropriate information, such as the number of previous violations by the content creator, the severity of the violation, and / or any other appropriate information. Additionally, in some embodiments, the mechanism may cause the content item to be blocked and / or removed from the media content platform.

[0037] In some embodiments, in response to determining that the user-generated content or use of the functionality to generate user-generated content is acceptable (e.g., in response to a reviewer determining that the content does not violate the media content platform's policies, in response to a determination that the content is likely not to violate the media content platform's policies, etc.), the facility may transmit a specific time that the user-generated content will be posted or otherwise made available on the media content platform (e.g., rather than publishing or posting the user-generated content prior to any review for violations of one or more policies of the media content platform, rather than there being a delay after publishing or posting the content during which review of the content is submitted, etc.). Alternatively, the facility may transmit a specific time that the user-generated content may be posted or published at the direction of the content creator.

[0038] In some embodiments, the facility may receive an appeal of the decision from the content creator. In some such embodiments, the facility may add the appeal to a queue of appeals, and the appeal may be assigned to a reviewer. In some embodiments, any suitable information may be provided to the reviewer of the appeal, such as the content item, information associated with the original decision being appealed (e.g., information indicating the policy that was violated, timing information indicating the portion of the content item that violated the policy, and / or any other suitable information), text from the content creator (e.g., the content creator's assertion, and / or any other suitable text), and / or any other suitable information. In some embodiments, a decision on the appeal may then uphold the original decision, or conversely, the original decision may be reversed. In cases where the original decision is reversed and the content item was previously blocked, the facility may make the content item available on the media content platform.

[0039] In some embodiments, the content item may be received from a content creator, where the content item is to be provided in association with the content creator's use of the functionality. For example, in some embodiments, the content item may be a media content item received by the media content platform for provision to a viewer in association with a particular functionality. As a more specific example, in some embodiments, the functionality may include providing membership services to viewers of a channel of content associated with the media content platform, where the membership may enable members to access any suitable benefits associated with the membership (e.g., access to special content, access to chat rooms or forums with other members, access to custom emojis or badges, and / or any other suitable benefits). As another more specific example, in some embodiments, the functionality may include enabling content creators to offer merchandise (e.g., branded clothing, and / or any other suitable branded merchandise) associated with the content item and / or associated with the channel of content to which the content item belongs. Additional details regarding the functionality are described in more detail below in connection with FIG. 2. In some such embodiments, the mechanism may determine whether a content item should be manually reviewed to determine whether the content item is eligible to be provided to a user in association with use of the feature. For example, in some embodiments, as described in more detail below in connection with FIG. 2, the mechanism may determine whether a content creator is eligible to use a feature, determine whether a content item should be manually reviewed, determine whether a content item should be blocked from being provided in association with the feature, etc.

[0040] In a more specific example, prior to making user-generated content available on the media content platform, the facility may use human and machine reviewers to determine whether a user account is eligible to access the media content platform functionality to generate user-generated content, which may, for example, limit access to the media content platform functionality to user accounts that have been evaluated as clean or non-inappropriate user accounts.

[0041] In some embodiments, a content item as referred to herein may refer to any suitable type of user-generated content, such as media generated and uploaded by a content creator, membership benefit offerings provided by a content creator in exchange for purchasing a membership associated with a channel of content produced by the content creator, comments entered by viewers of content in a comment section or social networking post, and / or any other suitable type of content. That is, in some embodiments, the mechanisms described herein may be used to determine whether any suitable type of user-generated content should be manually reviewed prior to posting the user-generated content to a particular page, forum, comment section, etc. For example, the mechanisms described herein may be used to review comments (e.g., comments submitted to a comment section associated with a content item, comments submitted to a chat forum, comments submitted to a social networking post, and / or any other suitable comments). As a more specific example, in some embodiments, the mechanisms described herein may be used to determine whether a particular comment should be added to a queue of comments to be manually reviewed or to determine whether a comment should be deleted. As another example, in some embodiments, the mechanisms described herein may be used to review benefits offered by a content creator in association with membership to a channel of content offered by the content creator. As a more specific example, in some embodiments, the mechanisms described herein may be used to determine whether a particular benefit offered by a content creator should be added to a queue of benefits to be manually reviewed and / or to determine whether a particular benefit should be revoked (e.g., for violating any particular policy associated with the media content platform).

[0042] 1, an illustrative example 100 of an information flow diagram for identifying inappropriate content items is shown, in accordance with some embodiments of the disclosed subject matter. As shown, in some embodiments, blocks of process 100 may be performed on a content creator device 308, a server 302, and / or a reviewer device 310, as shown and described below in connection with FIG.

[0043] At 102, the content creator device 308 can transmit a user-generated content item to the server 302. In some embodiments, the user-generated content item can include any suitable type of content, such as a video, a playlist of videos, a link to a live stream of videos, an image, an animation, a photo, a photo slideshow, a document, and / or any other suitable type of content. In a more specific example, the content creator device 308 can upload the user-generated content item for publication on a channel associated with a user account of a video sharing service. As another example, in some embodiments, the user-generated content item can include a comment submitted by a user of the content creator device 308 to be posted in a chat room, in a live chat, in a comment section associated with a particular page, media content item, or social networking post, and / or any other suitable type of comment. In some embodiments, the content creator device 308 can transmit the user-generated content item in association with a user account associated with the content creator device 308 (e.g., a user account on a media content platform associated with the server 302, and / or any other suitable user account). In some embodiments, the server 302 may be associated with a media content platform that hosts uploaded content and streams or otherwise provides content hosted by the media content platform to one or more user devices.

[0044] It should be noted that in some embodiments, in response to receiving the user-generated content item, the server 302 can determine whether the user-generated content item should be reviewed by one or more review users. The server 302 can then cause a notice or other suitable interface to be presented on the content creator device 308 indicating that the user-generated content item or access to the functionality for generating user-generated content is being reviewed. For example, the notice can indicate a particular amount of time until a decision from the reviewer is received. In another example, the notice can indicate that an updated notice will be presented upon approval of the user-generated content, thereby giving the content creator at the content creator device 308 control over publishing the user-generated content on the media content platform or otherwise making the user-generated content available on the media content platform.

[0045] At 104, the server 302 may add the content item to a queue of content items to be reviewed. In some embodiments, the queue of content items may correspond to a queue of content items to be manually reviewed by a human reviewer and / or to be reviewed in any other suitable manner. In some embodiments, the server 302 may maintain the queue in any suitable manner. For example, in some embodiments, the content item may be added to the queue based on the time the content item is received by the server 302 (e.g., such that items received later are placed later in the queue, and / or in any other manner). As another example, in some embodiments, the content item may be added to the queue based on information associated with a content creator associated with the content creator device 308. As a more specific example, in some embodiments, server 302 may determine the popularity of a content creator (e.g., based on the number of subscribers to a channel of content associated with the content creator, based on the number of views of content generated by the content creator, and / or based on any other suitable information) and may place content items in a queue based on the popularity of the content creator (e.g., content items received from relatively popular content creators may be inserted relatively higher in the queue, and / or in any other suitable manner). In some embodiments, server 302 may maintain a queue of content items based on any suitable combination of factors.

[0046] It should be noted that in some embodiments, the server 302 may determine whether the content item should be added to a queue, as described in more detail below in connection with block 208 of FIG. 2. In some embodiments, the server 302 may add the content item to a queue of content items to be reviewed in response to determining that the content item should be reviewed. Alternatively, in some embodiments, the server 302 may determine that the user-generated content item meets one or more avoidance criteria, such that the user-generated content item will not be reviewed and will not be added to the queue of content items to be reviewed at 104. In some such cases, the server 302 may use any suitable technique or combination of techniques to determine whether the user-generated content item should be posted or removed (or otherwise prevented from being made available on the media content platform), as described in more detail below in connection with blocks 208 and 210 of FIG. 2. For example, in some embodiments, the server 302 can determine that a user-generated content item is substantially similar (e.g., a threshold magnitude of similarity) to previously approved user-generated content and, in response, can allow the user-generated content to be made available on the media content platform. In another example, in some embodiments, the server 302 can determine that a user-generated content is substantially similar (e.g., a threshold magnitude of similarity) to previously removed user-generated content and, in response, can automatically flag the user-generated content, remove the user-generated content, or otherwise prevent the user-generated content from being offered on the media content platform without sending the user-generated content to one or more reviewers.

[0047] In some embodiments, server 302 may receive an indication, such as a flag from any suitable user of the media content platform, that the user-generated content may violate the policies of the media content platform (e.g., by including objectionable content within the user-generated content). In response to receiving such an indication, server 302 may add the user-generated content to a queue of content items to be reviewed by one or more reviewers (e.g., reviewing users at suitable computing devices). This may, for example, ensure that the user-generated content is reviewed by one or more reviewers.

[0048] At 106, the examiner device 310 may receive an indication of the content item from the queue. In some embodiments, the examiner device 310 may receive the indication in any suitable manner. For example, in some embodiments, the examiner device 310 may receive this indication via a user interface through which a user of the examiner device 310 views the content item and transmits a judgment regarding the content item (e.g., a judgment indicating that the content item violates a particular policy associated with the media content platform, a judgment indicating that the content item does not violate any policy associated with the media content platform, and / or any other suitable judgment). As a more specific example, in some embodiments, the examiner device 310 may receive a notification indicating that the content item has reached the top of a queue of items to be reviewed and / or that the content item has been assigned to a user of the examiner device 310 for review. In some such embodiments, the notification may include any suitable information, such as a link to the content item, the name of the content creator associated with the content creator device 308, a topic associated with the content item, information associated with the content creator (e.g., the number of previously uploaded and / or approved content items created by the content creator, the number of times content previously uploaded by the content creator has been associated with policy violations, and / or any other suitable information), and / or any other suitable information.

[0049] In some embodiments, a user of the examiner device 310 may select a content item based on the instructions in any suitable manner (e.g., by selecting a link presented in the instructions and / or in any other suitable manner), which may cause the content item to be presented in a video player window or any other suitable user interface on the examiner device 310.

[0050] At 108, the reviewer device 310 may transmit a decision regarding the content item. For example, in some embodiments, the decision may indicate any suitable information regarding the content item, such as whether the content item includes content that violates one or more policies associated with the media content platform to which the content item was uploaded. As a more specific example, in some embodiments, the policies may include any suitable policy, such as content items that include violent content, sexual content, inappropriate language, hate speech, and / or any other suitable type of objectionable content.

[0051] In some embodiments, the transmitted decision may include any suitable information, such as an indication of the type of objectionable content contained in the content item, an indication of one or more portions of the content item that contain the objectionable content (e.g., based on a timestamp of the video, based on a frame of the video, and / or in any other suitable manner), and / or any other suitable information. Note that in some embodiments, the decision may indicate that the content item does not violate any policies.

[0052] At 110, the server 302 can update an account associated with the content creator device based on the determination received from the reviewer device 310. For example, in the case where the determination indicates that the content item violates a particular policy, the server 302 can update a record indicating the number of times the content creator has uploaded content that violates a policy associated with the media content platform. As another example, in the case where the determination indicates that the content item does not violate a particular policy, the server 302 can update a record indicating the number of times the content creator has uploaded content that does not violate any policy associated with the media content platform (i.e., the number of times the content creator has uploaded content that has been approved by the reviewer).

[0053] In some embodiments, the determination may be viewed by a user of the content creator device 308. For example, in some embodiments, a user of the content creator device 308 may log into a user account associated with the content creator device 308 and the media content platform and view the determination in a user interface associated with the user account. In another example, in response to a determination indicating that the user-generated content or use of a feature to generate user-generated content is acceptable (e.g., in response to a reviewer determining that the content does not violate the policies of the media content platform, in response to a determination that the content is likely not to violate the policies of the media content platform, etc.), the determination may be presented on the content creator device 308 along with a specific time that the user-generated content will be posted or otherwise made available on the media content platform (e.g., rather than publishing or posting the user-generated content prior to any review for violations of one or more policies of the media content platform, rather than with a delay following the publication or posting of the content during which review of the content is submitted, etc.). Alternatively, in some embodiments, the determination may be presented along with a specific time that the user-generated content may be posted or published at the direction of the content creator.

[0054] It should be noted that in some embodiments, in response to a reviewer providing a decision that the user-generated content violates the media content platform's policies, the server 302 may add the content to a queue of content items to be reviewed by another reviewer. It should also be noted that in some embodiments, the additional reviewer may be a different reviewer who is more experienced compared to the reviewer who provided the initial decision. In such an embodiment, the server 302 may remove the user-generated content or otherwise prevent the user-generated content from being provided by the media content platform in response to receiving two negative decisions from two different reviewers indicating that the user-generated content violates the media content platform's policies.

[0055] In some embodiments, the decision may be appealed by a user of the content creator device 308. For example, in some embodiments, the decision may be appealed in cases where the user of the content creator device 308 disagrees with the decision (e.g., in cases where the decision indicates that a content item contains a particular type of objectionable content and / or violates a particular policy and the user disagrees, and / or in any other suitable cases). In some embodiments, blocks 112 through 118 shown in FIG. 1 and described below may be performed in cases where the decision is appealed. In some embodiments, blocks 112 through 118 may be omitted in cases where the decision is not appealed.

[0056] At 112, the content creator device 308 may transmit an indication that the decision is being appealed. In some embodiments, the content creator device 308 may transmit the indication in any suitable manner. For example, in some embodiments, a user of the content creator device 308 may select any suitable button or user interface control associated with the decision that, when selected, causes a user interface for entering any suitable text associated with the appeal of the decision (e.g., any suitable reasons for contesting the decision and / or other suitable text). In some such embodiments, the text associated with the appeal of the decision may be transmitted to the server 302 in association with the indication that the decision is being appealed.

[0057] At 114, the server 302 may add the content item to the appeals queue. In some embodiments, the content item may be added to the appeals queue in any suitable manner. For example, in some embodiments, the content item may be added to the appeals queue based on the time that the indication that the decision is being appealed was sent by the content creator device 308 at 112. As another example, in some embodiments, the content item may be added to the appeals queue based on the relative popularity of the content creator associated with the content item, as described above in connection with block 104.

[0058] At 116, the reviewer device 310 may receive an indication of the content item from the appeals queue. In some embodiments, the reviewer device 110 may receive the indication of the content item in any suitable manner. For example, in some embodiments, the reviewer device 110 may present a notice indicating that a decision associated with the content item has been appealed by a content creator associated with the content item.

[0059] In some embodiments, a user of the reviewer device 310 may select this indication in any suitable manner, and in some embodiments, selection of this indication may cause the content item to be presented on the reviewer device 310 in association with any suitable information associated with the decision and / or any suitable information associated with the appeal. For example, in some embodiments, this information may include text associated with the appeal (e.g., as described above in connection with block 110), information associated with the decision being appealed (e.g., information indicating the objectionable content or policy violations indicated in the decision, information indicating a timestamp at which the particular objectionable content as presented in the decision was indicated, and / or any other suitable information), and / or any other suitable information.

[0060] It should be noted that while FIG. 1 shows the same device (i.e., the reviewer device 310) as both reviewing the content item at 106 and reviewing the appeal of the content item at 116, in some embodiments, two different reviewers may be used. That is, in some embodiments, a different reviewer than the reviewer that submitted the decision being appealed may review the appeal of the decision. For example, a different reviewer on a different reviewer device may be used to review the user-generated content in the appeal queue, the different reviewer designated as more experienced than the original reviewer who provided a negative decision, and in response to both the original reviewer and the different appeal reviewer providing a negative decision, action may be taken, such as suspending a user account or preventing the presentation of the user-generated content.

[0061] At 118, the reviewer device 310 may transmit a decision regarding the appeal of the content item. In some embodiments, the decision may indicate any suitable information, such as that the user of the reviewer device 310 agrees with the decision being appealed, that the user of the reviewer device 310 does not agree with the decision being appealed, and / or any other suitable information.

[0062] At 120, the server 302 may update an account associated with the content creator device based on the appeal decision. For example, in cases where the appealed decision indicates that the content item violates a particular policy and the appeal decision disagrees with the appealed decision, the server 302 may update the account to reflect the discrepancy between the two decisions. As a more specific example, in some embodiments, the server 302 may decrement a record of the number of times the content creator uploaded a content item that violates a policy associated with the media content platform. As another example, in cases where the appeal decision confirms the appealed decision that the content item violates a particular policy, the server 302 may take any appropriate action, such as blocking the content item, prohibiting the content creator device 308 from uploading the content item for a predetermined length of time (e.g., one month, 90 days, and / or any other suitable length of time), and / or any other appropriate action. Further details regarding actions that may be performed by the server 302 are described below in connection with block 224 of FIG. 2.

[0063] 2, an illustrative example 200 of a process for identifying inappropriate content is shown in accordance with some embodiments of the disclosed subject matter. In some embodiments, the blocks of process 200 may be performed on a server, such as a server associated with a media content platform that hosts content items uploaded by one or more content creators.

[0064] Process 200 may begin at 202 by receiving from a content creator a content item to be uploaded or posted in association with a media content platform. For example, in some embodiments, the content item may be a video or audio content item produced by the content creator to be uploaded to the media content platform and made available for viewing by other users of the media content platform. As another example, in some embodiments, the content item may be a comment submitted by a viewer of a content item hosted by the media content platform. As a more specific example, in some embodiments, the comment may be a comment submitted by a viewer in association with a chat room or live chat associated with a particular channel of content, a comment submitted by a viewer to be posted in a particular comment section (e.g., a comment section associated with a particular media content item hosted by the media content platform and / or any other suitable comment section), and / or any other suitable comment. As yet another example, in some embodiments, the content item may be a benefit being offered by the content creator via the media content platform in association with a channel of content produced by the content creator. As a more specific example, a reward may be an offering of a particular type of content (e.g., exclusive video content, unreleased content, clothing or other merchandise, and / or any other suitable type of content) provided by a content creator in exchange for a user purchasing a membership or subscription to a channel of that content.

[0065] In some embodiments, the uploaded content items may be uploaded in association with a particular feature of the media content platform, e.g., a feature that the content creator is interested in using to generate user-generated content. In some embodiments, the feature may include functionality associated with comments or messages that may be submitted by reviewers of the content items. For example, in some embodiments, the feature may relate to comments in a comment section associated with the content item and / or comments posted in a live chat associated with the content item. As a more specific example, the feature may include enabling reviewers of the content items to purchase "Super Chats," which may include enabling viewers to pin comments associated with the purchased "Super Chats" to a particular location in the chat or comment section (e.g., to the top of a live chat, to the top of a comment section, and / or to any other suitable location). Continuing with this example, in some embodiments, the feature may include determining the length of time for which the "Super Chat" comment should be pinned to a particular location based on the amount of payment made by the reviewer who purchased the "Super Chat." As another more specific example, in some embodiments the functionality may include enabling viewers of a content item to purchase "Super Stickers" that enable viewers of the content item to include an animated image or icon in a comment section or live chat associated with the content item. Continuing with this example, in some embodiments the functionality may include pinning a particular "Super Sticker" image or icon at a particular location in the live chat or comment section in response to determining that any particular criteria have been met (e.g., the viewer has paid more than a predetermined amount for a "Super Sticker" and / or any other suitable criteria).It should be noted that in some embodiments, the animated images or icons selectable by a viewer of a content item may be organized in any suitable manner and generated and made available, for example, by the content creator, by the media content platform, and / or by any other suitable entity. As yet another more specific example, in some embodiments, the functionality may include enabling a viewer of a content item to purchase "super thanks." In some such embodiments, the viewer's purchase of a "super thanks" may result in a selected animated image or icon (e.g., selected by the viewer of the content item in connection with the purchase of the "super thanks") being presented in association with the content item and / or may result in any suitable percentage of the viewer's payment being shared with the content creator.

[0066] It should be noted that in some embodiments, the functionality may enable the content creator to receive any suitable percentage of the revenue from purchased "Super Chats," "Super Stickers," and / or "Super Thanks." For example, in some embodiments, the content creator may receive a percentage of the revenue from the purchased options, and the media content platform may receive the remainder of the revenue. Additionally or alternatively, in some embodiments, the functionality may enable the content creator to designate any suitable charity as receiving a percentage of the revenue.

[0067] In some embodiments, the functionality may enable content creators to offer membership services to viewers of content produced by the content creator. For example, in some embodiments, the membership services may enable viewers who purchase a membership to receive any suitable benefits in association with the content produced by the content creator. In some embodiments, the benefits may include any suitable special offers available to viewers who purchase a membership, such as access to special or exclusive content produced by the content creator (e.g., available only to viewers who purchase a membership), access to exclusive emojis or other animated icons that may be included in chat messages or comment sections associated with the content produced by the content creator, access to exclusive chat forums where viewers with purchased memberships can chat with other viewers who have similarly purchased memberships, and / or any other suitable special offers available to members.

[0068] In some embodiments, the functionality may enable a content creator to offer merchandise through the media content platform. For example, in some embodiments, the functionality may enable a content creator to include links to merchandise (e.g., clothing, mugs or cups, artwork, and / or any other suitable merchandise) on a page that presents a content item or other content provided by the content creator through the media content platform. As a more specific example, the functionality may enable a content creator to include links to one or more merchandise available for purchase on a video viewing page that presents a content item. As another more specific example, the functionality may enable a content creator to include links to one or more merchandise available for purchase on a page associated with a channel of content associated with the content creator.

[0069] In some embodiments, the functionality may enable a content creator to provide one or more links for purchasing tickets to a live show associated with the content creator. For example, in instances where the content item is a music video associated with a particular band or other musical group, the functionality may enable a link for purchasing tickets to a concert associated with the band or musical group to be presented on the page on which the content item is presented (e.g., below a video player window on which the content item is presented, and / or in any other suitable location). Note that in some embodiments, the functionality may enable a link for purchasing tickets to be presented in association with any other suitable page or content, such as a page associated with a channel of content produced by the content creator, and / or any other suitable content.

[0070] At 204, process 200 may determine whether the content creator is eligible to use the functionality for generating user-generated content. In some embodiments, process 200 may determine whether the content creator is eligible to use the functionality based on any suitable information and using any suitable technique. For example, in some embodiments, process 200 may determine whether the content creator is eligible to use the functionality based on content items that the content creator previously uploaded to the media content platform. As a more specific example, in some embodiments, process 200 may use any suitable classifier or machine learning technique to determine a risk score indicative of the probability that the content creator will upload video content that includes one or more types of objectionable content based on the previously uploaded content items. As one particular example, process 200 may determine that a content creator whose risk score is below a predetermined threshold is not eligible to use the functionality. As another particular example, in some embodiments, process 200 may determine that a content creator whose risk score is above a predetermined threshold is eligible to use the functionality. As another more specific example, in some embodiments, process 200 may determine the number of times a content creator has previously violated one or more policies associated with the media content platform (e.g., policies prohibiting a particular type of objectionable content, and / or any other suitable policies). As one particular example, in some embodiments, process 200 may determine that a content creator who has violated one or more policies of the media content platform more than a predetermined threshold number of times (e.g., more than once, more than twice, more than once in the past month, and / or any other suitable number of times) is ineligible to use the functionality.

[0071] As another example, in some embodiments, process 200 may determine whether a content creator is eligible to use the feature based on information associated with the content creator, such as the size of the content creator's audience and / or a location associated with the content creator. As a more specific example, in some embodiments, process 200 may determine that a content creator with a relatively small audience is not eligible to use the feature. As another more specific example, in some embodiments, process 200 may determine that a content creator with a relatively large audience is eligible to use the feature. In some embodiments, the audience size may be determined based on any suitable information, such as the total number of views of content items previously uploaded by the content creator, the average number of views of content items previously uploaded by the content creator, the number of subscribers to a channel of content produced by the content creator, and / or any other suitable measure.

[0072] As yet another example, in some embodiments, process 200 may determine whether a content creator is eligible to use a feature based on a manual review of content posted by the content creator. As a more specific example, in some embodiments, a human reviewer may review one or more content items previously posted by the content creator and / or content items uploaded by the content creator, as described above in connection with block 202, and may determine whether the content items violate one or more policies of the media content platform. In some embodiments, in response to determining that the content item violates one or more policies of the media content platform, process 200 may determine that the content creator is not eligible to use the feature.

[0073] As yet another example, in some embodiments, process 200 can determine whether a content creator is eligible to use a feature based on any other suitable information associated with the content creator and / or associated with a content item associated with the content creator. As a more specific example, in some embodiments, process 200 can determine whether a content creator has previously been approved to participate in a program associated with a media content platform that allows the content creator to use a particular feature associated with membership in the program. As another more specific example, in some embodiments, process 200 can determine whether a content creator's age is above a predetermined age (e.g., above 18 years of age and / or above any other suitable age). As yet another more specific example, process 200 can determine whether a content item associated with the content creator is associated with a particular genre (e.g., age-restricted content, content targeted to children, music video content, and / or any other suitable genre of content). As one particular example, in some embodiments, process 200 may determine that a content creator is not eligible to use the feature if the content items associated with the content creator typically belong to a particular genre, such as age-restricted content, content targeted to children, music video content, etc.

[0074] As yet another example, in some embodiments, process 200 may determine whether a content creator is eligible to use a feature based on determining whether viewers of content created by the content creator are satisfied or dissatisfied with the content creator's use of the feature. As a more specific example, in some embodiments, in cases where a feature includes providing benefits to viewers who purchase a membership associated with content created by the content creator, process 200 may determine whether users who purchased a membership are generally satisfied or dissatisfied with the purchase. As another more specific example, in some embodiments, in cases where a feature includes enabling viewers to purchase pinned comments, pinned animated images, or other messages or chat content, process 200 may determine whether users who purchased pinned messages or chat content are generally satisfied or dissatisfied with the purchase. In some embodiments, process 200 may determine the viewer satisfaction measure in any suitable manner. For example, in some embodiments, process 200 may determine the number or percentage of viewers who cancelled memberships, the number or percentage of viewers who left negative reviews in response to an offered reward, the number or percentage of viewers who unsubscribed from a channel of content produced by the content creator, and / or any other suitable satisfaction measure. In some embodiments, process 200 may determine that the content creator is ineligible to use the feature in response to determining that the calculated satisfaction measure is below a predetermined threshold. As another example, in some embodiments, process 200 may determine that the content creator is ineligible to use the feature in response to determining that a predetermined percentage of viewers (e.g., greater than 10%, greater than 20%, and / or any other suitable percentage) cancelled a purchased membership or left a negative review in connection with an offered reward or purchase option.

[0075] It should be noted that in cases where the content item corresponds to a submitted comment, process 200 can determine whether the user submitting the comment is eligible to post the comment in a particular chat or in a particular comment section. For example, in cases where the comment is submitted in connection with a chat or comment section that requires subscription to a particular channel of the content of the media content platform, or any other suitable criteria to be met, process 200 can determine whether the user submitting the comment is a subscriber to the channel of the content, or meets any other suitable criteria. Additionally or alternatively, in some embodiments, in cases where the content item corresponds to a submitted comment, block 204 can be omitted.

[0076] Additionally, in some embodiments, process 200 may determine whether a content creator is eligible to use a feature based on any suitable combination of information. For example, in some embodiments, process 200 may determine whether a content creator is eligible to use a feature based on a combination of a risk score associated with the content creator, information indicative of the size of the content creator's audience, information indicative of previous violations of policies by the content creator, and / or manual review of the content item. In some embodiments, process 200 may combine the information in any other suitable manner. For example, in some embodiments, process 200 may assign a score to each item of information and may determine an aggregate score associated with the content creator based on the scores of each item of information. In some embodiments, process 200 may then determine whether a content creator is eligible to use a feature based on a determination of whether the aggregate score associated with the content creator exceeds a predetermined threshold.

[0077] If at 204, process 200 determines that the content creator is not eligible to use the functionality ("no" at 204), process 200 may proceed to 206 and may reject the content item. In some embodiments, process 200 may reject the content item in any suitable manner. For example, in some embodiments, process 200 may block uploading of the content item to a server associated with the media content platform. As another example, in some embodiments, process 200 may send a message to the content creator and / or cause a notification to be presented on a user device associated with the content creator indicating that the content item has been rejected. In some such embodiments, the message and / or notification may include any suitable information, such as the reason the content item was rejected (e.g., a previous violation by the content creator of one or more policies associated with the media content platform and / or any other suitable reason). As yet another example, in some embodiments, in instances where the content item corresponds to a comment to be posted to a chat or comment section, process 200 may prevent the comment from being posted to the chat or comment section.

[0078] It should be noted that in some embodiments, in response to determining that a content creator is not eligible to use the functionality, process 200 may allow the content item to be uploaded but may deny the content creator use of the functionality. For example, in instances where process 200 determines that the content creator has not previously violated any policies associated with the media content platform but that the content creator is not eligible to use the functionality for other reasons (e.g., because the content creator does not have a large enough audience, because the content creator has not previously uploaded more than a predetermined number of content items, and / or for any other suitable reason), process 200 may allow the content item to be uploaded to the media content platform but may block or inhibit the content creator from using the functionality.

[0079] If process 200 determines at 204 that the content creator is eligible to use the functionality ("yes" at 204), process 200 may proceed to 208 to determine whether to selectively avoid review of user-generated content items. In some embodiments, process 200 may determine whether to avoid review of a content item based on any suitable information.

[0080] For example, in some embodiments, process 200 may determine whether review of a content item should be avoided if the content item includes objectionable content based on the relative risk of the content item's publication. In some embodiments, the relative risk of the content item's publication may be determined based on any suitable information, such as the expected size of the content item's audience (e.g., the expected number of people likely to view the content item), whether the content creator will receive revenue from viewers of the content item, the expected revenue that the content creator will receive from viewers of the content item, and / or any other suitable information. It should be noted that in some embodiments, process 200 may determine that the content item should be assigned a relatively high risk score in response to determining that the content item meets any particular criteria. For example, in some embodiments, process 200 may determine that a content item should be assigned a relatively high risk score in response to determining that the content item is being offered in exchange for a payment by a viewer (e.g., in instances where the content item is exclusive content offered to users who pay for the content, where the content item is a perk offered to users who pay for membership to a channel of the content, and / or any other suitable type of content).

[0081] In some embodiments, process 200 can determine the relative risk in any suitable manner. For example, in some embodiments, process 200 can identify multiple risk scores (e.g., a first risk score corresponding to an estimated audience size, a second risk score corresponding to the content creator's expected revenue from viewers of the content item, and / or any other suitable risk score) and can combine the multiple risk scores in any suitable manner (e.g., as a weighted average and / or in any other suitable manner). As a more specific example, in the case where the content item corresponds to a video content item to be posted to a content channel that has a relatively large expected audience (e.g., based on the channel having more than a predetermined number of subscribers, based on the content associated with the channel having more than a predetermined number of previous views, and / or based on any other suitable measure), process 200 can determine that the relative risk of publishing the content item is high. As another more specific example, in cases where the content item corresponds to a video content item that has been monetized in any suitable manner (e.g., by the insertion of advertisements, by the content item being provided to users who have paid to receive the content item, and / or in any other suitable manner), process 200 may determine that the relative risk of publication of the content item is high. As yet another more specific example, in cases where the content item corresponds to a comment that will be posted in a comments section, process 200 may determine that the relative risk of publication of the content item is low. In some embodiments, process 200 may determine that content items associated with a relative risk of publication above a predetermined threshold should be manually reviewed (i.e., not to be avoided for manual review, or "no" at 208).It should be noted that in some embodiments, in instances where process 200 determines that a content item is associated with a relative risk of disclosure that exceeds a certain predetermined threshold, process 200 can determine that the content item should be automatically rejected.

[0082] As another example, in some embodiments, process 200 may determine that review of a content item should be avoided in response to determining that the content item is similar to or matches a previously uploaded content item that was previously reviewed. As a more specific example, in some embodiments, process 200 may determine that the content item is similar to or matches a content item that was previously reviewed and approved. As another more specific example, in some embodiments, process 200 may determine that the content item is similar to or matches a content item that was previously reviewed and rejected.

[0083] It should be noted that in some embodiments, process 200 may use any suitable technique or combination of techniques to determine whether an uploaded content item is similar to or matches a previously reviewed content item. For example, in some embodiments, process 200 may compare one or more audio and / or video fingerprints of the uploaded content item to a database of previously uploaded content items to identify matches with previously uploaded and reviewed content items.

[0084] Additionally, it should be noted that in some embodiments, process 200 may determine a degree of similarity required between an uploaded content item and at least one previously reviewed content item to determine that review should be avoided. For example, in some embodiments, process 200 may determine that an uploaded content item should have at least a predetermined threshold similarity to a previously reviewed content item (e.g., at least 70% similarity, at least 80% similarity, and / or any other suitable threshold) to determine that review should be avoided. In some embodiments, the predetermined threshold of similarity may be determined or set in any suitable manner. For example, in some embodiments, the predetermined threshold of similarity may be determined by process 200 such that, upon application of the predetermined threshold of similarity, a predetermined range of content items are sent for manual review. That is, in some embodiments, process 200 may adjust the predetermined threshold of similarity over time based on the number of content items uploaded to the media content platform, thereby allowing the amount of manual review performed to be adjusted.

[0085] Additionally, it should be noted that in some embodiments, process 200 can determine whether manual review should be avoided based on any suitable combination of the relative risk of publication of the content item and the similarity between the uploaded content item and the previously uploaded content item. For example, in some embodiments, process 200 can calculate a risk score indicative of the relative risk of publication of the content item and a similarity score indicative of the similarity between the content item and the previously uploaded content item, and can determine whether manual review should be avoided based on both the risk score and the similarity score. As a more specific example, in some embodiments, process 200 can determine that manual review should not be avoided ("no" at 208) in response to determining that the risk score exceeds a first predetermined threshold and the similarity score is below a second predetermined threshold (i.e., not sufficiently similar to the previously uploaded content item). As another more specific example, in some embodiments, process 200 may determine that manual review should be avoided (“yes” at 208) in response to determining that the risk score is below a first predetermined threshold and the similarity score exceeds a second predetermined threshold.

[0086] In some embodiments, process 200 may determine that certain types of content items should be manually reviewed, regardless of the risk score. For example, in some embodiments, process 200 may determine that content items offered in exchange for payment by a reviewer of the content item (e.g., exclusive content offered to paying users, and / or any other suitable type of paid content) should always be manually reviewed. As another example, in some embodiments, process 200 may determine that content items generated by content creators who are offering paid content for the first time should be manually reviewed. In some embodiments, paid content items may include video content, audio content, "super stickers," "super comments" (as described above with respect to block 202), and / or any other suitable type of content made available to viewers in exchange for payment.

[0087] If at 208 the process 200 determines that review of the content item should be avoided ("yes" at 208), the process 200 may proceed to 210 and determine whether the content item should be rejected as inappropriate content. In some embodiments, the process 200 may determine whether the content item should be rejected based on whether the uploaded content item was determined at block 208 to be similar to or consistent with a previously rejected content item. For example, in some embodiments the process 200 may determine that the uploaded content item is similar to or consistent with a previously rejected content item that was rejected for including one or more types of objectionable content, and in some such embodiments the process 200 may determine that the content item should be rejected. As another example, in some embodiments the process 200 may determine that the uploaded content item is similar to or consistent with a previously approved content item, and in some such embodiments the process 200 may determine that the content item should not be rejected. As yet another example, in some embodiments, process 200 may use any suitable technique to determine the probability that a content item contains any particular type of objectionable content. As a more specific example, in some embodiments, process 200 may use any suitable classifier or combination of classifiers to determine the probability that a content item contains objectionable content. It is noted that in instances where the content item corresponds to video or audio content, process 200 may use a classifier that takes as input any suitable audio or video fingerprint of the content item.In some embodiments, in cases where the content item corresponds to a submitted comment, process 200 may use a classifier that takes as input any suitable text or links contained in the comment to determine the probability that the text contains objectionable content or the probability that the link is to a site associated with objectionable content. Continuing with this example, in some embodiments, process 200 may determine that the content item should be rejected in response to determining that the determined probability that the content item contains objectionable content exceeds a predetermined threshold (e.g., greater than 70%, greater than 80%, and / or any other suitable threshold).

[0088] If the process 200 determines at 210 that the content item should be rejected ("yes" at 210), the process 200 may reject the content item at 206, as described above.

[0089] If at 210, process 200 determines that the content item should not be rejected ("no" at 210), process 200 may proceed to 212, where the content item may be posted to the media content platform. In some embodiments, process 200 may post the content item to the media content platform in any suitable manner. For example, in some embodiments, process 200 may upload the content item to a particular channel associated with the content creator. As another example, in some embodiments, process 200 may make a link to the content item available to other user devices to stream or download the content item. As another example, in some embodiments, process 200 may send a notification to the content creator indicating that the content item has been approved. In some such embodiments, the content creator may then initiate posting of the content item in any suitable manner, such as by selecting any suitable selectable input that makes the content item available to other viewers. As yet another example, in instances where the content item corresponds to a live stream of content, process 200 may make the live stream available to viewers after any suitable time delay (e.g., after 30 seconds, after 1 minute, and / or after any other suitable time delay. As yet another example, in instances where the content item corresponds to a comment to be posted to a comment section or chat, process 200 may cause the comment to be added to the chat or comment section in any suitable manner (e.g., at the bottom of the chat, at the bottom of the comment section, in a pinned position in the chat or comment section, and / or in any other suitable manner).

[0090] Returning to block 208, if process 200 determines at 208 that review of the content item should not be avoided ("no" at 208), process 200 may proceed to 214, where the content item may be added to a queue of content items to be reviewed. As described above in connection with block 104 of FIG. 1, in some embodiments, process 200 may add the content item to the queue of content items to be reviewed in any suitable manner. For example, in some embodiments, the content item may be added to the queue based on the date or time the content item was received at block 202 (e.g., such that previously received items are placed earlier in the queue, and / or in any other suitable manner). As another example, in some embodiments, the content item may be added to the queue based on the popularity of other content items associated with the content creator. As a more specific example, in some embodiments, content items associated with a content creator who previously uploaded content items that received more than a predetermined number of views, shares, comments, etc. may be placed higher in the queue. As yet another example, in some embodiments, in instances where the release of a content item is time-sensitive (e.g., where the content item is live streamed content, where the content item is live chat comments, and / or any other suitable time-sensitive situation), process 200 may add the content item to a relatively high position in the queue (e.g., to the front of the queue and / or any other suitable position), thereby allowing the content item to be manually reviewed more quickly.

[0091] In response to the content item being moved to the head of a queue of content items to be reviewed, as described above in connection with Figure 1, the content item may be assigned to a reviewer. In some such embodiments, the content item may be presented to a user device associated with the reviewer, as described above in connection with Figure 1. In some embodiments, the reviewer may then use the user device associated with the reviewer to enter any information associated with a judgment regarding the content item, such as whether the content item is determined to include one or more types of objectionable content, frames or timestamps of the content item that include the objectionable content in cases where the content item corresponds to video or audio content, one or more policies that the content item violates, and / or any other suitable information.

[0092] At 216, process 200 may receive a decision regarding the content item. In some embodiments, process 200 may receive the decision in any other suitable manner. For example, in some embodiments, server-executed process 200 may receive a message from a user device associated with a reviewer who reviewed the content item. In some such embodiments, the message may include any suitable information, such as an indication that the content item does not violate any policies associated with the media content platform, an indication that the content item violates one or more policies associated with the media content platform, an indication of one or more types of objectionable content included in the content item, a timestamp of the content item that includes the one or more types of objectionable content, and / or any other suitable information.

[0093] At 218, the process 200 can determine whether the determination indicates a policy violation.

[0094] If the process 200 determines at 218 that the determination does not indicate a policy violation ("no" at 218), the process 200 may post the content item to the media content platform at 212, as described above.

[0095] If at 218, process 200 determines that the determination indicates a policy violation ("yes" at 218), process 200 may proceed to 220 and may send a message to the content creator indicating the policy violation. In some embodiments, process 200 may send the message to the content creator in any suitable manner. For example, in some embodiments, process 200 may send the message to an email address associated with the content creator. As another example, in some embodiments, process 200 may send the message in association with the content creator and a user account associated with the media content platform. As a more specific example, in some embodiments, the message may be a notice presented on a home page of a user account associated with the content creator in association with the media content platform. This may provide the content creator with accurate feedback, for example, about policy violations and other reasons for taking action that may prevent user-generated content from being made available on the media content platform.

[0096] In some embodiments, the message may include any suitable information. For example, in some embodiments, the message may indicate one or more policies that the content item violates. As a more specific example, in some embodiments, the message may indicate that the content item includes one or more types of objectionable content that violate one or more policies of the media content platform. As another example, in some embodiments, the message may indicate one or more portions of the content item that violate a policy of the media content platform using any suitable timing information, such as a timestamp of the content item, a frame of the content item, and / or any other suitable timing information. As yet another example, in some embodiments, the message may indicate a penalty. As a more specific example, in some embodiments, the message may indicate that the content item has been blocked from being uploaded to the media content platform. As another more specific example, in some embodiments, the message may indicate that the content creator is prohibited from uploading content items for a predetermined length of time (e.g., for one month, 90 days, and / or any other suitable length of time). As yet another more specific example, in some embodiments, the message may indicate that an account associated with the content creator is suspended. As yet another more specific example, in some embodiments, the message may indicate that the content creator is not allowed to use the feature for a predetermined length of time (e.g., 30 days, 90 days, and / or any other suitable length of time). As yet another more specific example, in some embodiments, the message may indicate that the content creator is not allowed to stream live content for a predetermined length of time (e.g., 30 days, 90 days, and / or any other suitable length of time).As yet another more specific example, in the instance where the content item corresponds to a comment to be posted to a chat or comment section, the message may indicate that the user will not be allowed to post comments for a particular length of time in the comment section associated with the rejected comment or in any comment section associated with the media content platform.

[0097] It should be noted that in some embodiments, process 200 may determine a penalty based on the policy that the content item violates. For example, in some embodiments, process 200 may determine a penalty based on the severity of the violation. As a more specific example, in some embodiments, process 200 may determine a first penalty (e.g., the content item is blocked) in response to determining that the policy violation corresponds to the inclusion of a first type of objectionable content in the content item, and may determine a second penalty (e.g., a user account associated with the content creator is suspended) in response to determining that the policy violation corresponds to the inclusion of a second type of objectionable content in the content item. Additionally, it should be noted that in some embodiments, process 200 may determine that a particular penalty (e.g., account suspension, prohibition from uploading content for a predetermined amount of time, and / or any other suitable penalty) should be applied in response to determining that the content creator has violated one or more policies more than a predetermined number of times (e.g., more than three violations, more than five violations, and / or any other suitable number of times). This can discourage policy violations, for example, through increasingly stringent enforcement actions. Additionally, it is noted that in some embodiments, process 200 can determine a penalty based on the type of content item. For example, in some embodiments, in cases where the content item corresponds to an upload of video or audio content to be made available to other viewers, process 200 can identify a penalty associated with a temporary or permanent block on uploading other video or audio content to the media content platform.As another example, in some embodiments, in cases where a content item corresponds to a comment submitted to a chat or comment section, process 200 can identify a penalty associated with a temporary or permanent blocking of the submission of the comment.

[0098] It should be noted that in some embodiments, before proceeding to block 220, process 200 may add the content item to a second queue such that the content item is then reviewed a second time by a second reviewer. That is, in some embodiments, in response to a determination that the judgment indicates a policy violation, process 200 may cause the content item to be reviewed a second time by a second reviewer, different from the first reviewer, to confirm that the content item violates the policies of the media content platform.

[0099] At 222, process 200 may update an account associated with the content creator based on the policy violation. For example, in some embodiments, process 200 may update the account to indicate that the uploaded content item violated a particular policy. As another example, in some embodiments, process 200 may update the account to indicate that the uploaded content item was determined to include a particular type of objectionable content. As yet another example, in some embodiments, process 200 may update a record of the number of times the content creator uploaded a content item that violated a policy associated with the media content platform. It should be noted that in some embodiments, as described above in connection with block 220, process 200 may use information associated with the content creator's account to determine a penalty for the content creator.

[0100] At 224, process 200 can block the upload of the content item to the media content platform. Note that in instances where the content item is a live stream of content, process 200 can stop or prevent the live stream from streaming to any user devices.

[0101] It should be noted that, although processes 100 and 200 are described above as relating to screening of content items (e.g., videos, live streamed video content, and / or any other suitable content items), in some embodiments, processes 100 and 200 may be used to screen any other suitable type of content, such as comments in a comment section, comments on a social networking post, messages in a live chat, and / or any other suitable comments. For example, in some embodiments, process 200 may determine whether a particular comment should be added to a queue of comments to be screened (e.g., at blocks 208 and 214) based on a classifier applied to a group of comments. As a more specific example, in some embodiments, process 200 may add a comment to a queue of comments to be manually screened in response to determining, using any suitable machine learning classifier, that the comment is likely to include one or more types of objectionable content (e.g., hate speech, objectionable language, objectionable images or links, and / or any other suitable type of objectionable content). Continuing with this example, in some embodiments, process 200 may add the comment to a queue of comments to be reviewed (e.g., at block 214) and proceed through blocks of process 200 to determine whether the comment should be removed from the live chat or comment section in which it was posted based on manual review. As another example, in some embodiments, as described above in connection with block 202, processes 100 and 200 may be used to review benefits offered by content creators in connection with the purchase of a membership.As a more specific example, in some embodiments, process 200 may determine whether a particular offered reward should be added to a queue of rewards to be reviewed (e.g., at blocks 208 and 214) based on any suitable information, such as whether complaints have been received from viewers who purchased memberships, whether more than a predetermined threshold number of viewers have cancelled their purchased memberships, and / or based on any other suitable criteria. Continuing with the example, process 200 may then proceed through blocks of process 200 to determine whether the reward offer should be cancelled or discontinued based on manual review.

[0102] 3, an illustrative example of hardware for identifying inappropriate content items 300 is shown that may be used in accordance with some embodiments of the disclosed subject matter. As shown, the hardware 300 may include a server 302, a communication network 304, and / or one or more user devices 306, such as a content creator device 308 and / or a reviewer device 310.

[0103] Server 302 may be any suitable server for storing information, data, media content, and / or any other suitable type of content. For example, in some embodiments, server 302 may store user-generated media content uploaded by one or more content creator devices 308. As a more specific example, in some embodiments, the user-generated media content may include videos, movies, photos, slideshows, animations, documents, and / or any other suitable type of user-generated media content. As another example, in some embodiments, server 302 may store information on server 302 about one or more content creators who uploaded user-generated content, such as whether a particular content creator has previously uploaded content that violates a particular policy of the media content platform, and / or any other suitable information. In some embodiments, server 302 may implement any suitable process for queuing uploaded content items for review, storing decisions regarding the uploaded content items (e.g., a decision indicating whether the content item includes objectionable content, and / or any other suitable decision), and / or performing any other suitable function. For example, in some embodiments, server 302 may perform any appropriate blocks of process 200 as shown and described above in connection with FIG.

[0104] The communication network 304 may be any suitable combination of one or more wired and / or wireless networks in some embodiments. For example, the communication network 304 may include any one or more of the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 306 may be connected by one or more communication links (e.g., communication link 312) to the communication network 304, which may be linked to the server 302 via one or more communication links (e.g., communication link 314). The communication link may be any communication link suitable for communicating data between the user device 306 and the server 302, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communication link, or any suitable combination of such links.

[0105] The user device 306 may include any one or more user devices. For example, in some embodiments, the user device 306 may include a content creator device 308. In some embodiments, the content creator device 308 may be any user device suitable for capturing user-generated content, uploading user-generated content, receiving notifications indicating judgments regarding uploaded content items, and / or performing any other suitable functions. As another example, in some embodiments, the user device 306 may include a reviewer device 310. In some embodiments, the reviewer device 310 may be any user device suitable for viewing uploaded content items, submitting judgments regarding uploaded content items, and / or performing any other suitable functions. In some embodiments, the user device 306 may include any suitable type of user device. For example, in some embodiments, the user device 306 may include a mobile phone, a wearable computer, a tablet computer, a desktop computer, a laptop computer, a vehicle entertainment system, a game console, a television, and / or any other suitable user device.

[0106] Although server 302 is shown as one device, in some embodiments the functions performed by server 302 may be performed using any suitable number of devices. For example, in some embodiments, multiple devices may be used to implement the functions performed by server 302.

[0107] To avoid overcomplicating the diagram, two user devices 308 and 310 are shown in FIG. 3, although in some embodiments any suitable number and / or type of user devices may be used.

[0108] In some embodiments, the server 302 and the user device 306 may be implemented using any suitable hardware. For example, in some embodiments, the devices 302 and 306 may be implemented using any suitable general-purpose or special-purpose computer. For example, a mobile phone may be implemented using a special-purpose computer. Any such general-purpose or special-purpose computer may include any suitable hardware. For example, as shown in the example hardware 400 of FIG. 4, such hardware may include a hardware processor 402, memory and / or storage 404, an input device controller 406, an input device 408, a display / audio driver 410, a display and audio output circuit 412, a communication interface 414, an antenna 416, and a bus 418.

[0109] The hardware processor 402, in some embodiments, may include any suitable hardware processor, such as a microprocessor, a microcontroller, a digital signal processor, special purpose logic circuitry, and / or any other suitable circuitry for controlling the functions of a general purpose or special purpose computer. In some embodiments, the hardware processor 402 may be controlled by a server program stored in memory and / or storage of a server, such as the server 302. In some embodiments, the hardware processor 402 may be controlled by a computer program stored in memory and / or storage 404 of the user device 306.

[0110] The memory and / or storage 404, in some embodiments, may be any suitable memory and / or storage for storing programs, data, and / or any other suitable information. For example, the memory and / or storage 404 may include random access memory, read-only memory, flash memory, hard disk storage, optical media, and / or any suitable memory.

[0111] The input device controller 406, in some embodiments, may be any suitable circuitry for controlling and receiving input from one or more input devices 408. For example, the input device controller 406 may be circuitry for receiving input from a touch screen, from a keyboard, from one or more buttons, from a voice recognition circuit, from a microphone, from a camera, from an optical sensor, from an accelerometer, from a temperature sensor, from a near-field sensor, from a pressure sensor, from an encoder, and / or from any other type of input device.

[0112] Display / audio driver 410, in some embodiments, may be any suitable circuitry for controlling and driving output to one or more display / audio output devices 412. For example, display / audio driver 410 may be circuitry for driving a touch screen, a flat panel display, a cathode ray tube display, a projector, one or more speakers, and / or any other suitable display and / or presentation device.

[0113] The communications interface 414 may be any suitable circuitry for interfacing with one or more communications networks (e.g., the computer network 304). For example, the interface 414 may include network interface card circuitry, wireless communications circuitry, and / or any other suitable type of communications network circuitry.

[0114] The antenna 416, in some embodiments, may be any suitable antenna or antennas for wirelessly communicating with a communications network (e.g., the communications network 304). In some embodiments, the antenna 416 may be omitted.

[0115] The bus 418 may be any suitable mechanism for communicating between two or more components 402, 404, 406, 410, and 414 in some embodiments.

[0116] According to some embodiments, any other suitable components may be included in hardware 400.

[0117] In some embodiments, at least some of the above-described blocks of the processes of Figures 1 and 2 may be executed or performed in any order or permutation, including but not limited to the order and permutation shown in and described in connection with the figures. Also, some of the above-described blocks of Figures 1 and 2 may be executed or performed substantially simultaneously, where appropriate, or in parallel to reduce latency and processing time. Additionally, or alternatively, some of the above-described blocks of the processes of Figures 1 and 2 may be omitted.

[0118] In some embodiments, any suitable computer-readable medium may be used to store instructions for performing the functions and / or processes herein. For example, in some embodiments, the computer-readable medium may be transitory or non-transient. For example, a non-transient computer-readable medium may include a medium such as a non-transient form of magnetic medium (such as a hard disk, a floppy disk, and / or any other suitable magnetic medium), a non-transient form of optical medium (such as a compact disk, a digital video disk, a Blu-ray® disk, and / or any other suitable optical medium), a non-transient form of semiconductor medium (such as a flash memory, an electrically programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and / or any other suitable semiconductor medium), any suitable medium that is not ephemeral or does not lose apparent permanence during transmission, and / or any suitable tangible medium. As another example, a transitory computer-readable medium may include signals on a network, in wires, in conductors, in optical fibers, in circuits, in any suitable medium that is ephemeral and loses any apparent permanence during transmission, and / or in any suitable non-tangible medium.

[0119] Accordingly, methods, systems, and mediums are provided for identifying inappropriate content items.

[0120] While the present invention has been described and shown in the foregoing illustrative embodiments, it will be understood that the disclosure is made by way of example only, and that numerous changes in the details of the implementation of the invention may be made without departing from the spirit and scope of the invention, which is limited only by the following claims. The features of the disclosed embodiments may be combined and rearranged in various ways. [Explanation of symbols]

[0121] 100 Processes 200 processes 302 Server 304 Communication Network 306 User Devices 308 Content Creator Devices 310 Examiner Device 312 Communication Links 400 Hardware 402 Hardware Processor 404 Memory and / or Storage 406 Input Device Controller 408 Input Devices 410 Display / Audio Driver 412 Display and audio output circuits, display / audio output devices 414 Communication Interface 416 Antenna 418 Bus

Claims

1. 1. A computer-implemented method for identifying inappropriate content items, comprising: receiving, at a server from a user device associated with a user, a request for providing user-generated content on a media content platform; In response to receiving the request, identifying a second user-generated content similar to the user-generated content, the second user-generated content having been previously reviewed; determining a similarity of the user-generated content to the second user-generated content; determining whether the user-generated content should be reviewed by one or more review users by comparing at least the similarity to a similarity threshold indicated by one or more avoidance criteria; in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, adding a review request to a queue of review users, the review request including the user-generated content and information associated with a user account corresponding to the user; determining whether a determination responsive to the review request indicates that the user generated content should be hosted on the media content platform; in response to determining that the determination indicates that the user generated content violates at least one policy associated with the media content platform; sending a first message to the user of the user device indicating the at least one violated policy and a penalty to be imposed on the user account corresponding to the user; causing a corresponding action to be taken based on said penalty; A method comprising:

2. The method of claim 1, wherein determining whether the user-generated content should be reviewed is further based on determining that the second user-generated content has been deleted from the media content platform.

3. The method of claim 2, wherein determining that the user-generated content should be reviewed based on the one or more avoidance criteria includes determining that the similarity between the user-generated content and the second user-generated content is lower than a similarity indicated by the similarity threshold.

4. in response to determining that the determination indicates that the user-generated content violates at least one policy associated with the media content platform, determining that the user has violated one or more policies more than a predetermined threshold number of times, the penalty being determined further based on objectionable content types contained in other user-generated content previously provided by the user and determined to violate the one or more policies; The method of claim 1, further comprising: determining whether the user-generated content should be reviewed by one or more review users by comparing at least the similarity to a similarity threshold indicated by the one or more avoidance criteria, 2. The method of claim 1, further comprising determining whether the second user-generated content is associated with a particular objectionable content type that is blocked from being provided in association with a particular feature currently enabled for the user account associated with the user.

6. The particular objectionable content type comprises click bait content, said particular feature allows a user to collect revenue; The method of claim 5 , wherein the penalty includes preventing the user account of the user from using the particular feature.

7. The second user-generated content: determining that the second user-generated content was previously uploaded to the media content platform in association with the user account of the user; and determining whether the second user-generated content passed or failed a previous content review; The method of claim 1, wherein the method is determined based on 8. The method of claim 7, further comprising: receiving, at the server, from the user device associated with the user, a subsequent request for providing a third user-generated content on the media content platform; identifying the user-generated content that violates the at least one policy as similar to the third user-generated content; determining that the third user-generated content includes the same objectionable content type as the user-generated content that violated the at least one policy; determining that the third user-generated content should be reviewed based on The method of claim 1, further comprising: determining whether a determination responsive to reviewing the third user-generated content indicates that the third user-generated content does not violate any policy; in response to determining that the third user-generated content contains the same objectionable content type but does not violate any policy; causing the third user-generated content to be made available to a plurality of other users of the media content platform for display on a content page associated with the user account of the user; preventing inclusion of viewer interaction on the content page including the third user-generated content; 9. The method of claim 8, further comprising:

10. The user is a viewer, The method of claim 1 , wherein the request to provide the user-generated content includes a request to use a content feature that enables interaction with a content production.

11. Adjusting the similarity threshold based on an amount of one or more cues associated with the one or more screened users. The method of claim 1, further comprising:

12. The user is a content creator, The method of claim 1 , wherein the request to provide user-generated content comprises a request to use a content feature to create user-generated content.

13. The method of claim 1, wherein the second user-generated content is identified as similar to the user-generated content based on comparing an audio footprint associated with the user-generated content with the second user-generated content.

14. The user is a content creator, The method of claim 1 , wherein the corresponding action includes preventing viewers of the user-generated content from using content features that enable interaction with the content creator on a page of the media content platform that includes the user-generated content.

15. The user is a viewer, the request to provide user-generated content on the media content platform is a request to provide a comment on a page of the media content platform that includes the second user-generated content; The method of claim 1 , wherein the second user-generated content corresponds to content targeted to children.

16. The method of claim 15, wherein the request to provide user-generated content on the media content platform includes a request to use a particular content feature to generate the user-generated content, The method of claim 1 , wherein the second user-generated content is identified as similar to the user-generated content based on being created using the particular content feature.

17. The method described in claim 16, wherein the second user-generated content is identified as similar to the user-generated content further based on being created by the user of the user account using the particular content feature.

18. The method of claim 17, wherein the request to provide user-generated content on the media content platform includes a request to use a particular content feature to generate the user-generated content, 2. The method of claim 1, wherein the second user-generated content is identified as similar to the user-generated content based on the information associated with the user account that made a request to generate the user-generated content using the particular content feature.

19. The method of claim 18, wherein the second user-generated content is generated by the user; 20. The method of claim 18, wherein the information associated with the user account includes a number of viewers who left negative reviews related to the second user-generated content.

20. A system for identifying inappropriate content, comprising: a server having a memory; Hardware Processor and The hardware processor includes: receiving, at a server from a user device associated with a user, a request for providing user-generated content on a media content platform; In response to receiving the request, identifying a second user-generated content similar to the user-generated content, the second user-generated content having been previously reviewed; and determining a similarity of the user-generated content to the second user-generated content; determining whether the user-generated content should be reviewed by one or more review users by comparing at least the similarity to a similarity threshold indicated by one or more avoidance criteria; in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, adding a review request to a queue of review users, the review request including the user-generated content and information associated with a user account corresponding to the user; determining whether a determination in response to the review request indicates that the user generated content should be hosted on the media content platform; in response to determining that the determination indicates that the user generated content violates at least one policy associated with the media content platform; sending a first message to the user of the user device indicating the at least one violated policy and a penalty to be imposed on the user account corresponding to the user; causing a corresponding action to be taken based on said penalty; Including, the system.

Images