Method, system, and medium for identifying inappropriate content items

Abstract

To provide a method, a system and a medium for identifying an inappropriate content item.SOLUTION: A method includes the steps of: receiving a request for providing a user-generated content on a media content platform (PF) from a user device; determining whether or not the user-generated content should be examined by an examining user on the basis of an avoidance criterion; adding an examination request to a queue of the examination user; determining whether or not the content has to be provided on the PF; in the case of determining that the content infringes a policy of the PF, transmitting to the user device a message indicating the infringed policy and a penalty to be applied to a user; and transmitting the message indicating that the content is eligible for being provided on the PF to the user device in a case where an action to prohibit provision of the content on the PF is taken and no infringement is determined.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The disclosed subject matter relates to methods, systems, and media for identifying inappropriate content items. More specifically, the disclosed subject matter relates to determining whether user-generated content should be manually reviewed by one or more reviewers or whether the user-generated content can bypass manual review based on one or more criteria, and determining whether the user-generated content is inappropriate content so that corresponding action can be taken on the user-generated content. [Background technology]

[0002] Media content platforms have become the primary mechanism for users to discover and share digital media content. For example, content creators may upload videos or other content items that they have created to the media content platform, which then makes the uploaded content items available to viewers (e.g., by streaming the content items, by downloading the content items, etc.).

[0003] While many user-generated channels on a media content platform contain original and non-inappropriate content, there may be inappropriate content that violates the media content platform's policies and targets unsuspecting users, such as misleading, repetitive, suggestive, obscene, illegal, and / or "clickbait" content. For example, certain content items may contain objectionable content, such as violent content, offensive language, adult content, drug-related content, gambling content, etc., that the media content platform may want to remove or otherwise prevent users from accessing. As another example, the media content platform may want to prevent certain features of the service (e.g., features that enable content creators to collect revenue, and / or any other appropriate features) from being used with content items that violate these particular policies.

[0004] However, identifying content items that violate policies can be difficult. For example, some services may use human reviewers to manually review uploaded content items, but reviewing each uploaded piece of content to identify policy violations can be time-consuming. In such cases, a content item may be available on the media content platform for any number of days or weeks until the manual review process indicates that the content item should be blocked or removed due to a violation of the media content platform's policies. Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, it would be desirable to provide new methods, systems, and mediums for identifying inappropriate content items. [Means for solving the problem]

[0006] SUMMARY OF THE INVENTION Methods, systems, and media are provided for identifying inappropriate content items.

[0007] According to some embodiments of the disclosed subject matter, a method for identifying inappropriate content items is provided, the method including the steps of: receiving a request for providing user-generated content on a media content platform from a user device associated with a user; determining, in response to receiving the request, whether the user-generated content should be reviewed by one or more review users based on one or more avoidance criteria; and, in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, adding a review request to a queue of review users, the review request including the user-generated content and information associated with a user account corresponding to the user; and notifying the user of the review request that the user-generated content should be provided on the media content platform. and determining whether the determination responsive to the request indicates that the user-generated content item violates at least one policy associated with the media content platform, wherein in response to determining that the determination indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is sent to a user of the user device indicating the at least one violated policy and a penalty to be imposed on a user account corresponding to the user, and (ii) corresponding action is taken to prevent the user-generated content from being provided on the media content platform, and in response to determining that the determination indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is sent to the user of the user device indicating that the user-generated content is eligible for provision on the media content platform.

[0008] In some embodiments, the user is a content creator and the request to provide the user-generated content comprises an upload request to store the user-generated content on a storage device associated with the media content platform.

[0009] In some embodiments, the user is a content creator and the request to provide the user-generated content comprises a start request to begin a live stream using the media content platform.

[0010] In some embodiments, the user is a reviewer and the request to provide user-generated content includes a content function that interacts with the content creator.

[0011] In some embodiments, the method further comprises determining whether a user is eligible to provide user-generated content on the media content platform by calculating a risk score indicative of a risk associated with the user's user account based on other user-generated content previously provided to the media content platform, and comparing the risk score to a threshold risk score corresponding to a content type of the user-generated content, wherein the user is deemed eligible to provide user-generated content based on comparison of the risk score with the threshold risk score.

[0012] In some embodiments, the method further comprises determining whether the user is eligible to provide user-generated content on the media content platform based on audience information associated with the user's user account.

[0013] In some embodiments, the method further comprises determining whether the user is eligible to provide user-generated content on the media content platform based on location information associated with the user's user account.

[0014] In some embodiments, the method further comprises determining whether the user is eligible to provide user-generated content on the media content platform based on enforcement actions for previous policy violations associated with the user account.

[0015] In some embodiments, the method further comprises determining whether the user is eligible to provide user-generated content on the media content platform by sending a second review request to the review user's queue, the second review request including information associated with a user account corresponding to the user and prompting the review user to determine whether the information associated with the user account violates at least one policy associated with the media content platform.

[0016] In some embodiments, the method further comprises the step of causing, in response to receiving a request to provide user-generated content on the media content platform, timing instructions to be presented indicating when a decision responsive to the review request should be provided.

[0017] In some embodiments, the method further comprises receiving a flag from a viewer of the user-generated content on the media content platform, the flag indicating a possible policy violation; and in response to receiving the subsequent review request, adding the subsequent review request to a queue of review users that previously reviewed the user-generated content, the subsequent review request including the user-generated content and the flag from the viewer, and the user-generated content being restricted from being provided on the media content platform until a decision responsive to the subsequent review request is received.

[0018] In some embodiments, determining whether the user-generated content should be reviewed by the one or more review users based on the one or more avoidance criteria further comprises: identifying second user-generated content that is similar to the user-generated content, where the second user-generated content has previously been reviewed; and determining that the user-generated content should not be reviewed by the one or more review users based on the similarity of the user-generated content to the second user-generated content. In some embodiments, the method further comprises determining that the second user-generated content was previously restricted from being made available on the media content platform; and, in response to determining that the second user-generated content was previously restricted from being made available on the media content platform, performing a corresponding action that prevents the user-generated content from being made available on the media content platform, wherein the first message indicates the restriction based on the second user-generated content.

[0019] In some embodiments, determining whether the user-generated content should be reviewed by one or more review users further comprises determining whether at least one of the user-generated content and content related to the user-generated content has previously been flagged by a threshold number of viewers.

[0020] In some embodiments, the method further comprises updating a user account associated with the user to indicate at least one policy associated with the media content platform that the user-generated content violates.

[0021] In some embodiments, the penalty to be imposed on the user account corresponding to the user is based on the number of previous policy violations by the content creator.

[0022] In some embodiments, the penalty to be imposed on the user account corresponding to the user is based on the severity of at least one policy associated with the media content platform that the user-generated content violates.

[0023] In some embodiments, the method further comprises receiving a request to appeal the judgment from a user device, the appeal request including appeal text; adding the appeal request including the appeal text and the user-generated content to a queue of a second review user different from the review user that provided the judgment indicating that the user-generated content should be restricted from being provided on the media content platform; receiving an appeal judgment corresponding to the appeal request from a user device associated with the second review user; and determining at least one corresponding action to be taken on the user-generated content based on the judgment and the appeal judgment.

[0024] In some embodiments, the second message indicates a time period during which the user-generated content will be automatically made available on the media content platform in response to determining that the judgment indicates that the user-generated content item does not violate at least one policy associated with the media content platform.

[0025] According to some embodiments of the disclosed subject matter, a system for identifying inappropriate content items is provided, the system comprising: a memory; and a hardware processor, wherein the hardware processor, when executing computer-executable instructions stored in the memory, receives a request for providing user-generated content on a media content platform from a user device associated with a user; in response to receiving the request, determines whether the user-generated content should be reviewed by one or more review users based on one or more avoidance criteria; and in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, adds a review request to a queue of review users, the review request including the user-generated content and information associated with a user account corresponding to the user, and and (ii) a corresponding action is taken to prevent the user-generated content from being provided on the media content platform; and in response to determining that the determination indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is sent to the user of the user device indicating that the user-generated content item is eligible for provision on the media content platform.

[0026] According to some embodiments of the disclosed subject matter, a non-transitory computer-readable medium includes computer-executable instructions that, when executed by a processor, cause the processor to perform a method for identifying inappropriate content items, the method including: receiving a request to provide user-generated content on a media content platform from a user device associated with a user; determining, in response to receiving the request, whether the user-generated content should be reviewed by one or more review users based on one or more avoidance criteria; and, in response to determining, based on the one or more avoidance criteria, that the user-generated content should be reviewed, adding a review request to a queue of review users, the review request including the user-generated content and information associated with a user account corresponding to the user; and determining whether the user-generated content should be reviewed by one or more review users based on the one or more avoidance criteria. and determining whether a judgment responsive to the review request indicates that the user-generated content item should be provided on the platform, wherein in response to determining that the judgment indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is sent to a user of the user device indicating the at least one violated policy and a penalty to be imposed on a user account corresponding to the user, and (ii) corresponding action is taken to prevent the user-generated content from being provided on the media content platform, and in response to determining that the judgment indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is sent to the user of the user device indicating that the user-generated content is eligible for provision on the media content platform.

[0027] According to some embodiments of the disclosed subject matter, a system for identifying inappropriate content items is provided, the system including: means for receiving a request to provide user-generated content on a media content platform from a user device associated with a user; means for, in response to receiving the request, determining whether the user-generated content should be reviewed by one or more screening users based on one or more avoidance criteria; means for, in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, adding a screening request to a screening user's queue, the screening request including the user-generated content and information associated with a user account corresponding to the user; and means for determining that the user-generated content should be provided on the media content platform by a screening user. and means for determining whether a determination responsive to the request indicates that in response to the determination, (i) a first message is sent to a user of the user device indicating the at least one violated policy and a penalty to be imposed on a user account corresponding to the user, and (ii) corresponding action is taken to prevent the user-generated content from being provided on the media content platform, and in response to determining that the determination indicates that the user-generated content item does not violate at least one policy associated with the media content platform, a second message is sent to the user of the user device indicating that the user-generated content is eligible for provision on the media content platform.

[0028] Various objects, features, and advantages of the disclosed subject matter may be more fully understood by reference to the following detailed description of the disclosed subject matter when considered in conjunction with the following drawings, in which like reference numerals identify like elements and in which: [Brief explanation of the drawings]

[0029] [Figure 1] 1 shows an illustrative example of an information flow diagram for identifying inappropriate content items, according to some embodiments of the disclosed subject matter. [Figure 2] 1A-1C show illustrative examples of processes for identifying inappropriate content items according to some embodiments of the disclosed subject matter. [Figure 3] FIG. 1 is a schematic diagram of an illustrative system suitable for implementing mechanisms described herein for identifying inappropriate content items, according to some embodiments of the disclosed subject matter. [Figure 4] FIG. 4 illustrates a detailed example of hardware that may be used in the server and / or user device of FIG. 3 in accordance with some embodiments of the disclosed subject matter. DETAILED DESCRIPTION OF THE INVENTION

[0030] According to various embodiments, mechanisms (which may include methods, systems, and media) are provided for identifying inappropriate content items. More specifically, the disclosed subject matter relates to determining whether user-generated content should be manually reviewed by one or more reviewers or whether the user-generated content can be bypassed from manual review based on one or more criteria, and determining whether the user-generated content is inappropriate content so that corresponding action can be taken on the user-generated content.

[0031] Previous inappropriate content detection systems typically allow user-generated content to be published or otherwise made available until such content has been reviewed, which poses significant risks. The mechanisms disclosed herein can provide a multi-tiered inappropriateness mitigation approach that prevents user-generated content from being made available on a media content platform, for example, without requiring human review of each piece of content, while ensuring that user-generated content that does not meet one or more avoidance criteria is manually reviewed.

[0032] In some embodiments, the mechanisms described herein can receive content items from content item creators that are user-generated content items. User-generated content items may include any suitable type of user-generated content, such as, for example, video or audio content items generated by the content creator, comments entered by viewers of media content in connection with online chat or comment sections, use of content features to generate user-generated content, and / or any other suitable type of user-generated content. Note that user-generated content may include any suitable content, such as video content, audio content, text content, live-streamed video content, etc.

[0033] In some embodiments, the mechanism may determine whether the content item should be manually reviewed by one or more reviewers. In some embodiments, in response to determining that the user-generated content should be manually reviewed, the mechanism may add the user-generated content to a queue of content items to be reviewed by a reviewer (e.g., a reviewing user at an appropriate computing device). In some embodiments, the reviewer may then receive the user-generated content from the queue of content items and review the user-generated content to determine whether the user-generated content violates the policies of the media content platform. For example, in some embodiments, the reviewer may determine whether the user-generated content includes any particular type of objectionable content (e.g., violent content, offensive language, hate speech, adult content, and / or any other suitable type of objectionable content). In some embodiments, the reviewer may then transmit a decision indicating whether the user-generated content violates any policies of the media content platform. In some embodiments, the mechanism may then determine whether the content item should be posted to the media content platform in connection with the feature.

[0034] Note that in some embodiments, in response to receiving a determination from a first reviewer indicating that the user-generated content violates the media content platform's policies, the facility may add the user-generated content to another queue of content items to be reviewed by a different reviewer (e.g., a verifying reviewer, a reviewer with more experience than the first reviewer, etc.) Continuing with this example, in response to receiving determinations from the first reviewer and a second, different reviewer that both indicate that the user-generated content violates the media content platform's policies, corresponding action may be taken, such as removing the user-generated content from the media content platform or otherwise preventing the user-generated content from being made available on the media content platform.

[0035] In some embodiments, the mechanism may determine whether a content item should be manually reviewed based on any appropriate information. For example, in some embodiments, the mechanism may determine whether a content item should be manually reviewed based on information associated with the content creator, such as whether the content creator has previously uploaded an approved content item, whether the content creator has previously uploaded a content item that violates the media content platform's policies, and / or any other appropriate information associated with the content creator. As another example, in some embodiments, the mechanism may determine whether a content item should be manually reviewed based on the content item's similarity to previously reviewed content items. As a more specific example, in some embodiments, the mechanism may determine that a content item is similar to a previously rejected content item and may determine that the uploaded content item should also be rejected. In some embodiments, the mechanism may determine the criteria used to determine whether a content item should be added to a queue of items to be manually reviewed based on the number of received content items, thereby allowing the amount of manual review allocated to be adjusted.

[0036] In some embodiments, in instances where a review of the content item indicates that the content item violates a policy of the media content platform, the mechanism may cause a message to be sent to the content creator indicating any appropriate information. For example, in some embodiments, the message may indicate the policy that was violated, timestamp information indicating the portion of the content item that violates the policy, the penalty for the policy violation, and / or any other appropriate information. In some embodiments, the mechanism may determine the penalty based on any appropriate information, such as the number of previous violations by the content creator, the severity of the violation, and / or any other appropriate information. Additionally, in some embodiments, the mechanism may cause the content item to be blocked and / or removed from the media content platform.

[0037] In some embodiments, in response to determining that the user-generated content or use of a functionality to generate user-generated content is acceptable (e.g., in response to a reviewer determining that the content does not violate the media content platform's policies, in response to a determination that the content is likely to not violate the media content platform's policies, etc.), the facility may transmit a specific time when the user-generated content will be posted or otherwise made available on the media content platform (e.g., rather than publishing or posting the user-generated content before any review for violations of one or more policies of the media content platform, rather than having a delay after publishing or posting the content during which review of the content is submitted, etc.). Alternatively, the facility may transmit a specific time when the user-generated content may be posted or published at the direction of the content creator.

[0038] In some embodiments, the facility may receive an appeal of the decision from the content creator. In some such embodiments, the facility may add the appeal to a queue of appeals, and the appeal may be assigned to a reviewer. In some embodiments, any appropriate information may be provided to the reviewer of the appeal, such as the content item, information associated with the original decision being appealed (e.g., information indicating the policy that was violated, timing information indicating the portion of the content item that violated the policy, and / or any other appropriate information), text from the content creator (e.g., the content creator's argument and / or any other appropriate text), and / or any other appropriate information. In some embodiments, a decision on the appeal may then uphold the original decision, or conversely, may vacate the original decision. In cases where the original decision is vacated and the content item was previously blocked, the facility may make the content item available on the media content platform.

[0039] In some embodiments, the content item may be received from a content creator, where the content item is to be provided in connection with the content creator's use of the feature. For example, in some embodiments, the content item may be a media content item received by a media content platform for provision to viewers in connection with a particular feature. As a more specific example, in some embodiments, the feature may include providing membership services to viewers of a channel of content associated with the media content platform, where the membership may enable members to access any suitable benefits associated with the membership (e.g., access to special content, access to chat rooms or forums with other members, access to custom emojis or badges, and / or any other suitable benefits). As another more specific example, in some embodiments, the feature may include enabling content creators to offer merchandise (e.g., branded clothing and / or any other suitable branded merchandise) associated with the content item and / or associated with the channel of content to which the content item belongs. Additional details regarding the feature are described in more detail below in connection with FIG. 2. In some such embodiments, the mechanism may determine whether a content item should be manually reviewed to determine whether the content item is eligible to be provided to a user in connection with use of a feature. For example, in some embodiments, as described in more detail below in connection with FIG. 2, the mechanism may determine whether a content creator is eligible to use a feature, determine whether the content item should be manually reviewed, determine whether the content item should be blocked from being provided in connection with the feature, etc.

[0040] In a more specific example, before making user-generated content available on the media content platform, the institution may use human and machine reviewers to determine whether a user account is eligible to access media content platform functionality to generate user-generated content. This may, for example, limit access to media content platform functionality to user accounts that have been evaluated as innocent or non-inappropriate user accounts.

[0041] In some embodiments, a content item, as referred to herein, may refer to any suitable type of user-generated content, such as media generated and uploaded by a content creator, membership benefit offerings provided by a content creator in exchange for purchasing a membership associated with a channel of content produced by the content creator, comments entered by viewers of content in a comment section or social networking post, and / or any other suitable type of content. That is, in some embodiments, the mechanisms described herein may be used to determine whether any suitable type of user-generated content should be manually reviewed before posting the user-generated content to a particular page, forum, comment section, etc. For example, the mechanisms described herein may be used to review comments (e.g., comments submitted to a comment section associated with a content item, comments submitted to a chat forum, comments submitted to a social networking post, and / or any other suitable comment). As a more specific example, in some embodiments, the mechanisms described herein may be used to determine whether a particular comment should be added to a queue of comments to be manually reviewed or to determine whether the comment should be deleted. As another example, in some embodiments, the mechanisms described herein may be used to review benefits offered by a content creator in connection with membership to a channel of content offered by the content creator. As a more specific example, in some embodiments, the mechanisms described herein may be used to determine whether a particular benefit offered by a content creator should be added to a queue of benefits to be manually reviewed and / or whether a particular benefit should be revoked (e.g., due to violating any particular policy associated with the media content platform).

[0042] 1, an illustrative example 100 of an information flow diagram for identifying inappropriate content items is shown, in accordance with some embodiments of the disclosed subject matter. As shown, in some embodiments, the blocks of process 100 may be performed on a content creator device 308, a server 302, and / or a reviewer device 310, as shown and described below in connection with FIG.

[0043] At 102, the content creator device 308 can transmit user-generated content items to the server 302. In some embodiments, the user-generated content items may include any suitable type of content, such as videos, video playlists, links to video live streams, images, animations, photos, photo slideshows, documents, and / or any other suitable type of content. In a more specific example, the content creator device 308 can upload the user-generated content items for publication on a channel associated with a user account of a video sharing service. As another example, in some embodiments, the user-generated content items may include comments submitted by a user of the content creator device 308 to be posted in a chat room, in a live chat, in a comment section associated with a particular page, media content item, or social networking post, and / or any other suitable type of comment. In some embodiments, the content creator device 308 can transmit the user-generated content items in association with a user account associated with the content creator device 308 (e.g., a user account on a media content platform associated with the server 302 and / or any other suitable user account). In some embodiments, server 302 may be associated with a media content platform that hosts uploaded content and streams or otherwise provides content hosted by the media content platform to one or more user devices.

[0044] Note that in some embodiments, in response to receiving the user-generated content item, the server 302 can determine whether the user-generated content item should be reviewed by one or more review users. The server 302 can then cause a notice or other suitable interface to be presented on the content creator device 308 indicating that the user-generated content item or access to functionality for generating user-generated content is being reviewed. For example, the notice may indicate a particular amount of time until a decision from a reviewer is received. In another example, the notice may indicate that an updated notice will be presented upon approval of the user-generated content, thereby giving the content creator at the content creator device 308 control over publishing or otherwise making the user-generated content available on the media content platform.

[0045] At 104, the server 302 may add the content item to a queue of content items to be reviewed. In some embodiments, the queue of content items may correspond to a queue of content items to be manually reviewed by a human reviewer and / or to be reviewed in any other suitable manner. In some embodiments, the server 302 may maintain the queue in any suitable manner. For example, in some embodiments, the content item may be added to the queue based on the time the content item is received by the server 302 (e.g., such that items received later are placed later in the queue, and / or in any other manner). As another example, in some embodiments, the content item may be added to the queue based on information associated with a content creator associated with the content creator device 308. As a more specific example, in some embodiments, server 302 may determine the popularity of a content creator (e.g., based on the number of subscribers to a channel of content associated with the content creator, based on the number of views of content generated by the content creator, and / or based on any other suitable information) and may place content items in a queue based on the popularity of the content creator (e.g., content items received from relatively popular content creators may be inserted relatively higher in the queue, and / or in any other suitable manner). In some embodiments, server 302 may maintain a queue of content items based on any suitable combination of factors.

[0046] Note that in some embodiments, the server 302 may determine whether the content item should be added to a queue, as described in more detail below in connection with block 208 of FIG. 2 . In some embodiments, the server 302 may add the content item to a queue of content items to be reviewed in response to determining that the content item should be reviewed. Alternatively, in some embodiments, the server 302 may determine that the user-generated content item meets one or more avoidance criteria, such that the user-generated content item will not be reviewed and will not be added to the queue of content items to be reviewed at 104. In some such cases, the server 302 may use any suitable technique or combination of techniques to determine whether the user-generated content item should be posted or removed (or otherwise prevented from being made available on the media content platform), as described in more detail below in connection with blocks 208 and 210 of FIG. 2 . For example, in some embodiments, server 302 may determine that a user-generated content item is sufficiently similar (e.g., a threshold magnitude of similarity) to previously approved user-generated content, and in response, may allow the user-generated content to be made available on the media content platform. In another example, in some embodiments, server 302 may determine that the user-generated content is sufficiently similar (e.g., a threshold magnitude of similarity) to previously removed user-generated content, and in response, may automatically flag the user-generated content, remove the user-generated content, or otherwise prevent the user-generated content from being offered on the media content platform, without submitting the user-generated content to one or more reviewers.

[0047] In some embodiments, server 302 may receive an indication, such as a flag from any suitable user of the media content platform, that the user-generated content may violate the media content platform's policies (e.g., by including objectionable content within the user-generated content). In response to receiving such an indication, server 302 may add the user-generated content to a queue of content items to be reviewed by one or more reviewers (e.g., review users at suitable computing devices). This may, for example, ensure that the user-generated content is reviewed by one or more reviewers.

[0048] At 106, the examiner device 310 may receive an indication of the content item from the queue. In some embodiments, the examiner device 310 may receive the indication in any suitable manner. For example, in some embodiments, the examiner device 310 may receive this indication via a user interface through which a user of the examiner device 310 views the content item and submits a judgment regarding the content item (e.g., a judgment indicating that the content item violates a particular policy associated with the media content platform, a judgment indicating that the content item does not violate any policy associated with the media content platform, and / or any other appropriate judgment). As a more specific example, in some embodiments, the examiner device 310 may receive a notification indicating that the content item has reached the top of a queue of items to be reviewed and / or that the content item has been assigned to a user of the examiner device 310 for review. In some such embodiments, the notification may include any suitable information, such as a link to the content item, the name of the content creator associated with the content creator device 308, a topic associated with the content item, information associated with the content creator (e.g., the number of previously uploaded and / or approved content items created by the content creator, the number of times content previously uploaded by the content creator has been associated with policy violations, and / or any other suitable information), and / or any other suitable information.

[0049] In some embodiments, a user of the examiner device 310 may select a content item based on the instructions in any suitable manner (e.g., by selecting a link presented in the instructions and / or in any other suitable manner), which may cause the content item to be presented in a video player window or any other suitable user interface on the examiner device 310.

[0050] At 108, the reviewer device 310 may transmit a determination regarding the content item. For example, in some embodiments, the determination may indicate any suitable information regarding the content item, such as whether the content item includes content that violates one or more policies associated with the media content platform to which the content item was uploaded. As a more specific example, in some embodiments, the policies may include any suitable policy, such as content items that include violent content, sexual content, inappropriate language, hate speech, and / or any other suitable type of objectionable content.

[0051] In some embodiments, the transmitted determination may include any suitable information, such as an indication of the type of objectionable content contained in the content item, an indication of one or more portions of the content item that include the objectionable content (e.g., based on timestamps of the video, based on frames of the video, and / or in any other suitable manner), and / or any other suitable information. Note that in some embodiments, the determination may indicate that the content item does not violate any policies.

[0052] At 110, the server 302 may update an account associated with the content creator device based on the determination received from the reviewer device 310. For example, in the case where the determination indicates that the content item violates a particular policy, the server 302 may update a record indicating the number of times the content creator has uploaded content that violates policies associated with the media content platform. As another example, in the case where the determination indicates that the content item does not violate a particular policy, the server 302 may update a record indicating the number of times the content creator has uploaded content that does not violate any policies associated with the media content platform (i.e., the number of times the content creator has uploaded content that has been approved by the reviewer).

[0053] In some embodiments, the determination may be viewable by a user of the content creator device 308. For example, in some embodiments, a user of the content creator device 308 may log in to a user account associated with the content creator device 308 and the media content platform and view the determination in a user interface associated with the user account. In another example, in response to a determination indicating that the user-generated content or use of a feature to generate user-generated content is acceptable (e.g., in response to a reviewer determining that the content does not violate the media content platform's policies, in response to a determination that the content is likely to not violate the media content platform's policies, etc.), the determination may be presented on the content creator device 308 along with a specific time by which the user-generated content will be posted or otherwise made available on the media content platform (e.g., rather than publishing or posting the user-generated content before any review for violations of one or more policies of the media content platform, rather than there being a delay after publishing or posting the content during which review of the content is submitted, etc.). Alternatively, in some embodiments, the determination may be presented along with a specific time by which the user-generated content may be posted or published at the content creator's direction.

[0054] It should be noted that in some embodiments, in response to a reviewer providing a determination that the user-generated content violates the media content platform's policies, the server 302 may add the content to a queue of content items to be reviewed by another reviewer. It should also be noted that in some embodiments, the additional reviewer may be a different reviewer with more experience compared to the reviewer who provided the initial determination. In such embodiments, the server 302 may remove the user-generated content or otherwise prevent the user-generated content from being provided by the media content platform in response to receiving two negative determinations from two different reviewers indicating that the user-generated content violates the media content platform's policies.

[0055] In some embodiments, the decision may be appealed by a user of the content creator device 308. For example, in some embodiments, the decision may be appealed in cases where the user of the content creator device 308 disagrees with the decision (e.g., in cases where the decision indicates that a content item contains a particular type of objectionable content and / or violates a particular policy, and the user disagrees, and / or in any other suitable case). In some embodiments, blocks 112 through 118 shown in FIG. 1 and described below may be performed in cases where the decision is appealed. In some embodiments, blocks 112 through 118 may be omitted in cases where the decision is not appealed.

[0056] At 112, the content creator device 308 may transmit an indication that the decision is being appealed. In some embodiments, the content creator device 308 may transmit the indication in any suitable manner. For example, in some embodiments, a user of the content creator device 308 may select any suitable button or user interface control associated with the decision that, when selected, brings up a user interface for entering any suitable text associated with the appeal of the decision (e.g., any suitable reasons for disputing the decision and / or other suitable text). In some such embodiments, the text associated with the appeal of the decision may be transmitted to the server 302 in association with the indication that the decision is being appealed.

[0057] At 114, the server 302 may add the content item to the appeals queue. In some embodiments, the content item may be added to the appeals queue in any suitable manner. For example, in some embodiments, the content item may be added to the appeals queue based on the time the indication that the decision is being appealed was sent by the content creator device 308 at 112. As another example, in some embodiments, the content item may be added to the appeals queue based on the relative popularity of the content creator associated with the content item, as described above in connection with block 104.

[0058] At 116, the reviewer device 310 may receive the indication of the content item from the appeals queue. In some embodiments, the reviewer device 110 may receive the indication of the content item in any suitable manner. For example, in some embodiments, the reviewer device 110 may present a notice indicating that a decision associated with the content item has been appealed by a content creator associated with the content item.

[0059] In some embodiments, a user of the reviewer device 310 may select this indication in any suitable manner, and in some embodiments, selection of this indication may cause the content item to be presented on the reviewer device 310 in association with any suitable information associated with the decision and / or any suitable information associated with the appeal. For example, in some embodiments, this information may include text associated with the appeal (e.g., as described above in connection with block 110), information associated with the decision being appealed (e.g., information indicating the objectionable content or policy violations indicated in the decision, information indicating the timestamps at which particular objectionable content as presented in the decision was indicated, and / or any other suitable information), and / or any other suitable information.

[0060] 1 shows the same device (i.e., reviewer device 310) as both reviewing the content item at 106 and reviewing the content item appeal at 116, it should be noted that in some embodiments, two different reviewers may be used. That is, in some embodiments, a reviewer different from the reviewer that submitted the appealed decision may review the appeal of the decision. For example, a different reviewer on a different reviewer device may be used to review the user-generated content in the appeals queue, the different reviewer designated as more experienced than the original reviewer who provided a negative decision, and may take action such as suspending the user account or preventing the submission of the user-generated content in response to both the original reviewer and the different appeal reviewer providing negative decisions.

[0061] At 118, the reviewer device 310 may transmit a decision regarding the appeal of the content item. In some embodiments, the decision may indicate any suitable information, such as that the user of the reviewer device 310 agrees with the decision being appealed, that the user of the reviewer device 310 disagrees with the decision being appealed, and / or any other suitable information.

[0062] At 120, the server 302 may update an account associated with the content creator device based on the appeal decision. For example, in an instance where the appealed decision indicates that the content item violates a particular policy and the appeal decision disagrees with the appealed decision, the server 302 may update the account to reflect the discrepancy between the two decisions. As a more specific example, in some embodiments, the server 302 may decrement a record of the number of times the content creator has uploaded content items that violate policies associated with the media content platform. As another example, in an instance where the appeal decision confirms the appealed decision that the content item violates a particular policy, the server 302 may take any appropriate action, such as blocking the content item, prohibiting the content creator device 308 from uploading content items for a predetermined length of time (e.g., one month, 90 days, and / or any other suitable length of time), and / or any other appropriate action. Further details about actions that may be performed by the server 302 are described below in connection with block 224 of FIG. 2 .

[0063] 2, an illustrative example 200 of a process for identifying inappropriate content is shown, in accordance with some embodiments of the disclosed subject matter. In some embodiments, the blocks of process 200 may be performed on a server, such as a server associated with a media content platform that hosts content items uploaded by one or more content creators.

[0064] Process 200 may begin at 202 by receiving from a content creator a content item to be uploaded or posted in connection with a media content platform. For example, in some embodiments, the content item may be a video or audio content item produced by the content creator to be uploaded to the media content platform and made available for viewing by other users of the media content platform. As another example, in some embodiments, the content item may be a comment submitted by a viewer of a content item hosted by the media content platform. As a more specific example, in some embodiments, the comment may be a comment submitted by a viewer in connection with a chat room or live chat associated with a particular channel of content, a comment submitted by a viewer to be posted in a particular comment section (e.g., a comment section associated with a particular media content item hosted by the media content platform and / or any other suitable comment section), and / or any other suitable comment. As yet another example, in some embodiments, the content item may be a benefit being offered by the content creator via the media content platform in connection with a channel of content produced by the content creator. As a more specific example, a reward may be an offer of a particular type of content (e.g., exclusive video content, unreleased content, clothing or other merchandise, and / or any other suitable type of content) provided by a content creator in exchange for a user purchasing a membership or subscription to a channel of the content.

[0065] In some embodiments, the uploaded content item may be uploaded in association with a particular feature of the media content platform, e.g., a feature that the content creator is interested in using to generate user-generated content. In some embodiments, the feature may include functionality associated with comments or messages that may be submitted by reviewers of the content item. For example, in some embodiments, the feature may relate to comments in a comment section associated with the content item and / or comments posted in a live chat associated with the content item. As a more specific example, the feature may include enabling reviewers of the content item to purchase “Super Chats,” which may include enabling viewers to pin comments associated with the purchased “Super Chats” to a particular location in the chat or comment section (e.g., at the top of the live chat, at the top of the comment section, and / or any other suitable location). Continuing with this example, in some embodiments, the feature may include determining the length of time for which the “Super Chat” comment should be pinned to a particular location based on the amount of payment made by the reviewer who purchased the “Super Chat.” As another more specific example, in some embodiments, the functionality may include enabling viewers of a content item to purchase "Super Stickers" that enable viewers of the content item to include animated images or icons in a comment section or live chat associated with the content item. Continuing with this example, in some embodiments, the functionality may include pinning a particular "Super Sticker" image or icon at a particular location in the live chat or comment section in response to determining that any particular criteria have been met (e.g., the viewer has paid more than a predetermined amount for the "Super Sticker" and / or any other suitable criteria).It should be noted that in some embodiments, the animated images or icons selectable by a viewer of a content item may be organized in any suitable manner and may be generated and made available, for example, by the content creator, by the media content platform, and / or by any other suitable entity. As yet another more specific example, in some embodiments, a functionality may include enabling a viewer of a content item to purchase a "Super Thanks." In some such embodiments, a viewer's purchase of a "Super Thanks" may result in a selected animated image or icon (e.g., selected by the viewer of the content item in connection with the purchase of the "Super Thanks") being presented in association with the content item and / or may result in any suitable percentage of the viewer's payment being shared with the content creator.

[0066] It should be noted that in some embodiments, the functionality may enable the content creator to receive any suitable percentage of the revenue from purchased "Super Chats," "Super Stickers," and / or "Super Thanks." For example, in some embodiments, the content creator may receive a percentage of the revenue from the purchased options, and the media content platform may receive the remainder of the revenue. Additionally, or alternatively, in some embodiments, the functionality may enable the content creator to designate any suitable charity as receiving a percentage of the revenue.

[0067] In some embodiments, functionality may enable content creators to offer membership services to viewers of content produced by the content creator. For example, in some embodiments, the membership service may enable viewers who purchase a membership to receive any suitable benefits related to content produced by the content creator. In some embodiments, the benefits may include any suitable special offers available to viewers who purchase a membership, such as access to special or exclusive content produced by the content creator (e.g., available only to viewers who purchase a membership), access to exclusive emojis or other animated icons that may be included in chat messages or comment sections associated with content produced by the content creator, access to an exclusive chat forum where viewers who have purchased a membership can chat with other viewers who have also purchased a membership, and / or any other suitable special offers available to members.

[0068] In some embodiments, the functionality may enable content creators to offer merchandise through the media content platform. For example, in some embodiments, the functionality may enable content creators to include links to merchandise (e.g., clothing, mugs or cups, artwork, and / or any other suitable merchandise) on pages presenting content items or other content provided by the content creators through the media content platform. As a more specific example, the functionality may enable content creators to include links to one or more merchandise available for purchase on video viewing pages presenting content items. As another more specific example, the functionality may enable content creators to include links to one or more merchandise available for purchase on pages associated with channels of content associated with the content creators.

[0069] In some embodiments, the functionality may enable a content creator to provide one or more links for purchasing tickets to a live show associated with the content creator. For example, in instances where the content item is a music video associated with a particular band or other musical group, the functionality may enable a link for purchasing tickets to a concert associated with the band or musical group to be presented on the page on which the content item is presented (e.g., below a video player window on which the content item is presented, and / or in any other suitable location). Note that in some embodiments, the functionality may enable the link for purchasing tickets to be presented in connection with any other suitable page or content, such as a page associated with a channel for content produced by the content creator and / or any other suitable content.

[0070] At 204, process 200 may determine whether the content creator is eligible to use the functionality for generating user-generated content. In some embodiments, process 200 may determine whether the content creator is eligible to use the functionality based on any suitable information and using any suitable technique. For example, in some embodiments, process 200 may determine whether the content creator is eligible to use the functionality based on content items that the content creator previously uploaded to the media content platform. As a more specific example, in some embodiments, process 200 may use any suitable classifier or machine learning technique to determine a risk score indicative of the probability that the content creator will upload video content containing one or more types of objectionable content based on the previously uploaded content items. As one particular example, process 200 may determine that a content creator whose risk score is below a predetermined threshold is ineligible to use the functionality. As another particular example, in some embodiments, process 200 may determine that a content creator whose risk score is above a predetermined threshold is eligible to use the functionality. As another more specific example, in some embodiments, process 200 may determine the number of times a content creator has previously violated one or more policies associated with the media content platform (e.g., policies prohibiting particular types of objectionable content, and / or any other suitable policies). As a particular example, in some embodiments, process 200 may determine that a content creator who has violated one or more policies of the media content platform more than a predetermined threshold number of times (e.g., more than once, more than twice, more than once in the past month, and / or any other suitable number of times) is ineligible to use the functionality.

[0071] As another example, in some embodiments, process 200 may determine whether a content creator is eligible to use a feature based on information associated with the content creator, such as the size of the content creator's audience and / or a location associated with the content creator. As a more specific example, in some embodiments, process 200 may determine that a content creator with a relatively small audience is ineligible to use a feature. As another more specific example, in some embodiments, process 200 may determine that a content creator with a relatively large audience is eligible to use a feature. In some embodiments, audience size may be determined based on any suitable information, such as the total number of views of content items previously uploaded by the content creator, the average number of views of content items previously uploaded by the content creator, the number of subscribers to a channel of content produced by the content creator, and / or any other suitable measure.

[0072] As yet another example, in some embodiments, process 200 may determine whether a content creator is eligible to use a feature based on a manual review of content posted by the content creator. As a more specific example, in some embodiments, a human reviewer may review one or more content items previously posted by the content creator and / or content items uploaded by the content creator, as described above in connection with block 202, and may determine whether the content items violate one or more policies of the media content platform. In some embodiments, in response to determining that the content item violates one or more policies of the media content platform, process 200 may determine that the content creator is ineligible to use a feature.

[0073] As yet another example, in some embodiments, process 200 may determine whether a content creator is eligible to use a feature based on any other suitable information associated with the content creator and / or associated with the content item associated with the content creator. As a more specific example, in some embodiments, process 200 may determine whether the content creator has previously been approved to participate in a program associated with the media content platform that allows the content creator to use particular features associated with membership in the program. As another more specific example, in some embodiments, process 200 may determine whether the content creator's age is above a predetermined age (e.g., above 18 years of age and / or above any other appropriate age). As yet another more specific example, process 200 may determine whether a content item associated with the content creator is associated with a particular genre (e.g., age-restricted content, content targeted to children, music video content, and / or any other suitable genre of content). As one particular example, in some embodiments, process 200 may determine that a content creator is ineligible to use the feature if the content items associated with the content creator typically belong to a particular genre, such as age-restricted content, content targeted to children, music video content, etc.

[0074] As yet another example, in some embodiments, process 200 may determine whether a content creator is eligible to use a feature based on determining whether viewers of content created by the content creator are satisfied or dissatisfied with the content creator's use of the feature. As a more specific example, in some embodiments, in cases where a feature includes providing benefits to viewers who purchase memberships associated with content produced by the content creator, process 200 may determine whether users who purchased the memberships were generally satisfied or dissatisfied with their purchases. As another more specific example, in some embodiments, in cases where a feature includes allowing viewers to purchase pinned comments, pinned animated images, or other messages or chat content, process 200 may determine whether users who purchased the pinned messages or chat content were generally satisfied or dissatisfied with their purchases. In some embodiments, process 200 may determine the viewer satisfaction measure in any suitable manner. For example, in some embodiments, process 200 may determine the number or percentage of viewers who canceled memberships, the number or percentage of viewers who left negative reviews in response to an offered benefit, the number or percentage of viewers who unsubscribed from a channel of content produced by the content creator, and / or any other suitable satisfaction measure. In some embodiments, process 200 may determine that the content creator is ineligible to use a feature in response to determining that the calculated satisfaction measure is below a predetermined threshold. As another example, in some embodiments, process 200 may determine that the content creator is ineligible to use a feature in response to determining that a predetermined percentage of viewers (e.g., greater than 10%, greater than 20%, and / or any other suitable percentage) canceled a purchased membership or left a negative review in connection with an offered benefit or purchase option.

[0075] Note that in cases where the content item corresponds to a submitted comment, process 200 may determine whether the user submitting the comment is eligible to post the comment in a particular chat or a particular comment section. For example, in cases where the comment is submitted in connection with a chat or comment section that requires subscription to a particular channel of content on a media content platform, or any other suitable criteria to be met, process 200 may determine whether the user submitting the comment is a subscriber to the channel of the content, or meets any other suitable criteria. Additionally or alternatively, in some embodiments, in cases where the content item corresponds to a submitted comment, block 204 may be omitted.

[0076] Additionally, in some embodiments, process 200 can determine whether a content creator is eligible to use a feature based on any suitable combination of information. For example, in some embodiments, process 200 can determine whether a content creator is eligible to use a feature based on a combination of a risk score associated with the content creator, information indicative of the size of the content creator's audience, information indicative of previous violations of policies by the content creator, and / or manual review of the content item. In some embodiments, process 200 can combine information in any other suitable manner. For example, in some embodiments, process 200 can assign a score to each item of information and determine an aggregate score associated with the content creator based on the score of each item of information. In some embodiments, process 200 can then determine whether the content creator is eligible to use the feature based on a determination of whether the aggregate score associated with the content creator exceeds a predetermined threshold.

[0077] If process 200 determines at 204 that the content creator is not eligible to use the feature (“no” at 204), process 200 may proceed to 206 and reject the content item. In some embodiments, process 200 may reject the content item in any suitable manner. For example, in some embodiments, process 200 may block the content item from being uploaded to a server associated with the media content platform. As another example, in some embodiments, process 200 may send a message to the content creator and / or cause a notification to be presented on a user device associated with the content creator indicating that the content item has been rejected. In some such embodiments, the message and / or notification may include any suitable information, such as the reason the content item was rejected (e.g., a previous violation by the content creator of one or more policies associated with the media content platform and / or any other suitable reason). As yet another example, in some embodiments, in instances where the content item corresponds to a comment to be posted to a chat or comment section, process 200 may prevent the comment from being posted to the chat or comment section.

[0078] Note that in some embodiments, in response to determining that a content creator is ineligible to use a feature, process 200 may allow a content item to be uploaded but may deny the content creator use of the feature. For example, in cases where process 200 determines that the content creator has not previously violated any policies associated with the media content platform but that the content creator is ineligible to use the feature for other reasons (e.g., because the content creator does not have a sufficiently large audience, because the content creator has not previously uploaded more than a predetermined number of content items, and / or any other suitable reason), process 200 may allow a content item to be uploaded to the media content platform but may block or inhibit the content creator from using the feature.

[0079] If process 200 determines at 204 that the content creator is eligible to use the feature ("yes" at 204), process 200 may proceed to 208 to determine whether to selectively bypass review of user-generated content items. In some embodiments, process 200 may determine whether to bypass review of content items based on any suitable information.

[0080] For example, in some embodiments, process 200 may determine whether review of a content item should be avoided if the content item contains objectionable content based on the relative risk of the content item's publication. In some embodiments, the relative risk of the content item's publication may be determined based on any suitable information, such as the expected size of the content item's audience (e.g., the expected number of people who are likely to view the content item), whether the content creator will receive revenue from viewers of the content item, the expected revenue the content creator will receive from viewers of the content item, and / or any other suitable information. Note that in some embodiments, process 200 may determine that the content item should be assigned a relatively high risk score in response to determining that the content item meets any particular criteria. For example, in some embodiments, process 200 may determine that a content item should be assigned a relatively high risk score in response to determining that the content item is being offered in exchange for a payment by a viewer (e.g., in instances where the content item is exclusive content offered to users who pay for the content, where the content item is a perk offered to users who pay for membership to a channel of the content, and / or where the content item is any other suitable type of content).

[0081] In some embodiments, process 200 can determine the relative risk in any suitable manner. For example, in some embodiments, process 200 can identify multiple risk scores (e.g., a first risk score corresponding to an estimated audience size, a second risk score corresponding to expected revenue for the content creator from viewers of the content item, and / or any other suitable risk score) and can combine the multiple risk scores in any suitable manner (e.g., as a weighted average and / or in any other suitable manner). As a more specific example, in the case where the content item corresponds to a video content item to be posted to a content channel that has a relatively large expected audience (e.g., based on the channel having more than a predetermined number of subscribers, based on content associated with the channel having more than a predetermined number of previous views, and / or based on any other suitable measure), process 200 can determine that the relative risk of publishing the content item is high. As another more specific example, in instances where the content item corresponds to a video content item that has been monetized in any suitable manner (e.g., by inserting advertisements, by providing the content item to users who have paid to receive the content item, and / or in any other suitable manner), process 200 may determine that the relative risk of publication of the content item is high. As yet another more specific example, in instances where the content item corresponds to a comment that will be posted in a comments section, process 200 may determine that the relative risk of publication of the content item is low. In some embodiments, process 200 may determine that content items associated with a relative risk of publication above a predetermined threshold should be manually reviewed (i.e., not to be bypassed for manual review, or "no" at 208).It should be noted that in some embodiments, in instances where process 200 determines that a content item is associated with a relative risk of publication that exceeds a certain predetermined threshold, process 200 can determine that the content item should be automatically rejected.

[0082] As another example, in some embodiments, process 200 may determine that review of a content item should be avoided in response to determining that the content item is similar to or matches a previously uploaded content item that has been previously reviewed. As a more specific example, in some embodiments, process 200 may determine that the content item is similar to or matches a content item that has previously been reviewed and approved. As another more specific example, in some embodiments, process 200 may determine that the content item is similar to or matches a content item that has previously been reviewed and rejected.

[0083] It should be noted that in some embodiments, process 200 may use any suitable technique or combination of techniques to determine whether an uploaded content item is similar to or matches a previously reviewed content item. For example, in some embodiments, process 200 may compare one or more audio and / or video fingerprints of the uploaded content item to a database of previously uploaded content items to identify matches to previously uploaded and reviewed content items.

[0084] Additionally, it should be noted that in some embodiments, process 200 may determine a degree of similarity required between an uploaded content item and at least one previously reviewed content item to determine that review should be avoided. For example, in some embodiments, process 200 may determine that an uploaded content item should have at least a predetermined threshold similarity to a previously reviewed content item (e.g., at least 70% similarity, at least 80% similarity, and / or any other suitable threshold) to determine that review should be avoided. In some embodiments, the predetermined threshold of similarity may be determined or set in any suitable manner. For example, in some embodiments, the predetermined threshold of similarity may be determined by process 200 such that, when the predetermined threshold of similarity is applied, a predetermined range of content items are sent for manual review. That is, in some embodiments, process 200 may adjust the predetermined threshold of similarity over time based on the number of content items uploaded to the media content platform, thereby allowing the amount of manual review performed to be adjusted.

[0085] Additionally, it should be noted that in some embodiments, process 200 can determine whether manual review should be avoided based on any suitable combination of the relative risk of publication of the content item and the similarity between the uploaded content item and previously uploaded content items. For example, in some embodiments, process 200 can calculate a risk score indicating the relative risk of publication of the content item and a similarity score indicating the similarity between the content item and previously uploaded content items, and can determine whether manual review should be avoided based on both the risk score and the similarity score. As a more specific example, in some embodiments, process 200 can determine that manual review should not be avoided (“no” at 208) in response to determining that the risk score exceeds a first predetermined threshold and the similarity score is less than a second predetermined threshold (i.e., not sufficiently similar to the previously uploaded content item). As another more specific example, in some embodiments, process 200 may determine that manual review should be avoided (“yes” at 208) in response to determining that the risk score is below a first predetermined threshold and the similarity score is above a second predetermined threshold.

[0086] In some embodiments, process 200 may determine that certain types of content items should be manually reviewed, regardless of the risk score. For example, in some embodiments, process 200 may determine that content items offered in exchange for payment by a content item reviewer (e.g., exclusive content offered to paying users and / or any other suitable type of paid content) should always be manually reviewed. As another example, in some embodiments, process 200 may determine that content items generated by content creators who are offering paid content for the first time should be manually reviewed. In some embodiments, paid content items may include video content, audio content, “super stickers,” “super comments” (as described above with respect to block 202), and / or any other suitable type of content made available to viewers in exchange for payment.

[0087] If process 200 determines at 208 that review of the content item should be bypassed (“yes” at 208), process 200 may proceed to 210 and determine whether the content item should be rejected as inappropriate content. In some embodiments, process 200 may determine whether the content item should be rejected based on whether it was determined at block 208 that the uploaded content item is similar to or matches a previously rejected content item. For example, in some embodiments, process 200 may determine that the uploaded content item is similar to or matches a previously rejected content item that was rejected for containing one or more types of objectionable content, and in some such embodiments, process 200 may determine that the content item should be rejected. As another example, in some embodiments, process 200 may determine that the uploaded content item is similar to or matches a previously approved content item, and in some such embodiments, process 200 may determine that the content item should not be rejected. As yet another example, in some embodiments, process 200 may use any suitable technique to determine the probability that a content item contains any particular type of objectionable content. As a more specific example, in some embodiments, process 200 may use any suitable classifier or combination of classifiers to determine the probability that a content item contains objectionable content. It should be noted that in instances where the content item corresponds to video or audio content, process 200 may use a classifier that takes as input any suitable audio or video fingerprint of the content item.In some embodiments, in cases where the content item corresponds to a submitted comment, process 200 may use a classifier that takes as input any suitable text or link contained in the comment to determine the probability that the text contains objectionable content or the probability that the link is to a site associated with objectionable content. Continuing with this example, in some embodiments, process 200 may determine that the content item should be rejected in response to determining that the determined probability that the content item contains objectionable content exceeds a predetermined threshold (e.g., greater than 70%, greater than 80%, and / or any other suitable threshold).

[0088] If the process 200 determines at 210 that the content item should be rejected (“yes” at 210), the process 200 may reject the content item at 206 as described above.

[0089] If process 200 determines at 210 that the content item should not be rejected (“no” at 210), process 200 may proceed to 212, where the content item may be posted to the media content platform. In some embodiments, process 200 may post the content item to the media content platform in any suitable manner. For example, in some embodiments, process 200 may upload the content item to a particular channel associated with the content creator. As another example, in some embodiments, process 200 may make a link to the content item available to other user devices to stream or download the content item. As another example, in some embodiments, process 200 may send a notification to the content creator indicating that the content item has been approved. In some such embodiments, the content creator may then initiate posting of the content item in any suitable manner, for example, by selecting any suitable selectable input that makes the content item available to other viewers. As yet another example, in instances where the content item corresponds to a live stream of content, process 200 may make the live stream available to viewers after any suitable time delay (e.g., after 30 seconds, after 1 minute, and / or after any other suitable time delay). As yet another example, in instances where the content item corresponds to a comment to be posted to a comment section or chat, process 200 may cause the comment to be added to the chat or comment section in any suitable manner (e.g., at the bottom of the chat, at the bottom of the comment section, in a pinned position in the chat or comment section, and / or in any other suitable manner).

[0090] Returning to block 208, if process 200 determines at 208 that review of the content item should not be bypassed (“no” at 208), process 200 may proceed to 214, where the content item may be added to a queue of content items to be reviewed. As described above in connection with block 104 of FIG. 1 , in some embodiments, process 200 may add the content item to the queue of content items to be reviewed in any suitable manner. For example, in some embodiments, the content item may be added to the queue based on the date or time the content item was received at block 202 (e.g., such that previously received items are placed earlier in the queue, and / or in any other suitable manner). As another example, in some embodiments, the content item may be added to the queue based on the popularity of other content items associated with the content creator. As a more specific example, in some embodiments, content items associated with content creators who previously uploaded content items that received more than a predetermined number of views, shares, comments, etc. may be placed higher in the queue. As yet another example, in some embodiments, in instances where the release of a content item is time-sensitive (e.g., where the content item is live-streamed content, where the content item is a live chat comment, and / or any other suitable time-sensitive situation), process 200 may add the content item to a relatively high position in the queue (e.g., to the front of the queue and / or any other suitable position), thereby allowing the content item to be manually reviewed more quickly.

[0091] In response to the content item being moved to the head of a queue of content items to be reviewed, the content item may be assigned to a reviewer, as described above in connection with Figure 1. In some such embodiments, the content item may be presented to a user device associated with the reviewer, as described above in connection with Figure 1. In some embodiments, the reviewer may then use the user device associated with the reviewer to enter any information associated with their judgment regarding the content item, such as whether the content item is determined to include one or more types of objectionable content, frames or timestamps of the content item that include the objectionable content in cases where the content item corresponds to video or audio content, one or more policies that the content item violates, and / or any other suitable information.

[0092] At 216, process 200 may receive a decision regarding the content item. In some embodiments, process 200 may receive the decision in any other suitable manner. For example, in some embodiments, server-executed process 200 may receive a message from a user device associated with a reviewer who reviewed the content item. In some such embodiments, the message may include any suitable information, such as an indication that the content item does not violate any policies associated with the media content platform, an indication that the content item violates one or more policies associated with the media content platform, an indication of one or more types of objectionable content contained in the content item, timestamps of the content items that include the one or more types of objectionable content, and / or any other suitable information.

[0093] At 218, the process 200 can determine whether the determination indicates a policy violation.

[0094] If the process 200 determines at 218 that the determination does not indicate a policy violation (“no” at 218), the process 200 may post the content item to the media content platform at 212 as described above.

[0095] If process 200 determines at 218 that the determination indicates a policy violation (“yes” at 218), process 200 may proceed to 220, where a message indicating the policy violation may be sent to the content creator. In some embodiments, process 200 may send the message to the content creator in any suitable manner. For example, in some embodiments, process 200 may send the message to an email address associated with the content creator. As another example, in some embodiments, process 200 may send the message in association with the content creator and a user account associated with the media content platform. As a more specific example, in some embodiments, the message may be a notice presented on the home page of a user account associated with the content creator in association with the media content platform. This may provide the content creator with accurate feedback, for example, about policy violations and other reasons for taking action that may prevent user-generated content from being made available on the media content platform.

[0096] In some embodiments, the message may include any suitable information. For example, in some embodiments, the message may indicate one or more policies that the content item violates. As a more specific example, in some embodiments, the message may indicate that the content item includes one or more types of objectionable content that violate one or more policies of the media content platform. As another example, in some embodiments, the message may indicate one or more portions of the content item that violate a policy of the media content platform using any suitable timing information, such as a timestamp of the content item, a frame of the content item, and / or any other suitable timing information. As yet another example, in some embodiments, the message may indicate a penalty. As a more specific example, in some embodiments, the message may indicate that the content item has been blocked from being uploaded to the media content platform. As another more specific example, in some embodiments, the message may indicate that the content creator will be prohibited from uploading content items for a predetermined length of time (e.g., one month, 90 days, and / or any other suitable length of time). As yet another more specific example, in some embodiments, the message may indicate that an account associated with the content creator will be suspended. As yet another more specific example, in some embodiments, the message may indicate that the content creator is not allowed to use the feature for a predetermined length of time (e.g., 30 days, 90 days, and / or any other suitable length of time). As yet another more specific example, in some embodiments, the message may indicate that the content creator is not allowed to stream live content for a predetermined length of time (e.g., 30 days, 90 days, and / or any other suitable length of time).As yet another more specific example, in the instance where the content item corresponds to a comment to be posted to a chat or comment section, the message may indicate that the user will not be allowed to post comments in the comment section associated with the rejected comment or in any comment section associated with the media content platform for a particular length of time.

[0097] It should be noted that in some embodiments, process 200 can determine a penalty based on the policy that the content item violates. For example, in some embodiments, process 200 can determine a penalty based on the severity of the violation. As a more specific example, in some embodiments, process 200 can determine a first penalty (e.g., blocking the content item) in response to determining that the policy violation corresponds to the inclusion of a first type of objectionable content in the content item, and can determine a second penalty (e.g., terminating the user account associated with the content creator) in response to determining that the policy violation corresponds to the inclusion of a second type of objectionable content in the content item. Additionally, it should be noted that in some embodiments, process 200 can determine that a particular penalty (e.g., account suspension, prohibition from uploading content for a predetermined length of time, and / or any other appropriate penalty) should be applied in response to determining that a content creator has violated one or more policies more than a predetermined number of times (e.g., more than three violations, more than five violations, and / or any other suitable number of times). This can, for example, discourage policy violations through increasingly severe enforcement actions. Additionally, it should be noted that in some embodiments, process 200 can determine a penalty based on the type of content item. For example, in some embodiments, in cases where the content item corresponds to an upload of video or audio content to be made available to other viewers, process 200 can identify a penalty associated with a temporary or permanent block on uploading other video or audio content to the media content platform.As another example, in some embodiments, in cases where a content item corresponds to a comment submitted to a chat or comment section, process 200 may identify a penalty associated with a temporary or permanent block on submitting the comment.

[0098] It should be noted that in some embodiments, before proceeding to block 220, process 200 may add the content item to a second queue so that the content item is then reviewed a second time by a second reviewer. That is, in some embodiments, in response to a determination that the judgment indicates a policy violation, process 200 may cause the content item to be reviewed a second time by a second reviewer, different from the first reviewer, to confirm that the content item violates the media content platform's policies.

[0099] At 222, process 200 may update an account associated with the content creator based on the policy violation. For example, in some embodiments, process 200 may update the account to indicate that the uploaded content item violated a particular policy. As another example, in some embodiments, process 200 may update the account to indicate that the uploaded content item was determined to contain a particular type of objectionable content. As yet another example, in some embodiments, process 200 may update a record of the number of times the content creator has uploaded content items that violate a policy associated with the media content platform. Note that in some embodiments, as described above in connection with block 220, process 200 may use information associated with the content creator's account to determine a penalty for the content creator.

[0100] At 224, process 200 can block the upload of the content item to the media content platform. Note that in instances where the content item is a live stream of content, process 200 can stop or prevent the live stream from streaming to any user devices.

[0101] It should be noted that, while processes 100 and 200 are described above as relating to reviewing content items (e.g., videos, live-streamed video content, and / or any other suitable content items), in some embodiments, processes 100 and 200 may be used to review any other suitable type of content, such as comments in a comment section, comments on a social networking post, messages in a live chat, and / or any other suitable comments. For example, in some embodiments, process 200 may determine whether a particular comment should be added to a queue of comments to be reviewed (e.g., at blocks 208 and 214) based on a classifier applied to a group of comments. As a more specific example, in some embodiments, process 200 may add a comment to a queue of comments to be manually reviewed in response to determining, using any suitable machine learning classifier, that the comment is likely to contain one or more types of objectionable content (e.g., hate speech, offensive language, offensive images or links, and / or any other suitable type of objectionable content). Continuing with this example, in some embodiments, process 200 may add the comment to a queue of comments to be reviewed (e.g., at block 214) and proceed through blocks of process 200 to determine whether the comment should be removed from the live chat or comment section in which it was posted based on manual review. As another example, in some embodiments, as described above in connection with block 202, processes 100 and 200 may be used to review benefits offered by content creators in connection with the purchase of a membership.As a more specific example, in some embodiments, process 200 may determine whether a particular offered benefit should be added to a queue of benefits to be reviewed (e.g., at blocks 208 and 214) based on any suitable information, such as whether complaints have been received from viewers who purchased memberships, whether more than a predetermined threshold of viewers have canceled their purchased memberships, and / or based on any other suitable criteria. Continuing with the example, process 200 may then proceed through the blocks of process 200 to determine whether the benefit offer should be canceled or discontinued based on manual review.

[0102] 3, an illustrative example of hardware for identifying inappropriate content items 300 is shown that may be used in accordance with some embodiments of the disclosed subject matter. As shown, the hardware 300 may include a server 302, a communications network 304, and / or one or more user devices 306, such as a content creator device 308 and / or a reviewer device 310.

[0103] Server 302 may be any suitable server for storing information, data, media content, and / or any other suitable type of content. For example, in some embodiments, server 302 may store user-generated media content uploaded by one or more content creator devices 308. As a more specific example, in some embodiments, the user-generated media content may include videos, movies, photos, slideshows, animations, documents, and / or any other suitable type of user-generated media content. As another example, in some embodiments, server 302 may store information about one or more content creators who uploaded user-generated content, such as whether a particular content creator has previously uploaded content that violates a particular policy of the media content platform, and / or any other suitable information. In some embodiments, server 302 may implement any suitable process for queuing uploaded content items for review, storing decisions regarding the uploaded content items (e.g., a decision indicating whether the content item contains objectionable content and / or any other suitable decision), and / or performing any other suitable function. For example, in some embodiments, server 302 may perform any suitable blocks of process 200 as shown and described above in connection with FIG.

[0104] The communications network 304, in some embodiments, may be any suitable combination of one or more wired and / or wireless networks. For example, the communications network 304 may include any one or more of the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communications network. The user device 306 may be connected by one or more communications links (e.g., communications link 312) to the communications network 304, which may be linked to the server 302 via one or more communications links (e.g., communications link 314). The communications link may be any communications link suitable for communicating data between the user device 306 and the server 302, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communications link, or any suitable combination of such links.

[0105] The user device 306 may include any one or more user devices. For example, in some embodiments, the user device 306 may include a content creator device 308. In some embodiments, the content creator device 308 may be any user device suitable for capturing user-generated content, uploading user-generated content, receiving notifications indicating judgments regarding uploaded content items, and / or performing any other suitable function. As another example, in some embodiments, the user device 306 may include a reviewer device 310. In some embodiments, the reviewer device 310 may be any user device suitable for viewing uploaded content items, submitting judgments regarding uploaded content items, and / or performing any other suitable function. In some embodiments, the user device 306 may include any suitable type of user device. For example, in some embodiments, the user device 306 may include a mobile phone, a wearable computer, a tablet computer, a desktop computer, a laptop computer, a vehicle entertainment system, a game console, a television, and / or any other suitable user device.

[0106] Although server 302 is shown as one device, in some embodiments the functions performed by server 302 may be performed using any suitable number of devices. For example, in some embodiments, multiple devices may be used to implement the functions performed by server 302.

[0107] To avoid overcomplicating the diagram, two user devices 308 and 310 are shown in FIG. 3, although in some embodiments, any suitable number and / or type of user devices may be used.

[0108] In some embodiments, the server 302 and the user device 306 may be implemented using any suitable hardware. For example, in some embodiments, the devices 302 and 306 may be implemented using any suitable general-purpose or special-purpose computer. For example, a mobile phone may be implemented using a special-purpose computer. Any such general-purpose or special-purpose computer may include any suitable hardware. For example, as shown in the example hardware 400 of FIG. 4 , such hardware may include a hardware processor 402, memory and / or storage 404, an input device controller 406, input devices 408, a display / audio driver 410, display and audio output circuitry 412, a communication interface 414, an antenna 416, and a bus 418.

[0109] The hardware processor 402, in some embodiments, may include any suitable hardware processor, such as a microprocessor, a microcontroller, a digital signal processor, special purpose logic circuitry, and / or any other suitable circuitry for controlling the functions of a general purpose or special purpose computer. In some embodiments, the hardware processor 402 may be controlled by a server program stored in memory and / or storage of a server, such as the server 302. In some embodiments, the hardware processor 402 may be controlled by a computer program stored in memory and / or storage 404 of the user device 306.

[0110] Memory and / or storage 404, in some embodiments, may be any suitable memory and / or storage for storing programs, data, and / or any other suitable information. For example, memory and / or storage 404 may include random access memory, read-only memory, flash memory, hard disk storage, optical media, and / or any other suitable memory.

[0111] Input device controller 406, in some embodiments, may be any suitable circuitry for controlling and receiving input from one or more input devices 408. For example, input device controller 406 may be circuitry for receiving input from a touchscreen, from a keyboard, from one or more buttons, from a voice recognition circuit, from a microphone, from a camera, from an optical sensor, from an accelerometer, from a temperature sensor, from a near-field sensor, from a pressure sensor, from an encoder, and / or from any other type of input device.

[0112] Display / audio driver 410, in some embodiments, may be any suitable circuitry for controlling and driving output to one or more display / audio output devices 412. For example, display / audio driver 410 may be circuitry for driving a touch screen, a flat panel display, a cathode ray tube display, a projector, one or more speakers, and / or any other suitable display and / or presentation device.

[0113] The communication interface 414 may be any suitable circuitry for interfacing with one or more communication networks (e.g., the computer network 304). For example, the interface 414 may include network interface card circuitry, wireless communication circuitry, and / or any other suitable type of communication network circuitry.

[0114] The antenna 416, in some embodiments, may be any suitable antenna or antennas for wirelessly communicating with a communication network (e.g., the communication network 304). In some embodiments, the antenna 416 may be omitted.

[0115] The bus 418 may be any suitable mechanism for communicating between two or more components 402, 404, 406, 410, and 414 in some embodiments.

[0116] According to some embodiments, any other suitable components may be included in hardware 400.

[0117] In some embodiments, at least some of the above-described blocks of the processes of Figures 1 and 2 may be executed or performed in any order or permutation, including but not limited to the order and permutation shown in and described with respect to the figures. Also, some of the above-described blocks of Figures 1 and 2 may be executed or performed substantially simultaneously, where appropriate, or in parallel to reduce latency and processing time. Additionally, or alternatively, some of the above-described blocks of the processes of Figures 1 and 2 may be omitted.

[0118] In some embodiments, any suitable computer-readable medium may be used to store instructions for performing the functions and / or processes herein. For example, in some embodiments, the computer-readable medium may be transitory or non-transitory. For example, a non-transitory computer-readable medium may include a medium such as a magnetic medium in non-transitory form (such as a hard disk, a floppy disk, and / or any other suitable magnetic medium), an optical medium in non-transitory form (such as a compact disc, a digital video disc, a Blu-ray disc, and / or any other suitable optical medium), a semiconductor medium in non-transitory form (such as a flash memory, an electrically programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and / or any other suitable semiconductor medium), any suitable medium that is not transient or does not lose apparent permanence during transmission, and / or any suitable tangible medium. As another example, a transitory computer-readable medium may include signals on a network, in wires, in conductors, in optical fibers, in circuits, in any suitable medium that is transient and loses any apparent permanence during transmission, and / or in any suitable non-tangible medium.

[0119] Accordingly, methods, systems, and media are provided for identifying inappropriate content items.

[0120] While the present invention has been described and shown in the foregoing illustrative embodiments, it will be understood that the present disclosure is made by way of example only, and that numerous changes in the details of the implementation of the invention may be made without departing from the spirit and scope of the invention, which is limited only by the following claims. The features of the disclosed embodiments may be combined and rearranged in various ways. [Explanation of symbols]

[0121] 100 processes 200 processes 302 Server 304 Communication Network 306 User Devices 308 Content Creator Devices 310 Examiner Device 312 Communication Links 400 Hardware 402 Hardware Processor 404 Memory and / or Storage 406 Input Device Controller 408 Input Devices 410 Display / Audio Driver 412 Display and audio output circuits, display / audio output devices 414 Communication Interface 416 Antenna 418 Bus

Claims

1. 1. A computer-implemented method for identifying inappropriate content items, comprising: receiving, at a server from a user device associated with a user, a request to provide user-generated content on a media content platform; In response to receiving the request, identifying second user-generated content similar to the user-generated content, the second user-generated content having been previously reviewed; determining a similarity of the user-generated content to the second user-generated content; determining whether the user-generated content should be reviewed by one or more review users by comparing at least the similarity to a similarity threshold indicated by one or more avoidance criteria; adding a review request to a queue of a review user in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, the review request including the user-generated content and information associated with a user account corresponding to the user; determining whether a determination in response to the review request indicates that the user-generated content should be made available on the media content platform; In response to determining that the determination indicates that the user-generated content violates at least one policy associated with the media content platform, sending a first message to the user of the user device indicating the at least one violated policy and a penalty to be imposed on the user account corresponding to the user; causing a corresponding action to be taken based on the penalty; A method comprising:

2. The method of claim 1, wherein determining whether the user-generated content should be reviewed is further based on determining that the second user-generated content has been deleted from the media content platform.

3. The method described in claim 2, wherein determining that the user-generated content should be reviewed based on the one or more avoidance criteria includes determining that the similarity between the user-generated content and the second user-generated content is lower than the similarity indicated by the similarity threshold.

4. In response to determining that the determination indicates that the user-generated content violates at least one policy associated with the media content platform, determining that the user has violated one or more policies more than a predetermined threshold number of times, wherein the penalty is determined further based on objectionable content types contained in other user-generated content previously provided by the user and determined to violate the one or more policies. The method of claim 1 further comprising: determining whether the user-generated content should be reviewed by one or more review users by comparing at least the similarity to a threshold similarity indicated by the one or more avoidance criteria, 2. The method of claim 1, further comprising determining whether the second user-generated content is associated with a particular objectionable content type that is blocked from being provided in connection with a particular feature currently enabled for the user account associated with the user.

6. The particular objectionable content type includes clickbait content, The particular feature allows the user to collect revenue; The method of claim 5 , wherein the penalty includes preventing the user account of the user from using the particular feature.

7. The second user-generated content: determining that the second user-generated content was previously uploaded to the media content platform in association with the user account of the user; and determining whether the second user-generated content passed or failed a previous content review; The method of claim 1, wherein the method is determined based on the following:

8. The method of claim 7, further comprising: receiving, at the server, from the user device associated with the user, a subsequent request for providing a third user-generated content on the media content platform; identifying the user-generated content that violates the at least one policy as similar to the third user-generated content; determining that the third user-generated content contains the same objectionable content type as the user-generated content that violated the at least one policy; determining that the third user-generated content should be reviewed based on The method of claim 1 further comprising: determining whether a determination responsive to review of the third user-generated content indicates that the third user-generated content does not violate any policy; In response to determining that the third user-generated content contains the same objectionable content type but does not violate any policy, causing the third user-generated content to be made available to a plurality of other users of the media content platform for display on a content page associated with the user account of the user; preventing viewer interaction from being included in the content page containing the third user-generated content; 9. The method of claim 8, further comprising:

10. The user is a viewer, The method of claim 1 , wherein the request to provide the user-generated content includes a request to use a content feature that enables interaction with a content production.

11. Adjusting the similarity threshold based on an amount of one or more cues associated with the one or more review users. The method of claim 1 further comprising:

12. The user is a content creator, The method of claim 1 , wherein the request to provide user-generated content comprises a request to use a content feature to create user-generated content.

13. The method of claim 1, wherein the second user-generated content is identified as being similar to the user-generated content based on comparing an audio footprint associated with the user-generated content with the second user-generated content.

14. The user is a content creator, The method of claim 1 , wherein the corresponding action includes preventing viewers of the user-generated content from using content features on the media content platform page that includes the user-generated content that enable them to interact with the content creator.

15. The user is a viewer, the request to provide user-generated content on the media content platform is a request to provide a comment on a page of the media content platform that includes the second user-generated content; The method of claim 1 , wherein the second user-generated content corresponds to content targeted to children.

16. The method of claim 15, wherein the request to provide user-generated content on the media content platform includes a request to use a particular content feature to generate the user-generated content; The method of claim 1 , wherein the second user-generated content is identified as being similar to the user-generated content based on being created using the particular content feature.

17. The method described in claim 16, wherein the second user-generated content is identified as similar to the user-generated content further based on being created by the user of the user account using the particular content feature.

18. The method of claim 17, wherein the request to provide user-generated content on the media content platform includes a request to use a particular content feature to generate the user-generated content; 2. The method of claim 1, wherein the second user-generated content is identified as being similar to the user-generated content based on the information associated with the user account that made the request to generate the user-generated content using the particular content feature.

19. The second user-generated content is created by the user; 20. The method of claim 18, wherein the information associated with the user account includes a number of viewers who left negative reviews related to the second user-generated content.

20. A system for identifying inappropriate content, comprising: a server having a memory; Hardware processor and wherein the hardware processor comprises: receiving, at a server from a user device associated with a user, a request to provide user-generated content on a media content platform; In response to receiving the request, identifying second user-generated content similar to the user-generated content, the second user-generated content having been previously reviewed; and determining a similarity of the user-generated content to the second user-generated content; determining whether the user-generated content should be reviewed by one or more review users by comparing at least the similarity to a similarity threshold indicated by one or more avoidance criteria; adding a review request to a queue of a review user in response to determining that the user-generated content should be reviewed based on the one or more avoidance criteria, the review request including the user-generated content and information associated with a user account corresponding to the user; determining whether a determination in response to the review request indicates that the user-generated content should be made available on the media content platform; In response to determining that the determination indicates that the user-generated content violates at least one policy associated with the media content platform, sending a first message to the user of the user device indicating the at least one violated policy and a penalty to be imposed on the user account corresponding to the user; causing a corresponding action to be taken based on said penalty; Including, the system.

Images