Content review method and electronic device and computer-readable storage medium

Abstract

The application discloses a content review method, an electronic device and a computer readable storage medium. The method comprises the following steps: identifying a risk label of content that needs to be reviewed; determining a security policy of different risk fields; analyzing the risk label based on the security policy of different risk fields to determine a risk level of the content in different risk fields; determining target content that needs manual review in different risk fields according to the risk level of the content in different risk fields; obtaining a manual review result of the target content, and calculating a proportion of illegal content in the target content corresponding to different risk fields as a review risk proportion of different risk fields; and adjusting a threshold parameter in the security policy of the corresponding risk field based on the review risk proportion of different risk fields. The application improves the flexibility of the policy in the content review process.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to content moderation methods, electronic devices, and computer-readable storage media. Background Technology

[0002] As users publish more and more UGC (User Generated Content) on internet platforms, a large amount of illegal, non-compliant, and unhealthy content is included. Therefore, it is necessary to review UGC to prevent illegal content from being displayed and spread on internet platforms.

[0003] In related technologies, UGC review includes both machine review and human review. Machine review involves preliminary review of UGC based on preset security policies, while human review involves assigning UGC to reviewers for further review based on preset task allocation policies. However, in the above solutions, the security policies for machine review and the task allocation policies for human review lack the ability to be quickly and flexibly adjusted, resulting in poor policy flexibility.

[0004] Therefore, how to improve the flexibility of strategies in the content review process is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a content moderation method, an electronic device, and a computer-readable storage medium that improve the flexibility of strategies during the content moderation process.

[0006] To achieve the above objectives, the first aspect of this application provides a content moderation method, comprising:

[0007] Identify risk labels for content that requires review;

[0008] Determine security strategies for different risk areas;

[0009] The risk labels are analyzed based on security strategies for different risk areas to determine the target content that requires manual review in different risk areas.

[0010] Obtain the results of manual review of the target content, and calculate the proportion of non-compliant content in the target content corresponding to different risk areas as the review risk ratio of different risk areas;

[0011] Adjust the threshold parameters in the security strategy for the corresponding risk area based on the review risk ratio of different risk areas.

[0012] The risk labels used to identify content requiring review include:

[0013] Determine the media type of the content that needs to be reviewed, and use the identification algorithm corresponding to the media type to identify the risk label of the content.

[0014] The risk labels used to identify content requiring review include:

[0015] Identify the probability that the content to be reviewed contains different risky content, popularity information, and information about the content publisher, or any combination of these factors.

[0016] The determination of security strategies for different risk areas includes:

[0017] Identify the application scenarios for the content that needs to be reviewed, and determine the security strategies for different risk areas corresponding to the application scenarios.

[0018] The analysis of risk labels based on security strategies for different risk areas to determine the target content requiring manual review in different risk areas includes:

[0019] The risk tags are analyzed based on security strategies for different risk areas to determine the risk level of the content in different risk areas;

[0020] Based on the risk level of the content in different risk areas, the target content that requires manual review in different risk areas is determined.

[0021] After determining the risk level of the content in different risk areas, the method further includes:

[0022] The content is processed using the corresponding handling command for the risk level.

[0023] The threshold parameter in the security strategy for adjusting the review risk ratio of different risk areas includes:

[0024] If the risk ratio of the review is greater than the first preset value, the threshold parameter in the security strategy of the corresponding risk area is adjusted to increase the amount of target content in the corresponding risk area.

[0025] If the risk ratio of the review is less than the second preset value, the threshold parameter in the security policy of the corresponding risk area is adjusted to reduce the amount of target content in the corresponding risk area.

[0026] Wherein, the first preset value is greater than the second preset value.

[0027] The adjustment of the threshold parameters in the security policy for the corresponding risk area to increase the amount of target content in the corresponding risk area includes:

[0028] Adjust the threshold parameters in the security policy of the corresponding risk area in the target direction, and re-enter the step of analyzing the risk label based on the security policy of different risk areas to obtain the updated review risk ratio;

[0029] If the updated review risk ratio is less than the original review risk ratio, then the threshold parameter in the security policy of the corresponding risk area will continue to be adjusted in the direction of the target.

[0030] If the updated review risk ratio is greater than the previous review risk ratio, then the threshold parameter in the security policy of the corresponding risk area is adjusted in the opposite direction to the target direction.

[0031] Accordingly, adjusting the threshold parameters in the security policy for the corresponding risk area to reduce the amount of target content in the corresponding risk area includes:

[0032] Adjust the threshold parameters in the security policy of the corresponding risk area in the target direction, and re-enter the step of analyzing the risk label based on the security policy of different risk areas to obtain the updated review risk ratio;

[0033] If the updated review risk ratio is greater than the previous review risk ratio, then the threshold parameter in the security policy of the corresponding risk area will continue to be adjusted in the direction of the target.

[0034] If the updated review risk ratio is less than the original review risk ratio, then the threshold parameter in the security policy of the corresponding risk area is adjusted in the opposite direction to the target direction.

[0035] To achieve the above objectives, a second aspect of this application provides an electronic device, comprising:

[0036] Memory, used to store computer programs;

[0037] A processor for executing the computer program to implement the steps of the content review method described above.

[0038] To achieve the above objectives, a third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the content review method described above.

[0039] As can be seen from the above scheme, the content review method provided in this application includes: identifying risk tags of content that needs to be reviewed; determining security policies for different risk areas; analyzing the risk tags based on the security policies for different risk areas to determine the risk level of the content in different risk areas; determining the target content that needs to be manually reviewed in different risk areas based on the risk level of the content in different risk areas; obtaining the manual review results of the target content and calculating the proportion of non-compliant content in the target content corresponding to different risk areas as the review risk ratio for different risk areas; and adjusting the threshold parameters in the security policies of the corresponding risk areas based on the review risk ratios for different risk areas.

[0040] The content moderation method provided in this application assesses the effectiveness of security strategies by evaluating the proportion of non-compliant content within the target content requiring manual review. Based on the review results, the security strategy is automatically adjusted and continuously optimized, improving its flexibility and robustness. Furthermore, optimizing the security strategy can further optimize the target content requiring manual review, increasing the efficiency of manual review while reducing the rate of missed reviews of non-compliant content. This application also discloses an electronic device and a computer-readable storage medium that achieve the same technical effects.

[0041] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings are used to provide a further understanding of this disclosure and constitute a part of the specification. They are used together with the following detailed description to explain this disclosure, but do not constitute a limitation of this disclosure. In the drawings:

[0043] Figure 1 A flowchart illustrating a content moderation method provided in this application embodiment;

[0044] Figure 2 A flowchart illustrating another content moderation method provided in this application embodiment;

[0045] Figure 3 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0047] This application discloses a content moderation method that improves the flexibility of strategies during the content moderation process.

[0048] See Figure 1 The flowchart of a content moderation method provided in this application embodiment is as follows: Figure 1 As shown, it includes:

[0049] S101: Identify risk labels for content that requires review;

[0050] The purpose of this embodiment is to review UGC. It is understood that UGC published by users on Internet platforms usually integrates multiple media types, such as images, text, audio, and video. Therefore, after obtaining the content that needs to be reviewed, it can be broken down into a single media type to facilitate subsequent review.

[0051] In this step, risk tags are generated for the content requiring review. Risk tags can be understood as quantifiable descriptive attributes of the content. As a feasible implementation, this step may include: determining the media type of the content to be reviewed, and using a recognition algorithm corresponding to that media type to identify the risk tags of the content. In specific implementations, different media types are identified using recognition algorithms corresponding to different media types. These recognition algorithms may include AI (Artificial Intelligence) algorithms, etc. Risk tags may include: the probability of containing different risky content, popularity information, content publisher information, etc. Risky content includes, for example, pornography, black market content, violent content, nudity, sensitive words, sensitive figures, etc. Content publisher information includes, for example, the publisher's account level, credit score, etc. Popularity information includes, for example, click-through rate, number of impressions, number of reposts, etc. For example, if the content to be reviewed is an image, then the image recognition algorithm is used to identify it, obtaining the image's tags: probability of containing pornography, probability of containing sensitive figures, proportion of nudity, popularity information, and content publisher information.

[0052] S102: Determine security strategies for different risk areas;

[0053] In this step, the security policy for machine review is determined. The security policy can be understood as a set of rules, where each rule is a combination of multiple risk tags, such as: the probability of containing sensitive individuals is greater than 0.5 and the proportion of exposed skin is greater than 0.3%. The security policy is used to review whether UGC content violates regulations. Different risk areas correspond to different security policies, which may include areas involving pornography, violence, or organized crime.

[0054] In one preferred implementation, this step may include: determining the application scenarios of the content to be reviewed, and determining security strategies for different risk areas corresponding to the application scenarios. In specific implementations, the application scenarios of the content to be reviewed are determined, such as games, animation, emojis, and reality shows. Furthermore, different security strategies are set according to different application scenarios, with different security strategies corresponding to different review standards, thereby providing different review standards for different application scenarios and improving the efficiency of content review.

[0055] S103: Analyze the risk labels based on the security strategies for different risk areas to determine the target content that needs to be manually reviewed in different risk areas;

[0056] In this step, the risk tags of the content are reviewed by machine based on security policies for different risk areas to determine the risk level of the content in different risk areas, which can include high risk, medium risk, and low risk. Based on the risk level, the target content requiring manual review in different areas is determined; for example, medium-risk content is identified as the target content requiring manual review. Furthermore, the target content in different areas is sent to reviewers in the corresponding areas for manual review.

[0057] As a feasible implementation method, after determining the risk level of the content in different risk areas, the method further includes processing the content using the corresponding handling command for each risk level. In specific implementation, different processing strategies are adopted for content with different risk levels. For example, high-risk content is deleted, medium-risk content is identified as target content requiring manual review, and low-risk content is published normally. That is, in the machine review stage, the content requiring review can be divided into three categories: low-risk content can be published normally, medium-risk content requires manual review, and high-risk content needs to be deleted immediately.

[0058] Additionally, content can be categorized into different risk levels based on risk labels across various sectors. During manual review, content in higher-risk queues can be prioritized. For example, a high-risk queue for pornography-related content would have a probability of >0.5 for pornographic content or >0.8 for vulgar content, with a nudity percentage >0.3. A low-risk queue for pornography-related content would have a probability of <0.5 for pornographic content and <0.8 for vulgar content.

[0059] S104: Obtain the manual review results of the target content, and calculate the proportion of non-compliant content in the target content corresponding to different risk areas as the review risk ratio of different risk areas;

[0060] In this step, the manual review results of target content in different risk areas are obtained, including both non-violation and compliant content. The proportion of non-violation content in the target content of each risk area is calculated as the review risk ratio for each risk area. The review risk ratio can be used to evaluate the review effectiveness of the security strategy during machine review. The higher the review risk ratio, the better the review effectiveness of the security strategy during machine review.

[0061] S105: Adjust the threshold parameters in the security strategy for the corresponding risk area based on the review risk ratio of different risk areas.

[0062] In this step, the threshold parameters in the security policy for the corresponding risk area are adjusted based on the review risk ratio. Adjusting the threshold parameters in the security policy consequently adjusts the review risk ratio for the next stage. For risk areas with good review performance, the corresponding security policy can be relaxed, allowing more target content to be recalled in the next stage. For risk areas with poor review performance, the corresponding security policy can be relaxed, resulting in fewer target content being recalled in the next stage. This reduces the workload of manual review, increases the review risk ratio, and allows reviewers to focus their efforts on the most valuable areas, improving review efficiency and reducing the rate of missed reviews of non-compliant content.

[0063] In practice, if the review risk ratio is greater than the first preset value, the security policy for the corresponding risk area is relaxed, that is, the threshold parameters in the security policy for the corresponding risk area are adjusted to increase the amount of target content in the corresponding risk area. If the review risk ratio is less than the second preset value, the security policy for the corresponding risk area is tightened, and the threshold parameters in the security policy for the corresponding risk area are adjusted to reduce the amount of target content in the corresponding risk area.

[0064] For example, a review team reviews a maximum of 10,000 images per day. Under the current security policy, 5,000 images related to pornography and 5,000 images related to violence have been recalled. This means the target content for pornography and violence each contains 5,000 images. After reviewing 1,000 pornographic and 1,000 violent images, the team identifies 400 pornographic images and 50 violent images, resulting in a 40% risk of violation for pornography and a 5% risk of violation for violence. Without adjusting the security policy, after reviewing all data, 250 violent images and 2,000 pornographic images, totaling 2,250 violations, would be identified. If the first preset value is 30% and the second preset value is 8%, the security policy for pornography needs to be relaxed. The number of recalled pornographic images increases from 5,000 to 7,000, the risk of violation decreases from 40% to 30%, and the number of pornographic images increases from 2,000 to 2,100. Meanwhile, security strategies in the violent content sector were tightened, with the number of recalled pornographic images increasing from 5,000 to 3,000, the risk review rate rising from 4% to 8%, and the number of violent images increasing from 250 to 240. Through adaptive adjustments to the security strategy, while maintaining the total number of images reviewed at 10,000, the number of reviewed violation images increased from 2,250 to 2,340.

[0065] When adjusting threshold parameters in a security policy, methods such as reinforcement learning, policy parameter grid search, and evolutionary learning can be used to determine the optimal combination of threshold parameters. For reinforcement learning, adjusting the threshold parameters in the security policy for a corresponding risk domain to increase the amount of target content in that domain includes: adjusting the threshold parameters in the security policy for the corresponding risk domain in the target direction, and re-entering the step of analyzing the risk labels based on security policies for different risk domains to obtain an updated review risk ratio; if the updated review risk ratio is less than the previous review risk ratio, then continue adjusting the threshold parameters in the security policy for the corresponding risk domain in the target direction; if the updated review risk ratio is greater than the previous review risk ratio, then adjust the threshold parameters in the security policy for the corresponding risk domain in the opposite direction to the target direction.

[0066] When relaxing security policies, for each threshold parameter in the security policy, the adjustment is first made in the direction of the target, such as increasing or decreasing it. Based on the updated security policy, the target content that needs to be manually reviewed is re-determined, and then the review risk ratio is recalculated. If the review risk ratio decreases, the adjustment continues in the direction of the target; if the review risk ratio increases, the adjustment is made in the opposite direction of the target.

[0067] Accordingly, adjusting the threshold parameters in the security policy of the corresponding risk domain to reduce the amount of target content in the corresponding risk domain includes: adjusting the threshold parameters in the security policy of the corresponding risk domain in the target direction, and re-entering the step of analyzing the risk tags based on the security policies of different risk domains to obtain an updated review risk ratio; if the updated review risk ratio is greater than the previous review risk ratio, then continue to adjust the threshold parameters in the security policy of the corresponding risk domain in the target direction; if the updated review risk ratio is less than the previous review risk ratio, then adjust the threshold parameters in the security policy of the corresponding risk domain in the opposite direction to the target direction.

[0068] When tightening security policies, for each threshold parameter in the security policy, the adjustment is first made in the direction of the target, such as increasing or decreasing it. Based on the updated security policy, the target content that needs to be manually reviewed is re-determined, and then the review risk ratio is recalculated. If the review risk ratio increases, the adjustment continues in the direction of the target; if the review risk ratio decreases, the adjustment is made in the opposite direction of the target.

[0069] The content review method provided in this application evaluates the effectiveness of security strategies by assessing the proportion of non-compliant content in the target content requiring manual review. Based on the review results, the security strategy is automatically adjusted and continuously optimized, improving its flexibility and robustness. Furthermore, optimizing the security strategy can further optimize the target content requiring manual review, increasing the efficiency of manual review while reducing the rate of missed reviews of non-compliant content.

[0070] The following describes application scenario embodiments provided in this application, such as... Figure 2 As shown, the review process includes: parsing UGC content and integrating it into the review system; then calling a common machine algorithm for tagging to obtain the algorithm's descriptive tags for the content; next, selecting appropriate machine review and human review security strategies and executing them on the strategy engine to obtain the risk level of the content to be reviewed; content suspected of being risky will enter the human review system, and reviewers will verify and confirm the results; after a certain amount of human review data has been accumulated, the effectiveness of each strategy can be evaluated; finally, the configuration of the security strategy is dynamically adjusted in real time based on the evaluation effect of the strategy.

[0071] Review and Approval Module: User-generated content (UGC) published on internet platforms often combines multiple media formats. Before being integrated into the review process, it is first broken down into individual media types such as images, text, audio, and video, and then reviewed simultaneously. At the same time, the business scenario is analyzed to select appropriate security strategies.

[0072] Algorithm-based review and tagging: Artificial intelligence algorithms are used to identify and tag content such as images, text, audio, and video. Tags may include the probability of pornography, the probability of violation, whether it contains sensitive words, whether it contains sensitive figures, and auxiliary information such as the account level, credit score, and popularity of UGC content authors.

[0073] The strategy engine includes machine-based security policies and human-based security policies. Machine-based security policies are pre-set strategies that categorize content to be reviewed into three types based on tags: normal, risky and requiring deletion, and suspected risky and requiring review. Human-based security policies further classify / group / queue content suspected of being risky and requiring review, setting different queues based on risk level to facilitate reviewers prioritizing high-risk content.

[0074] The strategy engine is essentially an expert system that transforms algorithmic labels into actionable instructions. The transformation rules vary across different business scenarios, necessitating the maintenance of a large number of rules and the selection of appropriate rules based on the scenario. This requires dynamically adjusting the rule configuration. The strategy engine comprises two parts: pre-defined security policies and a dynamic rule calculation engine.

[0075] The strategy engine takes tags as input to the content to be reviewed and outputs the risk area and risk level of the content. For example: high risk of violence, low risk of violence, high risk of pornography, low risk of pornography. Based on the risk level, it can be converted into different handling instructions, such as: normal publication, deletion, pending review, etc.

[0076] Dynamic rule calculation engines can be Gengine, govaluate, or gorule. Gengine is a rule engine implemented based on AST (Abstract Syntax Tree) and the Golang language. It can dynamically load and configure rules without interrupting service and can perform dynamic calculations without compilation. It can calculate the risk queue to which the content to be reviewed belongs based on algorithm tags and security policies.

[0077] When using it, you only need to configure various security policies. The whole process does not require coding, which greatly improves the efficiency of policy formulation and reduces R&D manpower costs. Moreover, the policies are deployed in real time, and the effects can be viewed and the configuration adjusted in a timely manner, which greatly improves the efficiency of responding to risky content and reduces the exposure rate of risky content.

[0078] The human review module is a content review tool that helps reviewers view content awaiting review, determine whether it violates regulations, and prioritize the review of content in the high-risk queue.

[0079] The quality control module monitors the quality of reviewers' work, promptly identifying issues that arise during the review process. It also assesses the reviewers' proficiency and areas of expertise. Currently, there are numerous types of content violations, as well as a multitude of review standards and regulations. A single reviewer often only has expertise in a few areas. For example, some reviewers are proficient in pornography detection standards, some in violent content review regulations, and some are familiar with black market content violations, etc. In practice, it is often necessary to assign review content based on the reviewers' areas of expertise.

[0080] Strategy Evaluation Module: Based on the human review results, calculates the proportion of risky content for various strategies in real time. The higher the proportion of risky content, the better the strategy's effectiveness.

[0081] Adaptive adjustment strategy: Based on strategy evaluation, security strategies are adaptively adjusted. For strategies with good results, the policy conditions are relaxed to increase recall; for strategies with poor results, the policy conditions are tightened to reduce the amount of review. Through this automatic adjustment, reviewers can focus their energy on the most valuable areas, improve review efficiency, and thus reduce the rate of loss of risky content.

[0082] The policy engine can statistically evaluate the effectiveness of policies in real time, supports manual adjustments to security policies to improve effectiveness, and can adaptively adjust policy configurations to improve the efficiency of auditors. In short, whether adjusted automatically or manually, the entire audit system can detect more violations and reduce the risk of business content violations.

[0083] The entire strategy formulation and deployment process is highly automated and intelligent, with centralized storage for easy reuse and management. It solves problems such as fragmented and scattered security policies across different scenarios and difficulties in document maintenance, thereby saving R&D manpower and security policy engineer manpower, and reducing review costs.

[0084] The following describes a content moderation device provided in an embodiment of this application. The content moderation device described below and the content moderation method described above can be referred to each other.

[0085] The content moderation device provided in this application embodiment includes:

[0086] The identification module is used to identify risk labels for content that needs to be reviewed.

[0087] The determination module is used to determine security strategies for different risk areas;

[0088] The analysis module is used to analyze the risk tags based on security strategies in different risk areas to determine the target content that needs to be manually reviewed in different risk areas.

[0089] The calculation module is used to obtain the manual review results of the target content and calculate the proportion of non-compliant content in the target content corresponding to different risk areas as the review risk ratio of different risk areas;

[0090] The adjustment module is used to adjust the threshold parameters in the security policy for different risk areas based on the review risk ratio of different risk areas.

[0091] The content review device provided in this application evaluates the effectiveness of security strategies by assessing the proportion of non-compliant content in target content requiring manual review. Based on the review results, it automatically adjusts and continuously optimizes the security strategies, thereby improving their flexibility and robustness. Furthermore, optimizing the security strategies can further optimize the target content requiring manual review, increasing the efficiency of manual review while reducing the rate of missed reviews of non-compliant content.

[0092] Based on the above embodiments, as a preferred implementation, the identification module is specifically used to: determine the media type of the content to be reviewed, and use the identification algorithm corresponding to the media type to identify the risk label of the content.

[0093] Based on the above embodiments, as a preferred implementation, the identification module is specifically used to: identify any one or a combination of any of the following: the probability that the content to be reviewed contains different risky content, popularity information, and content publisher information.

[0094] Based on the above embodiments, as a preferred implementation, the determining module is specifically used to: determine the application scenario of the content that needs to be reviewed, and determine the security strategies for different risk areas corresponding to the application scenario.

[0095] Based on the above embodiments, as a preferred implementation, the analysis module is specifically used to: analyze the risk tags based on security strategies in different risk areas to determine the risk level of the content in different risk areas; and determine the target content that needs to be manually reviewed in different risk areas according to the risk level of the content in different risk areas.

[0096] Based on the above embodiments, as a preferred embodiment, it further includes:

[0097] The processing module is used to process the content using the disposal command corresponding to the risk level.

[0098] Based on the above embodiments, as a preferred embodiment, the adjustment module includes:

[0099] The first adjustment unit is used to adjust the threshold parameter in the security policy of the corresponding risk area when the review risk ratio is greater than the first preset value, so as to increase the amount of target content in the corresponding risk area.

[0100] The second adjustment unit is used to adjust the threshold parameter in the security policy of the corresponding risk area when the review risk ratio is less than the second preset value, so as to reduce the amount of target content in the corresponding risk area.

[0101] Wherein, the first preset value is greater than the second preset value.

[0102] Based on the above embodiments, as a preferred implementation, the first adjustment unit is specifically used to: adjust the threshold parameter in the security policy of the corresponding risk area in the target direction, and re-enter the step of analyzing the risk label based on the security policy of different risk areas to obtain an updated review risk ratio; if the updated review risk ratio is less than the previous review risk ratio, then continue to adjust the threshold parameter in the security policy of the corresponding risk area in the target direction; if the updated review risk ratio is greater than the previous review risk ratio, then adjust the threshold parameter in the security policy of the corresponding risk area in the opposite direction to the target direction.

[0103] Accordingly, the second adjustment unit is specifically used to: adjust the threshold parameters in the security policy of the corresponding risk area in the target direction, and re-enter the step of analyzing the risk label based on the security policies of different risk areas to obtain the updated review risk ratio; if the updated review risk ratio is greater than the previous review risk ratio, then continue to adjust the threshold parameters in the security policy of the corresponding risk area in the target direction; if the updated review risk ratio is less than the previous review risk ratio, then adjust the threshold parameters in the security policy of the corresponding risk area in the opposite direction to the target direction.

[0104] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0105] This application also provides an electronic device, see [link to document]. Figure 3 The present application provides a structural diagram of an electronic device 30, as shown in the embodiment. Figure 3 As shown, it may include a processor 31 and a memory 32.

[0106] The processor 31 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 31 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 31 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 31 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 31 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0107] The memory 32 may include one or more computer-readable storage media, which may be non-transitory. The memory 32 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 32 is used to store at least the following computer program 321, which, after being loaded and executed by the processor 31, is capable of implementing the relevant steps in the content moderation method executed by the electronic device side as disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 32 may also include an operating system 322 and data 323, etc., and the storage method may be temporary storage or permanent storage. The operating system 322 may include Windows, Unix, Linux, etc.

[0108] In some embodiments, the electronic device 30 may further include a display screen 33, an input / output interface 34, a communication interface 35, a sensor 36, a power supply 37, and a communication bus 38.

[0109] certainly, Figure 3 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than [other components]. Figure 3 More or fewer components as shown, or combinations of certain components.

[0110] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the content moderation method performed by the electronic device of any of the above embodiments.

[0111] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0112] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A content moderation method, characterized in that, include: Identify risk labels for content that needs to be reviewed; the content that needs to be reviewed includes any one or a combination of any combination of images, text, audio, and video. Determine security strategies for different risk areas; wherein, the security strategy includes multiple risk labels and corresponding threshold parameters; The risk labels are analyzed based on security strategies for different risk areas to determine the target content that requires manual review in different risk areas. Obtain the results of manual review of the target content, and calculate the proportion of non-compliant content in the target content corresponding to different risk areas as the review risk ratio of different risk areas; Adjust the threshold parameters in the security strategy for the corresponding risk area based on the review risk ratio of different risk areas; The threshold parameter in the security strategy for adjusting the review risk ratio of different risk areas includes: If the review risk ratio is greater than the first preset value, the security policy for the corresponding risk area is relaxed, and the threshold parameter in the security policy for the corresponding risk area is adjusted to increase the amount of target content in the corresponding risk area. If the review risk ratio is less than the second preset value, then tighten the security policy for the corresponding risk area and adjust the threshold parameter in the security policy for the corresponding risk area to reduce the amount of target content in the corresponding risk area. Wherein, the first preset value is greater than the second preset value; Specifically, when adjusting the threshold parameters in the security strategy for the corresponding risk domain, the optimal combination of threshold parameters is determined based on any one of the following methods: reinforcement learning, policy parameter grid search, or evolutionary learning.

2. The content review method according to claim 1, characterized in that, The risk labels used to identify content requiring review include: Determine the media type of the content that needs to be reviewed, and use the identification algorithm corresponding to the media type to identify the risk label of the content.

3. The content review method according to claim 1, characterized in that, The risk labels used to identify content requiring review include: Identify the probability that the content to be reviewed contains different risky content, popularity information, and information about the content publisher, or any combination of these factors.

4. The content review method according to claim 1, characterized in that, The determination of security strategies for different risk areas includes: Identify the application scenarios for the content that needs to be reviewed, and determine the security strategies for different risk areas corresponding to the application scenarios.

5. The content review method according to claim 1, characterized in that, The security strategies based on different risk areas analyze the risk tags to determine the target content that requires manual review in different risk areas, including: The risk tags are analyzed based on security strategies for different risk areas to determine the risk level of the content in different risk areas; Based on the risk level of the content in different risk areas, the target content that requires manual review in different risk areas is determined.

6. The content review method according to claim 5, characterized in that, After determining the risk level of the content in different risk areas, the process also includes: The content is processed using the corresponding handling command for the risk level.

7. The content review method according to claim 1, characterized in that, The adjustment of threshold parameters in the security policy for the corresponding risk area to increase the amount of target content in the corresponding risk area includes: Adjust the threshold parameters in the security policy of the corresponding risk area in the target direction, and re-enter the step of analyzing the risk label based on the security policy of different risk areas to obtain the updated review risk ratio; If the updated review risk ratio is less than the original review risk ratio, then the threshold parameter in the security policy of the corresponding risk area will continue to be adjusted in the direction of the target. If the updated review risk ratio is greater than the previous review risk ratio, then the threshold parameter in the security policy of the corresponding risk area is adjusted in the opposite direction to the target direction. Accordingly, adjusting the threshold parameters in the security policy for the corresponding risk area to reduce the amount of target content in the corresponding risk area includes: Adjust the threshold parameters in the security policy of the corresponding risk area in the target direction, and re-enter the step of analyzing the risk label based on the security policy of different risk areas to obtain the updated review risk ratio; If the updated review risk ratio is greater than the previous review risk ratio, then the threshold parameter in the security policy of the corresponding risk area will continue to be adjusted in the direction of the target. If the updated review risk ratio is less than the original review risk ratio, then the threshold parameter in the security policy of the corresponding risk area is adjusted in the opposite direction to the target direction.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the content moderation method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the content moderation method as described in any one of claims 1 to 7.

Images