Audit quality monitoring method, device and equipment

Abstract

The present invention provides an audit quality monitoring method, device, and equipment, the method comprising: displaying a page to be audited in response to an audit request for the page to be audited; collecting at least one type of operation behavior based on the operation on the page to be audited; comparing each type of operation behavior collected with the standard operation behavior of that type, and determining the weight coefficient corresponding to each type of operation behavior collected; multiplying the weight coefficient corresponding to each type of operation behavior collected with the score value of the standard operation behavior of that type and summing the results to obtain the audit quality score corresponding to the page to be audited. Utilizing the method provided by the present invention, by collecting the operation behavior on the page to be audited and calculating the corresponding weight coefficient to obtain the corresponding audit quality score, it is possible to monitor the audit quality.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to an audit quality monitoring method, device and equipment. Background Art

[0002] With the development of internet technology, media platforms have become a part of people's daily lives. Various media platforms display different page information to users, achieving efficient information dissemination. Media platform servers receive page information uploaded by users requesting display. This page information can be anything from advertisements to literary works. Taking advertising as an example, to prevent non-compliant advertisements from being displayed and disseminated, after the server receives the basic information and creative content uploaded by the advertiser, the media platform must review this information according to its review rules. Once page information that does not comply with review rules is delivered to the media platform, it can not only negatively impact media platform users but also pose significant security risks, such as requiring platform suspension and rectification, resulting in significant losses. Therefore, the quality of page information review is extremely important. Media platforms generally rely on human reviewers to review page information, and the quality of their review is crucial to the content security of both the page information delivery system and the media platform.

[0003] The audit scheme of related technologies can only evaluate the working time and efficiency of auditors by counting information such as the number of pages reviewed by auditors, the number of pages to be reviewed, and the time spent on auditing, but cannot monitor the quality of audits. Summary of the Invention

[0004] The embodiments of the present invention provide an audit quality monitoring method, device and equipment to solve the problem that the audit solutions in related technologies can only evaluate the working time and efficiency of auditors but cannot monitor the audit quality.

[0005] In a first aspect, an embodiment of the present invention provides an audit quality monitoring method, the method comprising:

[0006] In response to a review request for a page to be reviewed, displaying the page to be reviewed;

[0007] Collect at least one type of operation behavior based on the operation on the page to be reviewed;

[0008] Comparing each type of collected operation behavior with the standard operation behavior of that type, and determining a weight coefficient corresponding to each type of collected operation behavior;

[0009] The weight coefficient corresponding to each type of collected operation behavior is multiplied by the score value of the standard operation behavior of that type and the sum is calculated to obtain the audit quality score corresponding to the page to be audited.

[0010] In one possible implementation, the at least one type of operation behavior includes at least one of the following:

[0011] Statistics are collected for preset indicators related to operational behaviors, and the obtained indicators are statistical types of operational behaviors;

[0012] When a preset event is detected, the collected operational behavior of the event trigger type is detected.

[0013] The audit quality monitoring method provided by the embodiment of the present invention increases the comprehensiveness of the collected operational behaviors by specifically dividing the types of collected operational behaviors, thereby improving the rationality and reliability of the above-mentioned audit quality monitoring method in evaluating the audit quality.

[0014] In one possible implementation, the at least one type of operation behavior includes at least one of the following:

[0015] The time spent browsing the page to be reviewed;

[0016] The page browsing depth is determined based on the proportion of browsed pages among the pages to be reviewed;

[0017] A link click behavior triggered by clicking a link in the page to be reviewed;

[0018] Viewing resource viewing behavior triggered by image resources in the page to be reviewed;

[0019] Filling in the content to be filled in the page to be reviewed triggers the content filling behavior.

[0020] The audit quality monitoring method provided by the embodiment of the present invention improves the comprehensiveness of the collected operational behaviors by making specific settings for the collected operational behaviors, and improves the rationality and reliability of the audit quality evaluation by the above-mentioned audit quality monitoring method.

[0021] In a possible implementation, collecting at least one type of operation behavior includes at least one of the following steps:

[0022] Count the time from the start of displaying the page to be reviewed to the closing of the page to be reviewed, and obtain the page dwell time;

[0023] Obtaining the page browsing depth according to the ratio of the scroll bar height that the slider slides over to the total height of the scroll bar in the page to be reviewed when the page to be reviewed is closed;

[0024] When a click operation on a link-type information item is detected, it is determined that the link click behavior is collected, and the number of click operations on the preset mandatory links is counted;

[0025] When a viewing operation of an information item of an image resource type is detected, it is determined that the resource viewing behavior is collected, and the number of viewed information items of the image resource type is counted;

[0026] When monitoring the content input operation of the information item of the content type to be filled in with the audit status of failed, it is determined that the content filling behavior is collected, and the number of information items of the content type to be filled in with the audit status of failed that have been filled in and the number of information items of the content type to be filled in with the audit status of failed are counted.

[0027] The audit quality monitoring method provided by the embodiment of the present invention can reasonably collect operation behaviors by making specific settings for the steps of collecting operation behaviors, thereby improving the feasibility of the above-mentioned steps of collecting operation behaviors and improving the rationality, reliability and feasibility of the above-mentioned audit quality monitoring method.

[0028] In a possible implementation, counting the number of click operations on a preset mandatory link includes:

[0029] Record the identification information of the information item of the link type clicked;

[0030] When it is determined according to the identification information that the click operation is a click operation on a preset mandatory link and the same identification information has not been recorded before, the number of click operations on the preset mandatory link is increased by one.

[0031] The audit quality monitoring method provided by the embodiment of the present invention limits the specific process of collecting link click behaviors. By comparing whether the identification information of the information item of the clicked link type is recorded, it can avoid counting the same click behavior multiple times and accurately calculate the number of click operations on preset mandatory links, thereby improving the feasibility of the above-mentioned steps of collecting link click behaviors and improving the rationality, reliability and feasibility of the above-mentioned audit quality monitoring method.

[0032] In a possible implementation, comparing each type of collected operation behavior with a standard operation behavior of that type and determining a weight coefficient corresponding to each type of collected operation behavior includes:

[0033] Determine the weight coefficient corresponding to each type of collected operation behavior based on the difference between each type of collected operation behavior and the standard operation behavior of that type;

[0034] Among them, the greater the difference, the smaller the corresponding weight coefficient.

[0035] The audit quality monitoring method provided by the embodiment of the present invention determines the weight coefficient corresponding to each type of collected operation behavior by comparing the difference between each type of collected operation behavior and the standard operation behavior of that type, and specifically limits the impact of the difference on the weight coefficient, providing a specific method for determining the weight coefficient, providing the necessary technical basis for calculating the audit quality score, and improving the feasibility of the above-mentioned audit quality monitoring method.

[0036] In one possible implementation, determining a weight coefficient corresponding to each type of collected operation behavior based on the difference between each type of collected operation behavior and a standard operation behavior of that type includes at least one of the following steps:

[0037] Determining a first weight coefficient corresponding to the collected page dwell time according to a first difference between the collected page dwell time and the standard page dwell time;

[0038] Determining a second weight coefficient corresponding to the collected page browsing depth according to a second difference between the collected page browsing depth and the standard page browsing depth;

[0039] Determining a third weight coefficient corresponding to the collected link click behavior according to a third difference between the collected number of click operations on the preset mandatory link and the standard number of click operations on the preset mandatory link;

[0040] determining a fourth weight coefficient corresponding to the collected resource viewing behavior according to a ratio of the number of information items of the viewed image resource type to the number of information items of the standard image resource type;

[0041] A fifth weight coefficient corresponding to the collected content filling behavior is determined based on a fourth difference between the number of information items of the type of content to be filled in with a failed review status and the number of information items of the type of content to be filled in with a failed review status.

[0042] The audit quality monitoring method provided by the embodiment of the present invention provides a specific implementation method for determining the differences between different types of operating behaviors, thereby improving the feasibility and reliability of the above-mentioned audit quality monitoring method.

[0043] In a possible implementation, the greater the difference, the smaller the corresponding weight coefficient, including:

[0044] When the first difference is a non-negative value, it corresponds to the largest first weight coefficient. When the first difference is a negative value, different absolute value ranges of the first difference correspond to different first weight coefficients, and the first weight coefficient corresponding to the larger absolute value range is smaller than the first weight coefficient corresponding to the smaller absolute value range.

[0045] When the second difference is 0, it corresponds to the largest second weight coefficient. When the second difference is a negative value, different absolute value ranges of the second difference correspond to different second weight coefficients, and the second weight coefficient corresponding to the larger absolute value range is smaller than the second weight coefficient corresponding to the smaller absolute value range.

[0046] When the third difference is 0, it corresponds to the largest third weight coefficient; when the absolute value of the third difference is a positive value, it corresponds to the smallest third weight coefficient;

[0047] When the ratio is 1, it corresponds to the largest fourth weight coefficient; when the ratio is less than 1, different ratios correspond to different fourth weight coefficients, and the fourth weight coefficient corresponding to the smaller ratio is smaller than the fourth weight coefficient corresponding to the larger ratio;

[0048] When the fourth difference is 0, it corresponds to the largest fifth weight coefficient. When the fourth difference is a negative value, the corresponding fifth weight coefficient is the largest fifth weight coefficient minus the product of the preset coefficient and the number of information items of the unfilled content type whose review status is failed.

[0049] The audit quality monitoring method provided by the embodiment of the present invention provides a specific implementation method for calculating the corresponding weight coefficients of different types of operation behaviors, enhances the rationality and flexibility of the weight coefficient calculation, and improves the feasibility and reliability of the above-mentioned audit quality monitoring method.

[0050] In a possible implementation, after obtaining the review quality score corresponding to the page to be reviewed, the method further includes:

[0051] Obtaining the audit result of whether the page to be audited has passed the audit;

[0052] When audit information statistics are triggered, at least one of the following audit information will be counted;

[0053] Determine, based on the audit results, the number of pages to be audited of different subject types that have been audited and / or the number of pages that have not been audited during the statistical period;

[0054] Based on the audit results, determine the year-on-year change and / or month-on-month change in the number of audited and / or unaudited pages of different subject types during the statistical period;

[0055] Determine, based on the audit results, the sum of the audit quality scores of the pages to be audited that were audited by different audit accounts during the statistical period;

[0056] Determine, based on the audit results, the weight coefficients corresponding to the various types of operational behaviors on each page to be audited within the statistical period, and the audit quality scores corresponding to the pages to be audited;

[0057] According to the audit results, the number of pages to be audited in different quality score ranges within the statistical period is determined.

[0058] The audit quality monitoring method provided by the embodiment of the present invention specifically sets the form of audit information, provides a specific implementation method for statistical audit information, improves the feasibility of statistical audit information, and improves the feasibility and reliability of the above-mentioned audit quality monitoring method.

[0059] In a possible implementation, the statistical period includes different time periods, and after counting at least one audit information, the following steps are further included:

[0060] In response to the audit information display instruction, the audit information of any time period is displayed on the display interface in the form of text and / or statistical charts;

[0061] In response to a triggering operation on a time period switching icon on the display interface, review information of the time period indicated by the triggering operation is displayed on the display interface.

[0062] The audit quality monitoring method provided by the embodiment of the present invention provides a specific implementation method for displaying statistical audit information, displays the statistical audit information in the form of text and / or statistical charts, and can switch the time period of the statistical audit information, thereby improving the intuitiveness of the audit quality and can conveniently and quickly display the statistical audit information.

[0063] In a possible implementation, after obtaining the review quality score corresponding to the page to be reviewed, the method further includes:

[0064] When the audit quality test is triggered, the audit quality score is compared with a preset low audit quality alarm threshold;

[0065] When it is determined that the audit quality score is less than the preset low audit quality alarm threshold, a low audit quality alarm is triggered.

[0066] The audit quality monitoring method provided by the embodiment of the present invention provides a specific implementation method for low audit quality alarm. By monitoring whether the audit quality score is less than the preset low audit quality alarm threshold, it is determined whether to trigger the low audit quality alarm. When a low audit quality event occurs, the low audit quality alarm can be triggered in time, thereby improving the practicality of the above-mentioned audit quality monitoring method.

[0067] In a possible implementation, the score value of the standard page dwell time is smaller than the score values corresponding to other types of standard operation behaviors.

[0068] The audit quality monitoring method provided by the embodiment of the present invention can reduce the impact of page dwell time on the audit quality score by setting the score value of the standard page dwell time to be smaller than the score values corresponding to other types of standard operation behaviors, and alleviate the impact of deliberately delaying the page dwell time on the audit quality score, thereby improving the practicality and rationality of the above-mentioned audit quality monitoring method.

[0069] In a second aspect, an embodiment of the present invention provides an audit quality monitoring device, comprising:

[0070] A page display unit, configured to display the page to be reviewed in response to a review request for the page to be reviewed;

[0071] A behavior collection unit, configured to collect at least one type of operation behavior based on operations on the page to be reviewed;

[0072] A coefficient determination unit, configured to compare each type of collected operation behavior with a standard operation behavior of that type, and determine a weight coefficient corresponding to each type of collected operation behavior;

[0073] The quality evaluation unit is used to multiply the weight coefficient corresponding to each type of collected operation behavior by the score value of the standard operation behavior of that type and sum them up to obtain the audit quality score corresponding to the page to be audited.

[0074] In one possible implementation, the at least one type of operation behavior includes at least one of the following:

[0075] Statistics are collected for preset indicators related to operational behaviors, and the obtained indicators are statistical types of operational behaviors;

[0076] When a preset event is detected, the collected operational behavior of the event trigger type is detected.

[0077] In one possible implementation, the at least one type of operation behavior includes at least one of the following:

[0078] The time spent browsing the page to be reviewed;

[0079] The page browsing depth is determined based on the proportion of browsed pages among the pages to be reviewed;

[0080] A link click behavior triggered by clicking a link in the page to be reviewed;

[0081] Viewing resource viewing behavior triggered by image resources in the page to be reviewed;

[0082] Filling in the content to be filled in the page to be reviewed triggers the content filling behavior.

[0083] In a possible implementation, the behavior collection unit collects at least one type of operation behavior, including at least one of the following steps:

[0084] Count the time from the start of displaying the page to be reviewed to the closing of the page to be reviewed, and obtain the page dwell time;

[0085] Obtaining the page browsing depth according to the ratio of the scroll bar height that the slider slides over to the total height of the scroll bar in the page to be reviewed when the page to be reviewed is closed;

[0086] When a click operation on a link-type information item is detected, it is determined that the link click behavior is collected, and the number of click operations on the preset mandatory links is counted;

[0087] When a viewing operation of an information item of an image resource type is detected, it is determined that the resource viewing behavior is collected, and the number of viewed information items of the image resource type is counted;

[0088] When monitoring the content input operation of the information item of the content type to be filled in with the audit status of failed, it is determined that the content filling behavior is collected, and the number of information items of the content type to be filled in with the audit status of failed that have been filled in and the number of information items of the content type to be filled in with the audit status of failed are counted.

[0089] In a possible implementation, the behavior collection unit counts the number of click operations on the preset mandatory links, including:

[0090] Record the identification information of the information item of the link type clicked;

[0091] When it is determined according to the identification information that the click operation is a click operation on a preset mandatory link and the same identification information has not been recorded before, the number of click operations on the preset mandatory link is increased by one.

[0092] In a possible implementation, the coefficient determination unit compares each type of collected operation behavior with a standard operation behavior of that type to determine a weight coefficient corresponding to each type of collected operation behavior, including:

[0093] Determine the weight coefficient corresponding to each type of collected operation behavior based on the difference between each type of collected operation behavior and the standard operation behavior of that type;

[0094] Among them, the greater the difference, the smaller the corresponding weight coefficient.

[0095] In a possible implementation, the coefficient determination unit determines the weight coefficient corresponding to each type of collected operation behavior based on the difference between each type of collected operation behavior and a standard operation behavior of that type, including at least one of the following steps:

[0096] Determining a first weight coefficient corresponding to the collected page dwell time according to a first difference between the collected page dwell time and the standard page dwell time;

[0097] Determining a second weight coefficient corresponding to the collected page browsing depth according to a second difference between the collected page browsing depth and the standard page browsing depth;

[0098] Determining a third weight coefficient corresponding to the collected link click behavior according to a third difference between the collected number of click operations on the preset mandatory link and the standard number of click operations on the preset mandatory link;

[0099] determining a fourth weight coefficient corresponding to the collected resource viewing behavior according to a ratio of the number of information items of the viewed image resource type to the number of information items of the standard image resource type;

[0100] A fifth weight coefficient corresponding to the collected content filling behavior is determined based on a fourth difference between the number of information items of the type of content to be filled in with a failed review status and the number of information items of the type of content to be filled in with a failed review status.

[0101] In a possible implementation, the greater the difference, the smaller the corresponding weight coefficient, including:

[0102] When the first difference is a non-negative value, it corresponds to the largest first weight coefficient. When the first difference is a negative value, different absolute value ranges of the first difference correspond to different first weight coefficients, and the first weight coefficient corresponding to the larger absolute value range is smaller than the first weight coefficient corresponding to the smaller absolute value range.

[0103] When the second difference is 0, it corresponds to the largest second weight coefficient. When the second difference is a negative value, different absolute value ranges of the second difference correspond to different second weight coefficients, and the second weight coefficient corresponding to the larger absolute value range is smaller than the second weight coefficient corresponding to the smaller absolute value range.

[0104] When the third difference is 0, it corresponds to the largest third weight coefficient; when the absolute value of the third difference is a positive value, it corresponds to the smallest third weight coefficient;

[0105] When the ratio is 1, it corresponds to the largest fourth weight coefficient; when the ratio is less than 1, different ratios correspond to different fourth weight coefficients, and the fourth weight coefficient corresponding to the smaller ratio is smaller than the fourth weight coefficient corresponding to the larger ratio;

[0106] When the fourth difference is 0, it corresponds to the largest fifth weight coefficient. When the fourth difference is a negative value, the corresponding fifth weight coefficient is the largest fifth weight coefficient minus the product of the preset coefficient and the number of information items of the unfilled content type whose review status is failed.

[0107] In a possible implementation, after obtaining the audit quality score corresponding to the page to be audited, the quality evaluation unit is further configured to:

[0108] Obtaining the audit result of whether the page to be audited has passed the audit;

[0109] When audit information statistics are triggered, at least one of the following audit information will be counted;

[0110] Determine, based on the audit results, the number of pages to be audited of different subject types that have been audited and / or the number of pages that have not been audited during the statistical period;

[0111] Based on the audit results, determine the year-on-year change and / or month-on-month change in the number of audited and / or unaudited pages of different subject types during the statistical period;

[0112] Determine, based on the audit results, the sum of the audit quality scores of the pages to be audited that were audited by different audit accounts during the statistical period;

[0113] Determine, based on the audit results, the weight coefficients corresponding to the various types of operational behaviors on each page to be audited within the statistical period, and the audit quality scores corresponding to the pages to be audited;

[0114] According to the audit results, the number of pages to be audited in different quality score ranges within the statistical period is determined.

[0115] In a possible implementation, the statistical period includes different time periods, and after counting at least one item of audit information, the quality evaluation unit is further configured to:

[0116] In response to the audit information display instruction, the audit information of any time period is displayed on the display interface in the form of text and / or statistical charts;

[0117] In response to a triggering operation on a time period switching icon on the display interface, review information of the time period indicated by the triggering operation is displayed on the display interface.

[0118] In a possible implementation, after obtaining the audit quality score corresponding to the page to be audited, the quality evaluation unit is further configured to:

[0119] When the audit quality test is triggered, the audit quality score is compared with a preset low audit quality alarm threshold;

[0120] When it is determined that the audit quality score is less than the preset low audit quality alarm threshold, a low audit quality alarm is triggered.

[0121] In a possible implementation, the score value of the standard page dwell time is smaller than the score values corresponding to other types of standard operation behaviors.

[0122] In a third aspect, an embodiment of the present invention provides an audit quality monitoring device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes:

[0123] In response to a review request for a page to be reviewed, displaying the page to be reviewed;

[0124] Collect at least one type of operation behavior based on the operation on the page to be reviewed;

[0125] Comparing each type of collected operation behavior with the standard operation behavior of that type, and determining a weight coefficient corresponding to each type of collected operation behavior;

[0126] The weight coefficient corresponding to each type of collected operation behavior is multiplied by the score value of the standard operation behavior of that type and the sum is calculated to obtain the audit quality score corresponding to the page to be audited.

[0127] In one possible implementation, the at least one type of operation behavior includes at least one of the following:

[0128] Statistics are collected for preset indicators related to operational behaviors, and the obtained indicators are statistical types of operational behaviors;

[0129] When a preset event is detected, the collected operational behavior of the event trigger type is detected.

[0130] In one possible implementation, the at least one type of operation behavior includes at least one of the following:

[0131] The time spent browsing the page to be reviewed;

[0132] The page browsing depth is determined based on the proportion of browsed pages among the pages to be reviewed;

[0133] A link click behavior triggered by clicking a link in the page to be reviewed;

[0134] Viewing resource viewing behavior triggered by image resources in the page to be reviewed;

[0135] Filling in the content to be filled in the page to be reviewed triggers the content filling behavior.

[0136] In a possible implementation, the processor collects at least one type of operation behavior, including at least one of the following steps:

[0137] Count the time from the start of displaying the page to be reviewed to the closing of the page to be reviewed, and obtain the page dwell time;

[0138] Obtaining the page browsing depth according to the ratio of the scroll bar height that the slider slides over to the total height of the scroll bar in the page to be reviewed when the page to be reviewed is closed;

[0139] When a click operation on a link-type information item is detected, it is determined that the link click behavior is collected, and the number of click operations on the preset mandatory links is counted;

[0140] When a viewing operation of an information item of an image resource type is detected, it is determined that the resource viewing behavior is collected, and the number of viewed information items of the image resource type is counted;

[0141] When monitoring the content input operation of the information item of the content type to be filled in with the audit status of failed, it is determined that the content filling behavior is collected, and the number of information items of the content type to be filled in with the audit status of failed that have been filled in and the number of information items of the content type to be filled in with the audit status of failed are counted.

[0142] In a possible implementation, the processor counts the number of click operations on the preset mandatory link, including:

[0143] Record the identification information of the information item of the link type clicked;

[0144] When it is determined according to the identification information that the click operation is a click operation on a preset mandatory link and the same identification information has not been recorded before, the number of click operations on the preset mandatory link is increased by one.

[0145] In a possible implementation, the processor compares each type of collected operation behavior with a standard operation behavior of that type and determines a weight coefficient corresponding to each type of collected operation behavior, including:

[0146] Determine the weight coefficient corresponding to each type of collected operation behavior based on the difference between each type of collected operation behavior and the standard operation behavior of that type;

[0147] Among them, the greater the difference, the smaller the corresponding weight coefficient.

[0148] In one possible implementation, the processor determines the weight coefficient corresponding to each type of collected operation behavior based on the difference between each type of collected operation behavior and a standard operation behavior of that type, including at least one of the following steps:

[0149] Determining a first weight coefficient corresponding to the collected page dwell time according to a first difference between the collected page dwell time and the standard page dwell time;

[0150] Determining a second weight coefficient corresponding to the collected page browsing depth according to a second difference between the collected page browsing depth and the standard page browsing depth;

[0151] Determining a third weight coefficient corresponding to the collected link click behavior according to a third difference between the collected number of click operations on the preset mandatory link and the standard number of click operations on the preset mandatory link;

[0152] determining a fourth weight coefficient corresponding to the collected resource viewing behavior according to a ratio of the number of information items of the viewed image resource type to the number of information items of the standard image resource type;

[0153] A fifth weight coefficient corresponding to the collected content filling behavior is determined based on a fourth difference between the number of information items of the type of content to be filled in with a failed review status and the number of information items of the type of content to be filled in with a failed review status.

[0154] In a possible implementation, the greater the difference, the smaller the corresponding weight coefficient, including:

[0155] When the first difference is a non-negative value, it corresponds to the largest first weight coefficient. When the first difference is a negative value, different absolute value ranges of the first difference correspond to different first weight coefficients, and the first weight coefficient corresponding to the larger absolute value range is smaller than the first weight coefficient corresponding to the smaller absolute value range.

[0156] When the second difference is 0, it corresponds to the largest second weight coefficient. When the second difference is a negative value, different absolute value ranges of the second difference correspond to different second weight coefficients, and the second weight coefficient corresponding to the larger absolute value range is smaller than the second weight coefficient corresponding to the smaller absolute value range.

[0157] When the third difference is 0, it corresponds to the largest third weight coefficient; when the absolute value of the third difference is a positive value, it corresponds to the smallest third weight coefficient;

[0158] When the ratio is 1, it corresponds to the largest fourth weight coefficient; when the ratio is less than 1, different ratios correspond to different fourth weight coefficients, and the fourth weight coefficient corresponding to the smaller ratio is smaller than the fourth weight coefficient corresponding to the larger ratio;

[0159] When the fourth difference is 0, it corresponds to the largest fifth weight coefficient. When the fourth difference is a negative value, the corresponding fifth weight coefficient is the largest fifth weight coefficient minus the product of the preset coefficient and the number of information items of the unfilled content type whose review status is failed.

[0160] In a possible implementation, after obtaining the review quality score corresponding to the to-be-reviewed page, the processor is further configured to:

[0161] Obtaining the audit result of whether the page to be audited has passed the audit;

[0162] When audit information statistics are triggered, at least one of the following audit information will be counted;

[0163] Determine, based on the audit results, the number of pages to be audited of different subject types that have been audited and / or the number of pages that have not been audited during the statistical period;

[0164] Based on the audit results, determine the year-on-year change and / or month-on-month change in the number of audited and / or unaudited pages of different subject types during the statistical period;

[0165] Determine, based on the audit results, the sum of the audit quality scores of the pages to be audited that were audited by different audit accounts during the statistical period;

[0166] Determine, based on the audit results, the weight coefficients corresponding to the various types of operational behaviors on each page to be audited within the statistical period, and the audit quality scores corresponding to the pages to be audited;

[0167] According to the audit results, the number of pages to be audited in different quality score ranges within the statistical period is determined.

[0168] In a possible implementation, the statistical period includes different time periods, and after counting at least one item of audit information, the processor is further configured to:

[0169] In response to the audit information display instruction, the audit information of any time period is displayed on the display interface in the form of text and / or statistical charts;

[0170] In response to a triggering operation on a time period switching icon on the display interface, review information of the time period indicated by the triggering operation is displayed on the display interface.

[0171] In a possible implementation, after obtaining the review quality score corresponding to the to-be-reviewed page, the processor is further configured to:

[0172] When the audit quality test is triggered, the audit quality score is compared with a preset low audit quality alarm threshold;

[0173] When it is determined that the audit quality score is less than the preset low audit quality alarm threshold, a low audit quality alarm is triggered.

[0174] In a possible implementation, the score value of the standard page dwell time is smaller than the score values corresponding to other types of standard operation behaviors.

[0175] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the audit quality monitoring method provided in the first aspect above.

[0176] The audit quality monitoring method provided by the embodiment of the present invention can realize audit quality monitoring based on operational behavior by collecting operational behaviors on the page to be audited, calculating the corresponding weight coefficient, and calculating the audit quality score, so as to timely control the audit quality and ensure the content security of the page information delivery system and the delivery media platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0177] Figure 1 A schematic diagram for reviewing the basic information and creative ideas of advertisements for relevant technologies;

[0178] Figure 2 A schematic diagram of an application scenario of an audit quality monitoring method provided by an embodiment of the present invention;

[0179] Figure 3 A flowchart of an audit quality monitoring method provided by an embodiment of the present invention;

[0180] Figure 4 A schematic diagram of the distribution of operation behaviors on the advertiser qualification review page provided by an embodiment of the present invention;

[0181] Figure 5 A schematic diagram illustrating the distribution of another operation behavior on an advertiser qualification review page provided by an embodiment of the present invention;

[0182] Figure 6 A schematic diagram of the distribution of an operation behavior on an advertising creative review page provided by an embodiment of the present invention;

[0183] Figure 7 A schematic diagram of a page displaying audit information provided by an embodiment of the present invention;

[0184] Figure 8 A schematic diagram of a weight recording page for various types of operation behaviors provided by an embodiment of the present invention;

[0185] Figure 9 A flowchart of an audit quality test provided by an embodiment of the present invention;

[0186] Figure 10 A schematic diagram of an audit quality monitoring method provided by an embodiment of the present invention;

[0187] Figure 11 A flowchart of an implementation method of an audit quality monitoring method provided by an embodiment of the present invention;

[0188] Figure 12 A schematic diagram of an audit quality monitoring device provided by an embodiment of the present invention;

[0189] Figure 13 A schematic diagram of an audit quality monitoring device provided by an embodiment of the present invention;

[0190] Figure 14 A schematic diagram of an audit quality monitoring program product provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0191] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0192] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0193] According to an embodiment of the present invention, a method, apparatus and device for monitoring audit quality are proposed.

[0194] It should be understood herein that any number of elements in the drawings is for illustration only and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0195] For ease of understanding, the terms involved in the embodiments of this application are explained below:

[0196] Advertiser: refers to a legal person, other economic organization or individual who designs, produces and publishes advertisements on their own or entrusts others to do so in order to trade products or provide services.

[0197] Application Programming Interface (API): refers to some predefined interfaces, or the agreements for connecting different components of a software system.

[0198] Programmatic advertising: refers to the use of technological means to conduct advertising transactions and management. Advertisers can programmatically purchase media resources and use algorithms and technologies to automatically achieve precise target audience targeting; media can programmatically sell media resources across media and terminals, such as computers, mobile phones, tablets, and Internet TVs, and use technology to achieve tiered advertising traffic and differentiated pricing.

[0199] Year-on-year growth refers to the ratio of a parameter in a certain period of the current year to the corresponding parameter in the same period of the previous year. For example, if the statistical value of the parameter in August 2012 is 1000 and the statistical value of the parameter in August 2011 is 800, (1000 / 800-1)*100%=25%, that is, the parameter has increased by 25% year-on-year.

[0200] Month-on-month growth: refers to the ratio of a parameter in the current statistical period to the corresponding parameter in the previous statistical period. For example, if the statistical value of a parameter in July 2008 is 1000 and the statistical value in June 2008 is 800, (1000 / 800-1)*100%=25%, that is, the parameter has increased by 25% month-on-month.

[0201] The principles and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention. SUMMARY OF THE INVENTION

[0203] The inventors have discovered that the server of the media platform receives page information uploaded by the user for requesting page display, and the page information can be different types of page information such as advertisements or literary works. Taking advertisements as an example, with the development of Internet technology, the delivery of advertisements has entered the stage of programmatic advertising. Advertisers upload the basic information and advertising creativity of advertisements to the media platform through programs. In order to avoid non-compliant advertisements from being displayed and disseminated, after the server receives the basic information of the advertisements and the advertising creativity of the requested page display, it is necessary to review the basic information and advertising creativity according to the review rules. Once page information that does not comply with the review rules is delivered to the media platform, it will not only have an adverse impact on the users of the media platform, but also bring great security risks to the media platform, such as causing the media platform to stop operating and make rectifications, which will cause great losses to the media platform. Therefore, the review quality of page information is very important. Media platforms generally conduct page information review through reviewers, so the review quality of reviewers is related to the content security of the page information delivery system and the delivery media platform.

[0204] like Figure 1The figure shows a schematic diagram of the related technology reviewing the basic information and creative ideas of advertisements. Advertisers upload their information and creative materials on the advertising programmatic platform. The advertising programmatic platform pushes the review information to the advertising programmatic operation backend through the API. Reviewers review the advertiser information or creative materials in the advertising programmatic operation backend. The advertising programmatic operation backend returns the review results to the advertising programmatic platform. Managers can view the review workload and review progress counted by the advertising programmatic operation backend. Figure 1 It can be seen that the audit scheme of the relevant technology can only evaluate the working time and efficiency of the auditors by counting the number of pages reviewed by the auditors, the number of pages to be reviewed, the time taken for the audit, etc., but cannot monitor the quality of the audit.

[0205] In view of this, an embodiment of the present invention provides an audit quality monitoring method, including: displaying a page to be audited in response to an audit request for the page to be audited; collecting at least one type of operation behavior based on the operation on the page to be audited; comparing the collected types of operation behaviors with the standard operation behaviors of that type, and determining the weight coefficients corresponding to the collected types of operation behaviors; multiplying the weight coefficients corresponding to the collected types of operation behaviors with the score values of the standard operation behaviors of that type and summing them up to obtain the audit quality score corresponding to the page to be audited, so as to realize the monitoring of the audit quality.

[0206] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.

[0207] Application Scenario Overview

[0208] First reference Figure 2 , which is a schematic diagram of an application scenario of an audit quality monitoring method provided by an embodiment of the present invention.

[0209] The client device 21 is used to display the page to be reviewed in response to the review request for the page to be reviewed; collect at least one type of operation behavior based on the operation on the page to be reviewed; compare the collected types of operation behavior with the standard operation behavior of that type, and determine the weight coefficient corresponding to the collected types of operation behavior; multiply the weight coefficient corresponding to the collected types of operation behavior with the score value of the standard operation behavior of that type and sum them up to obtain the review quality score corresponding to the page to be reviewed.

[0210] The client device 22 is used to obtain the page to be reviewed and send the obtained page to be reviewed to the server 23.

[0211] The server 23 is configured to receive the page to be reviewed sent by the client device 22 and send the page to be reviewed to the client device 21 .

[0212] The client device 21 and the client device 22 are communicatively connected to the server 23 via a network, which may be a local area network, a wide area network, or the like.

[0213] The client device 21 and the client device 22 may be portable devices (eg, mobile phones, tablets, laptop computers, etc.) or personal computers (PCs).

[0214] The server 23 may be any device capable of providing Internet services.

[0215] The user interacts with the server 23 via the client installed in the client device 21 , wherein the client may be a web browser or an application client installed in a mobile user device such as a mobile phone or a tablet computer.

[0216] The user interacts with the server 23 via a client installed in the client device 22 , wherein the client may be a web browser or an application client installed in a mobile user device such as a mobile phone or a tablet computer.

[0217] It should be noted that the above application scenario is only an illustration of a possible implementation method in the embodiment of the present invention, and does not limit the embodiment of the present invention. On the contrary, the implementation method of the present application can be applied to any applicable scenario, and the above application scenario can be added, deleted, or modified according to the specific implementation situation.

[0218] For example, the functions performed by the above-mentioned client device 21 can be split into the client device and the server for execution, wherein the above-mentioned client device is used to display the page to be reviewed in response to the review request for the page to be reviewed; collect at least one type of operation behavior based on the operation on the page to be reviewed; send the collected at least one type of operation behavior to the server; the above-mentioned server is used to compare the collected various types of operation behaviors with the standard operation behaviors of that type, and determine the weight coefficients corresponding to the collected various types of operation behaviors; multiply the weight coefficients corresponding to the collected various types of operation behaviors with the score values of the standard operation behaviors of that type and sum them up to obtain the review quality score corresponding to the page to be reviewed.

[0219] Exemplary Methods

[0220] like Figure 3 FIG. 1 is a flowchart of a method for monitoring audit quality according to an embodiment of the present invention, which is applied to the client device 21, for example, and includes the following steps:

[0221] Step S301, in response to a review request for a page to be reviewed, displaying the page to be reviewed;

[0222] The above review request can be triggered by the reviewer performing a set operation, such as opening the review application, or clicking the review button displayed on the interface after the above review application is run.

[0223] The above-mentioned page to be reviewed may be different types of pages to be reviewed. For example, taking advertisements as an example, the above-mentioned page to be reviewed may be an advertiser qualification review page or an advertisement creative review page.

[0224] It should be noted that the embodiment of the present invention uses advertising as an example to illustrate the above-mentioned review quality monitoring method, but the above-mentioned review quality monitoring method can be applied to other platforms and systems that require review, such as review platforms for literary works, review platforms for video content, etc., which will not be repeated here.

[0225] Step S302: collecting at least one type of operation behavior based on the operation on the page to be reviewed;

[0226] The reviewer triggers multiple types of operations on the above-mentioned page to be reviewed. Depending on the collection method, the above-mentioned at least one type of operation includes at least one of the following:

[0227] Statistics are collected for preset indicators related to operational behaviors, and the obtained indicators are statistical types of operational behaviors;

[0228] When a preset event is detected, the collected operational behavior of the event trigger type is detected.

[0229] Depending on the specific operation behavior, the at least one type of operation behavior includes at least one of the following:

[0230] The time spent browsing the above-mentioned pages to be reviewed;

[0231] The page browsing depth is determined based on the percentage of browsed pages among the above pages to be reviewed;

[0232] Link click behavior triggered by clicking on the link in the above-mentioned page to be reviewed;

[0233] Check the resource viewing behavior triggered by the image resources in the above-mentioned pending review page;

[0234] Filling in the content to be filled in the above page to be reviewed triggers the content filling behavior.

[0235] The above-mentioned page dwell time and page browsing depth belong to the above-mentioned indicator statistical type of operation behaviors; the above-mentioned link clicking behavior, resource viewing behavior and content filling behavior belong to the above-mentioned event triggering type of operation behaviors.

[0236] It should be noted that the above-mentioned multiple types of operation behaviors provided in the embodiment of the present invention are only an example of the operation behaviors of the embodiment of the present invention. Other types of operation behaviors can be set according to the specific implementation situation, for example, the video viewing behavior triggered by watching the video resources in the above-mentioned page to be reviewed, etc. The embodiment of the present invention does not impose any limitations on this.

[0237] like Figure 4 As shown, an embodiment of the present invention provides a schematic diagram of the distribution of operation behaviors on the advertiser qualification review page.

[0238] The advertiser qualification review page includes different types of information items such as the advertiser's company name and website. The specific implementation methods of the above information items can be specifically set according to the specific implementation situation. The embodiment of the present invention does not impose any limitation on this and will not be repeated hereafter.

[0239] In the above Figure 4 In the advertiser qualification review interface shown, there are hidden preset indicators related to operational behaviors: page dwell time, page browsing depth, and different types of information items: link type information items, such as "website", and information items of content type to be filled in, such as "remarks" and "lack of qualifications". Link click behavior corresponds to the above-mentioned link type information items, and content filling behavior corresponds to the above-mentioned content type information items to be filled in.

[0240] like Figure 5 As shown, an embodiment of the present invention provides another distribution diagram of operation behaviors on the advertiser qualification review page.

[0241] In the above Figure 5 In the advertiser qualification review interface shown, there are hidden preset indicators related to operational behaviors: page dwell time, page browsing depth, and different types of information items: link type information items such as "website", image resource type information items such as "business license picture", and information items of content type to be filled in, such as "Remarks", "Review expiration date", "Reason for rejection", etc. The link clicking behavior corresponds to the above-mentioned link type information items, the resource viewing behavior corresponds to the above-mentioned image resource type information items, and the content filling behavior corresponds to the above-mentioned content type information items to be filled in.

[0242] like Figure 6 As shown, an embodiment of the present invention provides a schematic diagram of the distribution of operation behaviors on an advertising creative review page.

[0243] The above-mentioned advertising creative review page includes different types of information items such as the advertiser's company name, website, etc. The specific implementation methods of the above-mentioned information items can be specifically set according to the specific implementation situation. The embodiment of the present invention does not impose any limitation on this and will not be repeated hereafter.

[0244] In the above Figure 6 In the advertising creative review page shown, there are hidden preset indicators related to operational behaviors: page dwell time, page browsing depth, and different types of information items: link type information items, such as "website", "click URL", image resource type information items, such as "required qualifications", "industry qualifications", "alternative qualifications", "material resource pictures", and information items of content type to be filled in, such as "remarks". Link click behavior corresponds to the above-mentioned link type information items, resource viewing behavior corresponds to the above-mentioned image resource type information items, and content filling behavior corresponds to the above-mentioned content type information items to be filled in.

[0245] Return to Figure 3 In step S303, the collected operation behaviors of each type are compared with the standard operation behaviors of the type, and the weight coefficients corresponding to the collected operation behaviors of each type are determined;

[0246] Different types of standard operating behaviors and corresponding scoring values are set in advance according to the specific implementation situation, and the weight coefficients corresponding to the collected types of operating behaviors are determined by comparing the collected types of operating behaviors with the standard operating behaviors of that type.

[0247] It should be noted that the operation of determining the above-mentioned weight coefficient can be performed in real time or when a preset period is reached. For example, the various types of operation behaviors collected within the preset period are summarized through a real-time summary logic module, and the period is monitored through a scheduled task. After reaching the preset period, the corresponding weight coefficient is determined based on the above-mentioned multiple types of operation behaviors.

[0248] In step S304, the weight coefficient corresponding to each type of collected operation behavior is multiplied by the score value of the standard operation behavior of that type and the sum is calculated to obtain the audit quality score corresponding to the page to be audited.

[0249] In the embodiment of the present invention, during the operation of the auditor, the above-mentioned weight coefficient and the above-mentioned score value are used to perform behavioral weighting and calculate the audit quality score.

[0250] The above-mentioned behavior weighting refers to calculating the audit quality score using a preset formula based on the weight coefficient corresponding to each type of operation behavior and the score value of the standard operation behavior of that type.

[0251] Behavior weighting formula: f1+f2+…+f k=n, where audit quality score n<=100, f1, f2,…, f k It is called weight, which is the product of the weight coefficient corresponding to each type of operation behavior and the score value of the standard operation behavior of that type.

[0252] For example, the weight corresponding to the page dwell time is f1=5*0.8=4; the weight corresponding to the page browsing depth is f2=5*1=5; the weight corresponding to the link clicking behavior is f3=15*1=15; the weight corresponding to the resource viewing behavior is f4=45*0.8=36; the weight corresponding to the content filling behavior is f5=30*1=30; substitute the above behavior weighting formula, f1+f2+f3+f4+f5=4+5+15+36+30=90, and the audit quality score corresponding to the above page to be reviewed is calculated to be 90.

[0253] The embodiment of the present invention uses at least one of the following steps to collect the above-mentioned at least one type of operation behavior:

[0254] The time from the start of displaying the above-mentioned pending review page to the closing of the above-mentioned pending review page is counted to obtain the page dwell time;

[0255] The page browsing depth is obtained according to the ratio of the scroll bar height that the slider slides over to the total height of the scroll bar in the page to be reviewed when the page to be reviewed is closed;

[0256] When a click operation on a link-type information item is detected, it is determined that the link click behavior is collected, and the number of click operations on the preset mandatory links is counted;

[0257] When a viewing operation of an information item of an image resource type is detected, it is determined that the resource viewing behavior is collected, and the number of viewed information items of the image resource type is counted;

[0258] When monitoring the content input operation of the information item of the content type to be filled in with the audit status of failed, it is determined that the content filling behavior is collected, and the number of information items of the content type to be filled in with the audit status of failed that have been filled in and the number of information items of the content type to be filled in with the audit status of failed are counted.

[0259] It should be noted that in order to prevent auditors from using the collection rules of operation behaviors to unreasonably increase the weight coefficients corresponding to various types of operation behaviors to improve the audit quality score, corresponding rules are set when collecting operation behaviors to screen out cheating behaviors or reduce the impact of cheating behaviors on the audit quality score, as shown in Table 1 below.

[0260] Table 1: Measures for handling cheating behavior when collecting operational behavior

[0261]

[0262]

[0263] As an optional implementation, the score value of the above-mentioned standard page dwell time is smaller than the score values corresponding to other types of standard operation behaviors.

[0264] Lowering the score value of the standard page dwell time can reduce the impact of cheating behaviors such as deliberately delaying to increase page dwell time on the review quality score.

[0265] In addition, the page dwell time is recorded in the operation log, which can be used to check the page dwell time when the audit quality score is unreasonable.

[0266] The embodiment of the present invention adopts the following implementation method to filter link click behaviors. Specifically, the number of click operations on preset mandatory links is counted, including:

[0267] Record the identification information of the information item of the link type clicked;

[0268] When it is determined according to the above identification information that the click operation is a click operation on a preset mandatory link and the same identification information has not been recorded before, the number of click operations on the preset mandatory link is increased by one.

[0269] By repeatedly screening out click operations on preset mandatory links, the number of click operations on the preset mandatory links can be accurately counted, thereby ensuring the review of the preset mandatory links.

[0270] It should be noted that the above f1, f2, ..., f k The weight coefficient calculated based on the operation behavior after processing the above cheating behavior is multiplied by the score value of the standard operation behavior of this type.

[0271] The embodiment of the present invention defines weight coefficients corresponding to various types of operation behaviors based on different types of operation behaviors triggered by auditors on the page to be audited, such as page dwell time, page browsing depth, link clicking behavior, resource viewing behavior, content filling behavior, etc. During the operation process of the auditor, operation behaviors are collected, screening and resource comparison are performed to handle cheating behaviors, and the corresponding weight coefficients and audit quality scores are calculated for subsequent statistical quality monitoring data.

[0272] The embodiment of the present invention uses the following implementation to calculate the weight coefficient corresponding to each type of operation behavior. Specifically, the weight coefficient corresponding to each type of operation behavior collected is determined by comparing the collected operation behavior with the standard operation behavior of the type, including:

[0273] Determine the weight coefficient corresponding to each type of collected operation behavior based on the difference between each type of collected operation behavior and the standard operation behavior of that type;

[0274] Among them, the greater the difference, the smaller the corresponding weight coefficient.

[0275] Specifically, determining the weight coefficient corresponding to each type of collected operation behavior according to the difference between each type of collected operation behavior and the standard operation behavior of that type includes at least one of the following steps:

[0276] (1) determining a first weight coefficient corresponding to the collected page dwell time according to a first difference between the collected page dwell time and the standard page dwell time;

[0277] It should be noted that the specific value of the above-mentioned standard page dwell time can be set according to specific implementation conditions. For example, the above-mentioned standard page dwell time is set to 60 seconds, and the embodiment of the present invention does not impose any limitation on this.

[0278] When the first difference is a non-negative value, it corresponds to the largest first weight coefficient. When the first difference is a negative value, different absolute value ranges of the first difference correspond to different first weight coefficients, and the first weight coefficient corresponding to the larger absolute value range is smaller than the first weight coefficient corresponding to the smaller absolute value range.

[0279] When the above-mentioned first difference is a non-negative value, it means that the page residence time is greater than the standard page residence time, corresponding to the largest first weight coefficient, for example, 1; when the above-mentioned first difference is a negative value, it means that the page residence time is smaller than the standard page residence time, and the larger the absolute value of the above-mentioned first difference is, the smaller the page residence time is, and the smaller the corresponding first weight coefficient is.

[0280] The specific correspondence between the absolute value ranges of the different first differences and the different first weight coefficients can be specifically set according to specific implementation conditions, and the embodiment of the present invention does not impose any limitation on this.

[0281] (2) determining a second weight coefficient corresponding to the collected page browsing depth according to a second difference between the collected page browsing depth and the standard page browsing depth;

[0282] It should be noted that the specific value of the above-mentioned standard page browsing depth can be specifically set according to specific implementation conditions. For example, the above-mentioned standard page browsing depth is set to 80%, and the embodiment of the present invention does not impose any limitation on this.

[0283] When the second difference is 0, it corresponds to the largest second weight coefficient. When the second difference is a negative value, different absolute value ranges of the second difference correspond to different second weight coefficients, and the second weight coefficient corresponding to the larger absolute value range is smaller than the second weight coefficient corresponding to the smaller absolute value range.

[0284] When the above-mentioned second difference is 0, it means that the page browsing depth is equal to the standard page browsing depth, corresponding to the largest second weight coefficient, for example, 1; when the above-mentioned second difference is a negative value, it means that the page browsing depth is smaller than the standard page browsing depth, and the larger the absolute value of the above-mentioned second difference is, the smaller the page browsing depth is, and the smaller the corresponding second weight coefficient is.

[0285] The specific correspondence between the absolute value ranges of the different second differences and the different second weight coefficients can be specifically set according to specific implementation conditions, and the embodiment of the present invention does not impose any limitation on this.

[0286] (3) determining a third weight coefficient corresponding to the collected link click behavior based on a third difference between the collected number of click operations on the preset mandatory link and the standard number of click operations on the preset mandatory link;

[0287] It should be noted that the specific value of the number of click operations on the preset mandatory links in the above standard can be set according to the specific implementation situation. For example, the number of click operations on the preset mandatory links in the above standard is set to 3 times, and the embodiment of the present invention does not impose any limitation on this.

[0288] When the third difference is 0, it corresponds to the largest third weight coefficient; when the absolute value of the third difference is a positive value, it corresponds to the smallest third weight coefficient;

[0289] When the above-mentioned third difference is 0, it means that the number of collected click operations on the preset must-click links is equal to the standard number of click operations on the preset must-click links, corresponding to the largest third weight coefficient, for example 1; when the absolute value of the above-mentioned third difference is a positive value, it means that there are preset must-click links that have not been clicked, corresponding to the smallest third weight coefficient, for example 0.

[0290] (4) determining a fourth weight coefficient corresponding to the collected resource viewing behavior based on a ratio of the number of information items of the viewed image resource type to the number of information items of the standard image resource type;

[0291] It should be noted that the specific numerical value of the number of information items of the above-mentioned standard image resource type can be set according to the specific implementation situation. For example, the number of information items of the above-mentioned standard image resource type is set to 2, and the embodiment of the present invention does not impose any limitation on this.

[0292] When the above ratio is 1, it corresponds to the largest fourth weight coefficient. When the above ratio is less than 1, different ratios correspond to different fourth weight coefficients, and the fourth weight coefficient corresponding to the smaller ratio is smaller than the fourth weight coefficient corresponding to the larger ratio.

[0293] When the above ratio is 1, it indicates that the number of viewed information items of the image resource type is equal to the number of standard image resource type information items, corresponding to the largest fourth weight coefficient, for example 1; and the fourth weight coefficient is proportional to the above ratio.

[0294] (5) Determine a fifth weight coefficient corresponding to the collected content filling behavior based on a fourth difference between the number of information items of the type of content to be filled in that have been filled in and whose review status is rejected and the number of information items of the type of content to be filled in that have been filled in and whose review status is rejected.

[0295] It should be noted that the specific numerical value of the number of information items of the to-be-filled content type with the above-mentioned review status of "failed" needs to be obtained based on the collection of operation behaviors and is not a fixed value.

[0296] When the above-mentioned fourth difference is 0, it corresponds to the largest fifth weight coefficient. When the above-mentioned fourth difference is a negative value, the corresponding fifth weight coefficient is the above-mentioned largest fifth weight coefficient, minus the product of the preset coefficient and the number of information items of the unfilled content type whose review status is failed.

[0297] When the above-mentioned fourth difference is 0, it means that the number of information items of the type of content to be filled in whose audit status is rejected is equal to the number of information items of the type of content to be filled in whose audit status is rejected, corresponding to the largest fifth weight coefficient, for example 1; when the above-mentioned fourth difference is a negative value, it means that there are unfilled information items of the type of content to be filled in whose audit status is rejected, and the fifth weight coefficient is calculated based on the product of the preset coefficient and the number of unfilled information items of the type of content to be filled in whose audit status is rejected.

[0298] It should be noted that the number of the above-mentioned unfilled information items of the type of content to be filled in with a review status of "failed" can be obtained by collecting content filling behaviors, or by subtracting the number of information items of the type of content to be filled in with a review status of "failed" from the number of filled information items of the type of content to be filled in with a review status of "failed".

[0299] As shown in Table 2 below, an embodiment of the present invention provides a specific implementation method for determining weight coefficients corresponding to various types of collected operation behaviors.

[0300] Table 2: Specific implementation method for determining the weight coefficient corresponding to each type of collected operation behavior

[0301]

[0302]

[0303] It should be noted that the sum of the scoring values of the above standard operating behaviors shall not exceed 100.

[0304] The sum of the score values of the above standard operating behaviors is not less than 65 to pass.

[0305] After obtaining the above-mentioned audit quality score, the embodiment of the present invention collects and reports the audit results and related audit information to achieve monitoring of the audit quality.

[0306] For example, statistical audit information {ui, R, actions, T}, where ui is the auditor, R is the audit quality score, actions is the set of operation behaviors, and T is the audit type}.

[0307] After obtaining the audit quality score corresponding to the above-mentioned page to be audited, the embodiment of the present invention adopts the following implementation method to collect statistics on the audit results and related audit information:

[0308] Get the audit result of whether the above-mentioned page to be audited has passed the audit;

[0309] When audit information statistics are triggered, at least one of the following audit information will be counted;

[0310] Based on the above review results, determine the number of pages to be reviewed that have been reviewed and / or the number of pages that have not been reviewed for different subject types during the statistical period;

[0311] Based on the above review results, determine the year-on-year and / or month-on-month changes in the number of reviewed and / or unreviewed pages of different subject types during the statistical period;

[0312] Based on the above review results, determine the sum of the review quality scores of the pages to be reviewed that were reviewed by different review accounts during the statistical period;

[0313] Based on the above audit results, determine the weight coefficient corresponding to each type of operation behavior on each page to be audited during the statistical period, and the audit quality score corresponding to the above page to be audited;

[0314] Based on the above review results, determine the number of pages to be reviewed in different quality score ranges within the statistical period.

[0315] The above review results refer to whether the page to be reviewed has passed or failed the review.

[0316] It should be noted that the above-mentioned audit information is an example of the audit information of the embodiment of the present invention. Any relevant information that can reflect the audit results can be applied to the embodiment of the present invention and can be specifically set according to the specific implementation situation. The embodiment of the present invention does not impose any restrictions on this.

[0317] After collecting at least one item of audit information, the embodiment of the present invention uses the following implementation to display the audit information:

[0318] In response to the audit information display instruction, the audit information of any time period is displayed on the display interface in the form of text and / or statistical charts;

[0319] In response to a triggering operation on the time period switching icon on the above display interface, the audit information of the time period indicated by the above triggering operation is displayed on the above display interface.

[0320] The above statistical period includes different time periods. The period lengths of the above different time periods can be set specifically according to the specific implementation situation. For example, the period lengths can be set to hours, days, weeks, months, quarters, years, etc. The embodiment of the present invention does not impose any restrictions on this.

[0321] Among them, when the above-mentioned cycle length is an hour, the implementation method of real-time aggregation is adopted; when the above-mentioned cycle length is other lengths other than an hour, the implementation method of scheduled task aggregation is adopted.

[0322] The above audit information is generated into charts according to different auditors and / or different statistical periods and displayed, so that the audit administrator who views the above audit information can grasp the dynamics of the audit at any time, which is convenient for management.

[0323] like Figure 7 As shown, an embodiment of the present invention provides a schematic diagram of a page displaying audit information.

[0324] Audit administrators with administrator privileges can Figure 7 The Audit Administrator view page shown below displays the audit statistics.

[0325] above Figure 7The following are displayed in text form: (1) the total number of pages awaiting review of approved advertiser types, today's approved number, this week's approved number compared with the same period last year, and today's approved number compared with the same period last year; (2) the total number of pages awaiting review of approved advertising types, today's approved number, this week's approved number compared with the same period last year, and today's approved number compared with the same period last year; (3) the total number of pages awaiting review of unapproved advertising types, today's unapproved number, this week's unapproved number compared with the same period last year, and today's unapproved number compared with the same period last year; (4) the total number of pages awaiting review of unapproved advertising types, today's unapproved number, this week's unapproved number compared with the same period last year, and today's unapproved number compared with the same period last year.

[0326] above Figure 7 The ranking list shows the total review quality scores of the pending pages reviewed by different review accounts today.

[0327] above Figure 7 The bar chart shows the sum of the review quality scores of the advertiser type's pending review pages reviewed today by different review accounts, as well as the sum of the review quality scores of the advertising type's pending review pages. The horizontal axis of the bar chart is the review quality score, and the vertical axis is the different review accounts. The striped bar chart corresponds to the sum of the review quality scores of the advertiser type's pending review pages, and the solid color bar chart corresponds to the sum of the review quality scores of the advertising type's pending review pages.

[0328] above Figure 7 A pie chart is used to show the number of pages awaiting review in different quality score ranges today. For example, the number of pages awaiting review with a review quality score of 90 or above accounts for 30% of the number of pages awaiting review that have been reviewed today, the number of pages awaiting review with a review quality score of less than 65 accounts for 15% of the number of pages awaiting review that have been reviewed today, and the number of pages awaiting review with a review quality score of 65-90 accounts for 55% of the number of pages awaiting review that have been reviewed today.

[0329] above Figure 7 The table shows the audit quality scores that are below the low audit quality warning threshold today, as well as the corresponding audit accounts and types of pages to be reviewed.

[0330] It should be noted that by clicking the above Figure 7 The Today / This Week / This Month / All Year icons on the Audit Administrator View page can be used to switch the display of audit information for Today / This Week / This Month / All Year.

[0331] By clicking on the above Figure 7 The period selection icon on the audit administrator view page displayed allows you to freely set the start and end time of the time period.

[0332] like Figure 8 As shown, an embodiment of the present invention provides a schematic diagram of a weight record page for various types of operation behaviors.

[0333] above Figure 8 The displayed weight record page displays the operation weight record information of different reviewers in the form of a table. The above operation weight record information includes the name of the reviewer, the review quality score, the type of the page to be reviewed, the operation time, and the weight set, that is, the product of the weight coefficient of different types of operation behaviors and the score value of the standard operation behavior of that type, and the process of behavior weighted calculation.

[0334] It should be noted that, through the above Figure 8 On the displayed weight record page, enter the name of the reviewer in the search box and click the search icon to display the operation weight record information of the above reviewer.

[0335] By clicking on the above Figure 8 The Today / This Week / This Month / All Year icons on the displayed weight record page can switch to display the operation weight record information of Today / This Week / This Month / All Year.

[0336] By clicking on the above Figure 8 The period selection icon on the displayed weight record page allows you to freely set the start and end time of the time period.

[0337] After obtaining the audit quality score corresponding to the above-mentioned page to be audited, the embodiment of the present invention adopts the following implementation method to perform audit quality detection:

[0338] When the audit quality test is triggered, the above audit quality score is compared with the preset low audit quality alarm threshold;

[0339] When it is determined that the above audit quality score is less than the above preset low audit quality alarm threshold, a low audit quality alarm is triggered.

[0340] The triggering condition for the above-mentioned audit quality detection may be reaching a preset period, such as 1 hour, or receiving an instruction to detect the audit quality, etc.

[0341] The specific value of the low audit quality alarm threshold can be set according to the specific implementation situation. For example, the low audit quality alarm threshold is set to 65 points. The embodiment of the present invention does not impose any limitation on this.

[0342] like Figure 9 As shown, an embodiment of the present invention provides a flowchart of audit quality detection, including:

[0343] Step S901, setting a low audit quality alarm threshold;

[0344] Step S902, calculating the audit quality score;

[0345] The specific process of calculating the audit quality score is as follows Figure 3 As shown, no further details are given here.

[0346] Step S903, summarizing and calculating the audit quality scores of different audit accounts;

[0347] Step S904: When the audit quality test is triggered, it is determined whether there is an audit quality score less than the low audit quality warning threshold among the audit quality scores of different audit accounts. If so, step S905 is executed; otherwise, the process ends;

[0348] Step S905: triggering a low audit quality alarm for the above audit account.

[0349] It should be noted that the above Figure 9 What is provided is a low audit quality alarm process based on different audit accounts. The above low audit quality alarm can also be implemented based on different statistical periods or different pages to be audited. It can be specifically set according to the specific implementation situation. The embodiment of the present invention does not impose any limitations on this and will not be repeated here.

[0350] The embodiment of the present invention monitors the audit quality through the audit administrator view page, weight record page, and low audit quality alarm that display audit information, and can strictly control the content and quality of the audited advertisers and advertising materials to ensure the content security of the programmatic advertising system.

[0351] The embodiment of the present invention performs weighted statistics and aggregation on the collected operational behaviors, and performs report display and threshold alarms, providing new solutions and plans for controlling the quality of audits, and ensuring the content security of background systems that need to control content quality, such as programmatic advertising systems.

[0352] like Figure 10 As shown, an embodiment of the present invention provides a schematic diagram of an audit quality monitoring method.

[0353] Auditors trigger the collection of different types of operational behaviors on the audited page, and report the collected different types of operational behaviors, including page dwell time, page browsing depth, link click behavior, resource viewing behavior, content filling behavior, etc. The reported operational behaviors are summarized and analyzed, and the audit quality is monitored through statistical reports based on the audit quality scores of the operational behaviors of different auditors and low audit quality alarms.

[0354] The embodiment of the present invention proposes an audit quality monitoring method based on operational behavior, which can control and monitor the audit quality of different auditors, timely control the audit quality, and ensure the content security of content display platforms, such as programmatic advertising systems, etc., and overcome the defects of the audit solutions of related technologies that can only evaluate the working time and efficiency of auditors but cannot monitor the audit quality.

[0355] like Figure 11 As shown, an embodiment of the present invention provides a flowchart of an implementation method of an audit quality monitoring method, including:

[0356] Step S1101, in response to a review request for a page to be reviewed, displaying the page to be reviewed;

[0357] The above-mentioned pending review pages can be different types of pending review pages, such as advertiser-type pending review pages, or advertisement-type pending review pages.

[0358] Step S1102: collecting at least one type of operation behavior based on the operation on the page to be reviewed;

[0359] The at least one type of operation behavior mentioned above includes page dwell time, page browsing depth, link click behavior, resource viewing behavior, and content filling behavior.

[0360] Step S1103, comparing each type of collected operation behavior with a standard operation behavior of that type, and determining a weight coefficient corresponding to each type of collected operation behavior;

[0361] Step S1104: multiply the weight coefficient corresponding to each type of operation behavior collected by the score value of the standard operation behavior of that type and sum them to obtain the audit quality score corresponding to the above-mentioned page to be audited, and execute step S1105 or step S1108;

[0362] Step S1105: Obtain the audit result of whether the page to be audited has passed the audit, and then execute step S1106;

[0363] Step S1106: When audit information statistics are triggered, at least one item of audit information is counted and step S1107 is executed;

[0364] Step S1107: In response to the audit information display instruction, the audit information of any time period is displayed on the display interface in the form of text and / or statistical charts, and the process ends;

[0365] Step S1108: When the audit quality test is triggered, it is determined whether the audit quality score is less than the preset low audit quality warning threshold. If so, step S1109 is executed; otherwise, the process ends.

[0366] Step S1109 triggers a low audit quality alarm and the process ends.

[0367] Exemplary devices

[0368] After introducing the method of the exemplary embodiment of the present invention, the audit quality monitoring device and the like of the exemplary embodiment of the present application are introduced next.

[0369] like Figure 12 As shown, based on the same inventive concept, an embodiment of the present invention further provides an audit quality monitoring device, comprising:

[0370] The page display unit 1201 is configured to display the page to be reviewed in response to a review request for the page to be reviewed;

[0371] The behavior collection unit 1202 is used to collect at least one type of operation behavior based on the operation on the page to be reviewed;

[0372] The coefficient determination unit 1203 is used to compare the collected operation behaviors of each type with the standard operation behaviors of that type and determine the weight coefficient corresponding to each type of operation behaviors collected;

[0373] The quality evaluation unit 1204 is used to multiply the weight coefficient corresponding to each type of collected operation behavior by the score value of the standard operation behavior of that type and sum them up to obtain the audit quality score corresponding to the page to be audited.

[0374] In one possible implementation, the at least one type of operation behavior includes at least one of the following:

[0375] Statistics are collected for preset indicators related to operational behaviors, and the obtained indicators are statistical types of operational behaviors;

[0376] When a preset event is detected, the collected operational behavior of the event trigger type is detected.

[0377] In one possible implementation, the at least one type of operation behavior includes at least one of the following:

[0378] The time spent browsing the page to be reviewed;

[0379] The page browsing depth is determined based on the proportion of browsed pages among the pages to be reviewed;

[0380] A link click behavior triggered by clicking a link in the page to be reviewed;

[0381] Viewing resource viewing behavior triggered by image resources in the page to be reviewed;

[0382] Filling in the content to be filled in the page to be reviewed triggers the content filling behavior.

[0383] In a possible implementation, the behavior collection unit 1202 collects at least one type of operation behavior, including at least one of the following steps:

[0384] Count the time from the start of displaying the page to be reviewed to the closing of the page to be reviewed, and obtain the page dwell time;

[0385] Obtaining the page browsing depth according to the ratio of the scroll bar height that the slider slides over to the total height of the scroll bar in the page to be reviewed when the page to be reviewed is closed;

[0386] When a click operation on a link-type information item is detected, it is determined that the link click behavior is collected, and the number of click operations on the preset mandatory links is counted;

[0387] When a viewing operation of an information item of an image resource type is detected, it is determined that the resource viewing behavior is collected, and the number of viewed information items of the image resource type is counted;

[0388] When monitoring the content input operation of the information item of the content type to be filled in with the audit status of failed, it is determined that the content filling behavior is collected, and the number of information items of the content type to be filled in with the audit status of failed that have been filled in and the number of information items of the content type to be filled in with the audit status of failed are counted.

[0389] In a possible implementation, the behavior collection unit 1202 counts the number of click operations on the preset mandatory links, including:

[0390] Record the identification information of the information item of the link type clicked;

[0391] When it is determined according to the identification information that the click operation is a click operation on a preset mandatory link and the same identification information has not been recorded before, the number of click operations on the preset mandatory link is increased by one.

[0392] In a possible implementation, the coefficient determination unit 1203 compares each type of collected operation behavior with a standard operation behavior of that type to determine a weight coefficient corresponding to each type of collected operation behavior, including:

[0393] Determine the weight coefficient corresponding to each type of collected operation behavior based on the difference between each type of collected operation behavior and the standard operation behavior of that type;

[0394] Among them, the greater the difference, the smaller the corresponding weight coefficient.

[0395] In a possible implementation, the coefficient determination unit 1203 determines the weight coefficient corresponding to each type of collected operation behavior based on the difference between each type of collected operation behavior and a standard operation behavior of that type, including at least one of the following steps:

[0396] Determining a first weight coefficient corresponding to the collected page dwell time according to a first difference between the collected page dwell time and the standard page dwell time;

[0397] Determining a second weight coefficient corresponding to the collected page browsing depth according to a second difference between the collected page browsing depth and the standard page browsing depth;

[0398] Determining a third weight coefficient corresponding to the collected link click behavior according to a third difference between the collected number of click operations on the preset mandatory link and the standard number of click operations on the preset mandatory link;

[0399] determining a fourth weight coefficient corresponding to the collected resource viewing behavior according to a ratio of the number of information items of the viewed image resource type to the number of information items of the standard image resource type;

[0400] A fifth weight coefficient corresponding to the collected content filling behavior is determined based on a fourth difference between the number of information items of the type of content to be filled in with a failed review status and the number of information items of the type of content to be filled in with a failed review status.

[0401] In a possible implementation, the greater the difference, the smaller the corresponding weight coefficient, including:

[0402] When the first difference is a non-negative value, it corresponds to the largest first weight coefficient. When the first difference is a negative value, different absolute value ranges of the first difference correspond to different first weight coefficients, and the first weight coefficient corresponding to the larger absolute value range is smaller than the first weight coefficient corresponding to the smaller absolute value range.

[0403] When the second difference is 0, it corresponds to the largest second weight coefficient. When the second difference is a negative value, different absolute value ranges of the second difference correspond to different second weight coefficients, and the second weight coefficient corresponding to the larger absolute value range is smaller than the second weight coefficient corresponding to the smaller absolute value range.

[0404] When the third difference is 0, it corresponds to the largest third weight coefficient; when the absolute value of the third difference is a positive value, it corresponds to the smallest third weight coefficient;

[0405] When the ratio is 1, it corresponds to the largest fourth weight coefficient; when the ratio is less than 1, different ratios correspond to different fourth weight coefficients, and the fourth weight coefficient corresponding to the smaller ratio is smaller than the fourth weight coefficient corresponding to the larger ratio;

[0406] When the fourth difference is 0, it corresponds to the largest fifth weight coefficient. When the fourth difference is a negative value, the corresponding fifth weight coefficient is the largest fifth weight coefficient minus the product of the preset coefficient and the number of information items of the unfilled content type whose review status is failed.

[0407] In a possible implementation, after obtaining the audit quality score corresponding to the to-be-audited page, the quality evaluation unit 1204 is further configured to:

[0408] Obtaining the audit result of whether the page to be audited has passed the audit;

[0409] When audit information statistics are triggered, at least one of the following audit information will be counted;

[0410] Determine, based on the audit results, the number of pages to be audited of different subject types that have been audited and / or the number of pages that have not been audited during the statistical period;

[0411] Based on the audit results, determine the year-on-year change and / or month-on-month change in the number of audited and / or unaudited pages of different subject types during the statistical period;

[0412] Determine, based on the audit results, the sum of the audit quality scores of the pages to be audited that were audited by different audit accounts during the statistical period;

[0413] Determine, based on the audit results, the weight coefficients corresponding to the various types of operational behaviors on each page to be audited within the statistical period, and the audit quality scores corresponding to the pages to be audited;

[0414] According to the audit results, the number of pages to be audited in different quality score ranges within the statistical period is determined.

[0415] In a possible implementation, the statistical period includes different time periods. After collecting statistics on at least one audit information, the quality evaluation unit 1204 is further configured to:

[0416] In response to the audit information display instruction, the audit information of any time period is displayed on the display interface in the form of text and / or statistical charts;

[0417] In response to a triggering operation on a time period switching icon on the display interface, review information of the time period indicated by the triggering operation is displayed on the display interface.

[0418] In a possible implementation, after obtaining the audit quality score corresponding to the to-be-audited page, the quality evaluation unit 1204 is further configured to:

[0419] When the audit quality test is triggered, the audit quality score is compared with a preset low audit quality alarm threshold;

[0420] When it is determined that the audit quality score is less than the preset low audit quality alarm threshold, a low audit quality alarm is triggered.

[0421] In a possible implementation, the score value of the standard page dwell time is smaller than the score values corresponding to other types of standard operation behaviors.

[0422] Refer to the following Figure 13 The audit quality monitoring device 130 according to this embodiment of the present invention will be described. Figure 13 The audit quality monitoring device 130 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present invention.

[0423] like Figure 13 As shown, the audit quality monitoring device 130 can be in the form of a general-purpose computing device, such as a terminal device. The components of the audit quality monitoring device 130 may include, but are not limited to: the at least one processing unit 131, the at least one storage unit 132, and a bus 133 connecting different system components (including the storage unit 132 and the processing unit 131).

[0424] The bus 133 may include a data bus, an address bus, and a control bus.

[0425] The storage unit 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0426] The storage unit 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0427] The audit quality monitoring device 130 may also communicate with one or more external devices 134 (e.g., a keyboard, pointing device, etc.), and such communication may be performed via an input / output (I / O) interface 135. Furthermore, the audit quality monitoring device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules of the audit quality monitoring device 130 via a bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the audit quality monitoring device 130, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0428] Exemplary Program Products

[0429] In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to enable the client device to execute the steps of each module in the audit quality monitoring device according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the client device can execute the audit quality monitoring method, including: displaying the page to be audited in response to an audit request for the page to be audited; collecting at least one type of operation behavior based on the operation on the page to be audited; comparing the collected types of operation behaviors with the standard operation behaviors of the type, and determining the weight coefficients corresponding to the collected types of operation behaviors; multiplying the weight coefficients corresponding to the collected types of operation behaviors with the score values of the standard operation behaviors of the type and summing them up to obtain the audit quality score corresponding to the page to be audited.

[0430] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0431] like Figure 14As shown, a program product 140 for audit quality monitoring according to an embodiment of the present invention is described, which can be a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on a terminal device. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0432] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0433] Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0434] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0435] It should be noted that although several modules or submodules of the system are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0436] Furthermore, although the operations of the various modules of the system of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some operations may be omitted, multiple operations may be combined into one operation, and / or one operation may be decomposed into multiple operations.

[0437] Although the spirit and principles of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the specific embodiments disclosed, and the division into various aspects does not mean that the features of these aspects cannot be combined to benefit. Such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for monitoring audit quality, characterized in that: include: In response to a review request for a page to be reviewed, displaying the page to be reviewed; Collecting multiple types of operation behaviors based on the operation on the page to be reviewed, including: statistical operation behaviors of the indicator statistics type obtained by counting preset indicators related to the operation behaviors; and event-triggered operation behaviors collected when a preset event is triggered; Determine the weight coefficient corresponding to each type of collected operation behavior based on the difference between each type of collected operation behavior and the standard operation behavior of that type; Multiply the weight coefficient corresponding to each type of operation behavior collected by the score value of the standard operation behavior of that type and sum them up to obtain the audit quality score corresponding to the page to be audited; The greater the difference, the smaller the corresponding weight coefficient. Determining the weight coefficient corresponding to each type of collected operation behavior based on the difference between each type of collected operation behavior and the standard operation behavior of that type includes any number of the following steps: Determining a first weight coefficient corresponding to the collected page dwell time according to a first difference between the collected page dwell time and the standard page dwell time; Determining a second weight coefficient corresponding to the collected page browsing depth according to a second difference between the collected page browsing depth and the standard page browsing depth; Determining a third weight coefficient corresponding to the collected link click behavior according to a third difference between the collected number of click operations on the preset mandatory link and the standard number of click operations on the preset mandatory link; determining a fourth weight coefficient corresponding to the collected resource viewing behavior according to a ratio of the number of information items of the viewed image resource type to the number of information items of the standard image resource type; A fifth weight coefficient corresponding to the collected content filling behavior is determined based on a fourth difference between the number of information items of the type of content to be filled in with a failed review status and the number of information items of the type of content to be filled in with a failed review status.

2. The method according to claim 1, characterized in that The operational behaviors of the indicator statistics type include any of the following: The time spent browsing the page to be reviewed; The page browsing depth is determined based on the proportion of browsed pages among the pages to be reviewed; The event-triggered operation behaviors include any of the following: A link click behavior triggered by clicking a link in the page to be reviewed; Viewing resource viewing behavior triggered by image resources in the page to be reviewed; Filling in the content to be filled in the page to be reviewed triggers the content filling behavior.

3. The method according to claim 2, characterized in that The collection of various types of operation behaviors may include any number of the following steps: Count the time from the start of displaying the page to be reviewed to the closing of the page to be reviewed, and obtain the page dwell time; Obtaining the page browsing depth according to the ratio of the scroll bar height that the slider slides over to the total height of the scroll bar in the page to be reviewed when the page to be reviewed is closed; When a click operation on a link-type information item is detected, it is determined that the link click behavior is collected, and the number of click operations on the preset mandatory links is counted; When a viewing operation of an information item of an image resource type is detected, it is determined that the resource viewing behavior is collected, and the number of viewed information items of the image resource type is counted; When monitoring the content input operation of the information item of the content type to be filled in with the audit status of failed, it is determined that the content filling behavior is collected, and the number of information items of the content type to be filled in with the audit status of failed that have been filled in and the number of information items of the content type to be filled in with the audit status of failed are counted.

4. The method according to claim 3, characterized in that Count the number of clicks on pre-set mandatory links, including: Record the identification information of the information item of the link type clicked; When it is determined according to the identification information that the click operation is a click operation on a preset mandatory link and the same identification information has not been recorded before, the number of click operations on the preset mandatory link is increased by one.

5. The method according to claim 1, wherein The greater the difference, the smaller the corresponding weight coefficient, including: When the first difference is a non-negative value, it corresponds to the largest first weight coefficient. When the first difference is a negative value, different absolute value ranges of the first difference correspond to different first weight coefficients, and the first weight coefficient corresponding to the larger absolute value range is smaller than the first weight coefficient corresponding to the smaller absolute value range. When the second difference is 0, it corresponds to the largest second weight coefficient. When the second difference is a negative value, different absolute value ranges of the second difference correspond to different second weight coefficients, and the second weight coefficient corresponding to the larger absolute value range is smaller than the second weight coefficient corresponding to the smaller absolute value range. When the third difference is 0, it corresponds to the largest third weight coefficient; when the absolute value of the third difference is a positive value, it corresponds to the smallest third weight coefficient; When the ratio is 1, it corresponds to the largest fourth weight coefficient; when the ratio is less than 1, different ratios correspond to different fourth weight coefficients, and the fourth weight coefficient corresponding to the smaller ratio is smaller than the fourth weight coefficient corresponding to the larger ratio; When the fourth difference is 0, it corresponds to the largest fifth weight coefficient. When the fourth difference is a negative value, the corresponding fifth weight coefficient is the largest fifth weight coefficient minus the product of the preset coefficient and the number of information items of the unfilled content type whose review status is failed.

6. The method according to claim 1, characterized in that After obtaining the audit quality score corresponding to the page to be audited, it also includes: Obtaining the audit result of whether the page to be audited has passed the audit; When audit information statistics are triggered, at least one of the following audit information will be counted; Determine, based on the audit results, the number of pages to be audited of different subject types that have been audited and / or the number of pages that have not been audited during the statistical period; Based on the audit results, determine the year-on-year change and / or month-on-month change in the number of audited and / or unaudited pages of different subject types during the statistical period; Determine, based on the audit results, the sum of the audit quality scores of the pages to be audited that were audited by different audit accounts during the statistical period; Determine, based on the audit results, the weight coefficients corresponding to the various types of operational behaviors on each page to be audited within the statistical period, and the audit quality scores corresponding to the pages to be audited; According to the audit results, the number of pages to be audited in different quality score ranges within the statistical period is determined.

7. The method according to claim 6, characterized in that The statistical period includes different time periods, and after collecting statistics on at least one audit information, it also includes: In response to the audit information display instruction, the audit information of any time period is displayed on the display interface in the form of text and / or statistical charts; In response to a triggering operation on a time period switching icon on the display interface, review information of the time period indicated by the triggering operation is displayed on the display interface.

8. The method according to claim 1, characterized in that After obtaining the audit quality score corresponding to the page to be audited, it also includes: When the audit quality test is triggered, the audit quality score is compared with a preset low audit quality alarm threshold; When it is determined that the audit quality score is less than the preset low audit quality alarm threshold, a low audit quality alarm is triggered.

9. The method according to claim 1, characterized in that The score value of the standard page dwell time is smaller than the score values corresponding to other types of standard operation behaviors.

10. An audit quality monitoring device, characterized in that: include: A page display unit, configured to display the page to be reviewed in response to a review request for the page to be reviewed; A behavior collection unit is configured to collect multiple types of operation behaviors based on operations on the page to be reviewed, wherein the multiple types of operation behaviors include: operation behaviors of the indicator statistics type obtained by counting preset indicators related to the operation behaviors; and operation behaviors of the event triggering type obtained when monitoring the triggering of a preset event; A coefficient determination unit, configured to determine a weight coefficient corresponding to each type of collected operation behavior according to a difference between each type of collected operation behavior and a standard operation behavior of that type; A quality evaluation unit is used to multiply the weight coefficient corresponding to each type of collected operation behavior by the score value of the standard operation behavior of that type and sum them up to obtain the audit quality score corresponding to the page to be audited; The greater the difference, the smaller the corresponding weight coefficient. The coefficient determination unit determines the weight coefficient corresponding to each type of collected operation behavior according to the difference between each type of collected operation behavior and the standard operation behavior of that type, including any number of the following steps: Determining a first weight coefficient corresponding to the collected page dwell time according to a first difference between the collected page dwell time and the standard page dwell time; Determining a second weight coefficient corresponding to the collected page browsing depth according to a second difference between the collected page browsing depth and the standard page browsing depth; Determining a third weight coefficient corresponding to the collected link click behavior according to a third difference between the collected number of click operations on the preset mandatory link and the standard number of click operations on the preset mandatory link; determining a fourth weight coefficient corresponding to the collected resource viewing behavior according to a ratio of the number of information items of the viewed image resource type to the number of information items of the standard image resource type; A fifth weight coefficient corresponding to the collected content filling behavior is determined based on a fourth difference between the number of information items of the type of content to be filled in with a failed review status and the number of information items of the type of content to be filled in with a failed review status.

11. The device according to claim 10, characterized in that The operational behaviors of the indicator statistics type include any of the following: The time spent browsing the page to be reviewed; The page browsing depth is determined based on the proportion of browsed pages among the pages to be reviewed; The event-triggered operation behaviors include any of the following: A link click behavior triggered by clicking a link in the page to be reviewed; Viewing resource viewing behavior triggered by image resources in the page to be reviewed; Filling in the content to be filled in the page to be reviewed triggers the content filling behavior.

12. The device according to claim 11, characterized in that The behavior collection unit collects various types of operation behaviors, including any number of the following steps: Count the time from the start of displaying the page to be reviewed to the closing of the page to be reviewed, and obtain the page dwell time; Obtaining the page browsing depth according to the ratio of the scroll bar height that the slider slides over to the total height of the scroll bar in the page to be reviewed when the page to be reviewed is closed; When a click operation on a link-type information item is detected, it is determined that the link click behavior is collected, and the number of click operations on the preset mandatory links is counted; When a viewing operation of an information item of an image resource type is detected, it is determined that the resource viewing behavior is collected, and the number of viewed information items of the image resource type is counted; When monitoring the content input operation of the information item of the content type to be filled in with the audit status of failed, it is determined that the content filling behavior is collected, and the number of information items of the content type to be filled in with the audit status of failed that have been filled in and the number of information items of the content type to be filled in with the audit status of failed are counted.

13. The device according to claim 12, characterized in that The behavior collection unit counts the number of click operations on the preset mandatory links, including: Record the identification information of the information item of the link type clicked; When it is determined according to the identification information that the click operation is a click operation on a preset mandatory link and the same identification information has not been recorded before, the number of click operations on the preset mandatory link is increased by one.

14. The device according to claim 10, characterized in that The greater the difference, the smaller the corresponding weight coefficient, including: When the first difference is a non-negative value, it corresponds to the largest first weight coefficient. When the first difference is a negative value, different absolute value ranges of the first difference correspond to different first weight coefficients, and the first weight coefficient corresponding to the larger absolute value range is smaller than the first weight coefficient corresponding to the smaller absolute value range. When the second difference is 0, it corresponds to the largest second weight coefficient. When the second difference is a negative value, different absolute value ranges of the second difference correspond to different second weight coefficients, and the second weight coefficient corresponding to the larger absolute value range is smaller than the second weight coefficient corresponding to the smaller absolute value range. When the third difference is 0, it corresponds to the largest third weight coefficient; when the absolute value of the third difference is a positive value, it corresponds to the smallest third weight coefficient; When the ratio is 1, it corresponds to the largest fourth weight coefficient; when the ratio is less than 1, different ratios correspond to different fourth weight coefficients, and the fourth weight coefficient corresponding to the smaller ratio is smaller than the fourth weight coefficient corresponding to the larger ratio; When the fourth difference is 0, it corresponds to the largest fifth weight coefficient. When the fourth difference is a negative value, the corresponding fifth weight coefficient is the largest fifth weight coefficient minus the product of the preset coefficient and the number of information items of the unfilled content type whose review status is failed.

15. The device according to claim 10, characterized in that After obtaining the audit quality score corresponding to the page to be audited, the quality evaluation unit is further configured to: Obtaining the audit result of whether the page to be audited has passed the audit; When audit information statistics are triggered, at least one of the following audit information will be counted; Determine, based on the audit results, the number of pages to be audited of different subject types that have been audited and / or the number of pages that have not been audited during the statistical period; Based on the audit results, determine the year-on-year change and / or month-on-month change in the number of audited and / or unaudited pages of different subject types during the statistical period; Determine, based on the audit results, the sum of the audit quality scores of the pages to be audited that were audited by different audit accounts during the statistical period; Determine, based on the audit results, the weight coefficients corresponding to the various types of operational behaviors on each page to be audited within the statistical period, and the audit quality scores corresponding to the pages to be audited; According to the audit results, the number of pages to be audited in different quality score ranges within the statistical period is determined.

16. The device according to claim 15, characterized in that The statistical period includes different time periods. After collecting statistics on at least one audit information, the quality evaluation unit is further configured to: In response to the audit information display instruction, the audit information of any time period is displayed on the display interface in the form of text and / or statistical charts; In response to a triggering operation on a time period switching icon on the display interface, review information of the time period indicated by the triggering operation is displayed on the display interface.

17. The device according to claim 10, characterized in that After obtaining the audit quality score corresponding to the page to be audited, the quality evaluation unit is further configured to: When the audit quality test is triggered, the audit quality score is compared with a preset low audit quality alarm threshold; When it is determined that the audit quality score is less than the preset low audit quality alarm threshold, a low audit quality alarm is triggered.

18. The device according to claim 10, characterized in that The score value of the standard page dwell time is smaller than the score values corresponding to other types of standard operation behaviors.

19. An audit quality monitoring device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the audit quality monitoring method according to any one of claims 1 to 9 are implemented.

20. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the audit quality monitoring method described in any one of claims 1 to 9 are implemented.

Images