A method for improving the efficiency of manual review of text content

Abstract

The application provides a method for improving the artificial auditing efficiency of text content, which comprises the following steps: detecting a text generation result, calculating the grouping hash to classify similar content, calculating a unique identifier to remove duplicates, discarding the unique identifier if it exists, otherwise trying to inherit the historical artificial auditing result, automatically inheriting the auditing state, type and level, and marking the system auditing, storing the result and the hash, identifier and auditing result, displaying and batch auditing according to the grouping hash deduplication, and finally updating the auditing state of the whole group; the application can intelligently merge the massive text content to be audited through the grouping hash mechanism, so that the auditors do not need to process the repeated or highly similar content one by one, but only need to audit the representative data after deduplication according to the grouping; when processing the data of the order of 100,000, the auditing time can be shortened from dozens of hours in the traditional way to several hours.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for improving the efficiency of manual review of text content. Background Technology

[0002] The core efficiency bottleneck currently faced by large-scale text content review is that when the amount of data to be reviewed reaches hundreds of thousands or even millions of items, the traditional manual review method cannot complete the work within a reasonable time. In particular, for reprinted content, repeated sensitive information, attachments, or identical external links, reviewers need to repeatedly process essentially the same content, resulting in a large amount of ineffective work.

[0003] While existing review technologies utilize keyword filtering or simple hashing for deduplication to some extent, they still have shortcomings. First, traditional manual review methods are inefficient. When the amount of data to be reviewed reaches hundreds of thousands or even millions, it is basically impossible to conduct reviews normally. Reviewers need to process the same or similar content one by one, resulting in a large amount of repetitive work. Second, existing AI-powered intelligent review methods have the problem of false reviews and cannot meet the requirement of 100% accuracy. Once a problem occurs, it may lead to the client being reported by higher-level agencies during review.

[0004] Therefore, in response to the problems mentioned above, this invention proposes a method for improving the efficiency of manual review of text content. Summary of the Invention

[0005] To overcome the inefficiency of traditional manual review methods and the errors in existing AI-powered review methods, this invention proposes a method to improve the efficiency of manual review of text content. By combining technologies such as categorized storage, group hashing, and automatic inheritance, reviewers can focus on differentiated content that truly requires human judgment, thus significantly improving the efficiency of manual review.

[0006] The technical solution of this invention is: a method for improving the efficiency of manual review of text content, comprising the following steps:

[0007] S1, Detect the content of the text to be reviewed and generate the detection results;

[0008] S2, For each detection result, calculate the group hash value based on its detection type and key information, which is used to group similar content into the same group;

[0009] S3: For each detection result, a unique identifier is calculated based on its detection type, media identifier, webpage link, and key information for deduplication.

[0010] S4. Before the test results are entered into the database, query the database to see if the same record already exists based on the unique identifier. If it exists, discard the current test result. If it does not exist, continue to determine whether there is an inheritable historical manual review result.

[0011] S5. If there are inheritable historical audit results, the most recent manual audit result will be automatically inherited, including audit status, post-audit risk type and post-audit risk level. At the same time, the auditor will be marked as system and the audit time will be set to the current system time.

[0012] S6 stores the detection results, along with their group hash, unique identifier, and inherited audit results, into the database;

[0013] S7, In the manual review process, the data to be reviewed is deduplicated and displayed according to the group hash, and the reviewers conduct batch review of the deduplicated data;

[0014] S8. After the review is completed, update the review status of all detection results under this group hash.

[0015] Preferably, the calculation method of the group hash varies depending on the detection type, including:

[0016] For sensitive information, a hash calculation is performed on "detection type + sentence containing + sensitive words + recommended words";

[0017] For privacy information, a hash calculation is performed on "detection type + sentence containing + privacy content";

[0018] For sensitive individuals, a hash calculation is performed on "detection type + sentence in which it appears + name + position";

[0019] For risky backlinks, perform a hash calculation on "detection type + backlink";

[0020] For custom words, a hash calculation is performed on "detection type + sentence containing the custom word + recommended word".

[0021] Preferably, the calculation method for the unique identifier varies depending on the detection type, including:

[0022] For sensitive information, a hash calculation is performed on "detection type + media ID + webpage link + sentence containing + sensitive words + recommended words";

[0023] For privacy information, a hash calculation is performed on "detection type + media ID + webpage link + sentence containing + privacy content";

[0024] For sensitive individuals, a hash calculation is performed on "detection type + media ID + webpage link + sentence containing + name + position";

[0025] For risky backlinks, perform hash calculations on "detection type + media ID + webpage link + backlink";

[0026] For custom words, a hash calculation is performed on "detection type + media ID + webpage link + sentence containing + custom word + recommended word".

[0027] Preferably, step S5 specifically includes:

[0028] S51, Based on the group hash of the current detection results, query the database for existing manually reviewed records;

[0029] S52, sorted in reverse order by review time, select the most recent manual review result for inheritance;

[0030] S53 sets the current inspection result's review status, post-review risk type, and post-review risk level to the corresponding values ​​of the inherited record;

[0031] S54, the reviewer is marked as system, which is used to determine whether the partition is automatically reviewed by the system or manually reviewed, and the review time is set to the current system time.

[0032] Preferably, the review status is set to the corresponding field value of the most recent manual review record; the post-review risk type is set to the corresponding field value of the most recent manual review record; and the post-review risk level is set to the corresponding field value of the most recent manual review record.

[0033] Preferably, the batch review step in step S7 includes:

[0034] S71, Auditors can query data to be audited by test type;

[0035] S72, the system displays duplicate data to be reviewed based on group hash;

[0036] S73, the auditor selects one or more representative records to perform the audit operation;

[0037] S74, the system marks all detection results in the selected record group as audited based on the group hash.

[0038] As a preferred option, for the detection results of risky backlinks, three batch review modes are supported during the review process: review by main domain, review by host, and review by link. Review by main domain reviews all backlink detection results under the same main domain; review by host reviews all backlink detection results under the same host; and review by link reviews based on group hashes, consistent with other detection types.

[0039] Preferably, the detection results include, but are not limited to, one or more of the following: sensitive information, privacy information, sensitive individuals, risky backlinks, and custom words.

[0040] Preferably, the review status includes two states: "confirmed to be a problem that needs to be modified" and "confirmed to be a non-problem that does not need to be modified".

[0041] Preferably, the risk type and risk level after review are dynamically set according to the specific detection type and review results.

[0042] The beneficial effects of this invention are:

[0043] 1. This invention uses a group hashing mechanism to intelligently merge massive amounts of text content to be reviewed, so that reviewers no longer need to process duplicate or highly similar content one by one, but only need to review representative data after deduplication by group. When processing hundreds of thousands of data, the review time can be shortened from tens of hours in the traditional way to several hours, with an efficiency improvement of up to 97%, thereby solving the industry problem that large-scale text review tasks are difficult to complete within a reasonable time.

[0044] 2. This invention creates an automatic inheritance mechanism for audit results, ensuring the consistency of audit conclusions. The system can automatically query and apply historical manual audit results to newly detected similar issues based on group hashes. This not only avoids the accidental deviations that may occur when the same auditor makes repeated judgments on the same content, but also eliminates the possibility of different rulings by different auditors on essentially the same issues. This greatly improves the accuracy of the overall audit work and reduces the operational risks caused by misjudgments or inconsistent standards.

[0045] 3. This invention achieves accurate deduplication of detection results through a unique identifier (unique_id), avoiding duplicate detection results from the same media and the same page being entered into the database due to multiple scans. This mechanism reduces the total amount of the list to be reviewed from the data source, so that each record shown to the reviewers represents an independent and unique specific issue, further reducing unnecessary duplication.

[0046] 4. The reviewers of this invention can select the most suitable batch processing combination based on the risk characteristics and distribution of external links, so as to achieve efficient handling of a large number of related external links under the same domain name. This not only greatly simplifies the operation process, but also ensures the consistency of risk control for the entire site or a batch of related external links. Attached Figure Description

[0047] Figure 1 The diagram shown illustrates the workflow of this invention.

[0048] Figure 2 The diagram shown illustrates the review process of this invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] This invention provides an embodiment of a method for improving the efficiency of manual review of text content. This technology categorizes and stores the detection results of the text content, calculates a group hash and a unique ID (unique_id) for each detection result, and saves them together with the detection result. During the process of storing the detection results, deduplication is performed based on the unique ID. Simultaneously, historical manual review results are automatically matched and inherited based on the group hash, avoiding repeated manual review for the same issues. Furthermore, when processing data to be reviewed, reviewers no longer need to display all the original data; they only need to display the data after deduplication by group hash. That is, only one record is displayed for each group of data, and reviewing one record constitutes reviewing a batch of data, achieving rapid manual review.

[0051] This invention relies on the following text content review system, specifically:

[0052] The content detection engine is used to perform preliminary automated detection on the raw text content (such as web articles, attached text, user comments, etc.) that is crawled or submitted, and to identify the detection results that may contain sensitive information, privacy information, sensitive persons, risky external links or custom words.

[0053] The data processing and calculation module is responsible for receiving the preliminary detection results from the detection engine and executing the calculation logic for group hash and unique identifier.

[0054] The database stores all detection results and their related metadata, and contains at least the following fields: id (auto-incrementing primary key), media_id (media identifier), page_url (webpage link), detection_type (detection type), content_snippet (sentence or content fragment), keyword (sensitive words / private content / name / external links / custom words), suggested_word (recommended words, if applicable), position (job title, if applicable), group_hash (group hash value), unique_id (unique identifier), review_status (review status), reviewed_risk_type (risk type after review), reviewed_risk_level (risk level after review), reviewer (reviewer), review_time (review time), and create_time (creation time).

[0055] The review operation module provides a user interface for reviewers, displaying a list of deduplicated data to be reviewed by group, and provides a batch review operation interface.

[0056] The module for inheriting and updating audit results executes automatic inheritance logic during the data entry process and batch update logic after manual review.

[0057] This invention provides an embodiment, and the workflow of this invention is described in detail below:

[0058] The workflow of this method is mainly divided into two stages: the test results entry stage and the manual review stage.

[0059] Please see Figure 1 First, let's explain the process of storing the test results. This step involves cleaning, deduplicating, and preprocessing the massive amounts of raw results generated by the testing engine, laying the foundation for efficient manual review.

[0060] The first step is that after the data processing module receives a detection result, it first selects the appropriate algorithm to calculate its group_hash and unique_id according to its detection_type. The calculation uses a standard hash function to convert the concatenated string into a fixed-length hash value.

[0061] The calculation rules for group hash (group_hash) are explained below. These rules are used for content grouping and batch review:

[0062] Sensitive information: Calculate hash("sensitive information" + content_snippet + keyword + suggested_word). This rule ensures that identical sentence fragments containing the same sensitive words and suggested words appearing on different pages will be grouped together.

[0063] Privacy Information: Calculate hash("Privacy Information" + content_snippet + keyword). This rule ensures that identical sentence fragments containing the same privacy information (such as ID number or phone number) appearing on different pages are grouped together.

[0064] Sensitive Person: Calculate hash("sensitive person" + content_snippet + keyword + position). This rule ensures that identical sentence fragments reporting on the same person (same name and position) appearing on different pages are grouped together.

[0065] Risky backlinks: Calculate hash("risky backlink" + keyword). This rule ensures that all results detecting backlinks with the same URL are grouped together.

[0066] Custom words: Calculate hash("custom word" + content_snippet + keyword + suggested_word). This rule is similar to that for sensitive information, ensuring that the same custom words appearing in the same context are grouped together.

[0067] The calculation rules for the unique identifier (unique_id) are explained below. These rules are used for precise deduplication: Sensitive information: Calculate hash("sensitive information"+media_id+page_url+content_snippet+keyword+suggested_word).

[0068] Privacy information: Calculate hash("privacy information"+media_id+page_url+content_snippet+keyword).

[0069] Sensitive Person: Calculate hash("sensitive person"+media_id+page_url+content_snippet+keyword+position).

[0070] Risky backlinks: Calculate hash("risky backlink" + media_id + page_url + keyword).

[0071] Custom words:

[0072] Calculate hash("custom word"+media_id+page_url+content_snippet+keyword+suggested_word).

[0073] The unique_id introduces media_id and page_url, thereby identifying a specific question on a specific page under a specific media.

[0074] The second step involves the system querying the database using the calculated unique_id as a condition. If a record with the same unique_id already exists in the database, it indicates that the same specific problem under the same media and page has been detected, and the current result is considered a duplicate detection (e.g., the same page has been scanned multiple times). In this case, the result is discarded, and the process ends.

[0075] Third, if it does not exist, the system uses the group_hash of the current detection result as a condition to query all records in the database with the same detection_type and which already have manual review results. The query results are sorted in descending order of review_time.

[0076] Fourth step: If a record is found in the third step, select the most recent one (i.e., the latest manual review result).

[0077] Fifth step, set the review-related fields of the current test result to the values ​​of the inherited record:

[0078] review_status = inherited review_status (e.g., "Confirmed issue" or "No issue");

[0079] reviewed_risk_type = inherited record reviewed_risk_type (e.g., external link reviewed as "gambling");

[0080] reviewed_risk_level = inherited record reviewed_risk_level (e.g., privacy information reviewed as "high risk");

[0081] reviewer='system' (This explicitly marks this as an automatic system review);

[0082] review_time = current system time;

[0083] The sixth step is to store the current detection results (which already include the inherited audit results) into the database.

[0084] Step 7: If no historical manual review records are found in Step 3, proceed directly to Step 6 to store the current test results (with review-related fields empty) into the database.

[0085] Please see Figure 2 Furthermore, the manual review process is explained. This process involves direct participation from reviewers, and its interface and operational logic have been optimized based on the group_hash generated during the data entry process.

[0086] The first step is for the reviewers to log in to the review system and enter the manual review workbench.

[0087] The second step involves the reviewer selecting filtering criteria, such as searching by detection_type (sensitive information, privacy information, etc.). The system does not simply list all records with review_status as "unreviewed". Instead, it groups the unreviewed data by group_hash and detection_type, and then displays only one representative record for each group, such as the newest or earliest record in the group. At the same time, the interface clearly displays the issue content corresponding to the group_hash and the total number of pending records in the group.

[0088] The third step is for the reviewers to browse the deduplicated list, select one or more representative records, and then click the "Review" button.

[0089] The fourth step is for the system to determine the audit mode based on the detection_type of the selected record.

[0090] Fifth, if it is a risky external link, the system provides reviewers with three batch review granularity options:

[0091] Review by main domain: The system resolves the main domain of the external link (e.g., the main domain of https: / / job.xidian.edu.cn / is xidian.edu.cn). After the reviewer confirms the operation, the system will update all the detection results of external links in the database that have the same main domain and are in an unreviewed state to the result of this review.

[0092] Review by Host: The system parses the host of the external link.

[0093] (When the external link is https: / / job.xidian.edu.cn / campus / view / id / 751457, the corresponding Host is job.xidian.edu.cn. During batch review, external links with the same Host, such as https: / / job.xidian.edu.cn / campus / view / id / 751457 and https: / / job.xidian.edu.cn / campus / view / id / 751456, will be reviewed together.)

[0094] After the reviewer confirms the operation, the system will update all external links in the database with the same Host that are in an unreviewed state to the results of this review.

[0095] Review by link: This mode is consistent with the review logic of group_hash. After the reviewer confirms the operation, the system will update all the detection results of group_hash in the database that are the same as the selected record and are in an unreviewed state to the result of this review.

[0096] Step 6: If it is not a risky external link type, the system directly uses group hashing for batch review. After the reviewer makes a review decision and confirms it, the system will update all detection results in the database that have the same group_hash and detection_type as the selected record and are in an unreviewed state to the results of this review.

[0097] Once any batch review operation is successfully executed, the system page will automatically refresh, and all records of the one or more groups that were just reviewed will disappear from the list of pending reviews.

[0098] Step 7: The reviewer decides whether to continue the review. If so, they return to step 2 to query and review the new group. If not, the process terminates.

[0099] Furthermore, the present invention provides Embodiment 1:

[0100] This embodiment was applied in the Xi'an Boda Software Content Security Scanning and Monitoring Platform, and the supervision of 1154 websites under Xiamen University was used as an example. The table below compares the workload and time consumption of reviewing a full scan result before and after applying this invention:

[0101] Table 1 Comparison between traditional methods and the present invention

[0102]

[0103] As shown in Table 1, taking "risky backlinks" as an example, more than 840,000 original records were reduced to more than 28,000 unique groups after being grouped and hashed. Reviewers only need to process these 28,000 groups of data, which directly reduces the workload by 97%, and the total review time is reduced from about 992 hours to about 125 hours, showing a significant improvement in efficiency.

[0104] Furthermore, the present invention provides Embodiment 2:

[0105] This embodiment is based on a large news portal website, which conducts compliance review of articles on the platform. A large number of articles are reprinted from other media. The same article reprinted using traditional methods may be edited and published by multiple channels or recommended multiple times. Therefore, the reviewers need to perform repeated review operations on each reprinted article, which makes the efficiency very low.

[0106] Using this invention, this embodiment provides an example: The system detects that a reprinted article about a "major event" contains the sensitive word A. It calculates the group_hash (based on "sensitive information" + fragment + sensitive word A + recommended word). Assuming this is the first time the article has appeared and there is no historical review record, the detection result is marked as "pending review." The reviewer reviews the article, determines it to be "not a problem," and confirms its publication. Afterwards, another channel reprints the same article. The system detects it again and calculates its group_hash, finding it to be exactly the same as the value detected at the beginning. At this point, the system automatically marks the review status of the new detection result as "not a problem" through an automatic inheritance mechanism, marks the reviewer as "system," and adds it to the database. This article does not need to be submitted for review again. The reviewer will never see this already-reviewed reprinted article in the pending review list in the background.

[0107] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for improving the efficiency of manual review of text content, characterized in that, The method comprises the following steps: S1, detecting the content of the text to be audited to generate a detection result; S2, for each detection result, calculating a grouping hash value according to the detection type and key information, which is used to classify similar content into the same group; Wherein, the calculation method of the grouping hash is different according to the detection type, including: For sensitive information, "detection type + sentence + sensitive word + recommended word" is calculated by hash; For privacy information, "detection type + sentence + privacy content" is calculated by hash; For sensitive person, "detection type + sentence + name + position" is calculated by hash; For risk external link, "detection type + external link" is calculated by hash; For custom word, "detection type + sentence + custom word + recommended word" is calculated by hash; S3, for each detection result, calculating a unique identifier according to the detection type, media identifier, webpage link and key information, which is used for deduplication processing; Wherein, the calculation method of the unique identifier is different according to the detection type, including: For sensitive information, "detection type + mediaId + webpage link + sentence + sensitive word + recommended word" is calculated by hash; For privacy information, "detection type + mediaId + webpage link + sentence + privacy content" is calculated by hash; For sensitive person, "detection type + mediaId + webpage link + sentence + name + position" is calculated by hash; For risk external link, "detection type + mediaId + webpage link + external link" is calculated by hash; For custom word, "detection type + mediaId + webpage link + sentence + custom word + recommended word" is calculated by hash; S4, before the detection result is stored in the database, whether there is the same record in the database according to the unique identifier is queried, if there is, the current detection result is discarded, if there is not, whether there is a historical artificial auditing result that can be inherited is judged; S5, if there is a historical auditing result that can be inherited, the latest artificial auditing result is automatically inherited, including the auditing state, the risk type after auditing and the risk level after auditing, and the auditor is marked as system, and the auditing time is set as the current system time; Wherein, the step S5 specifically comprises: S51, according to the grouping hash of the current detection result, querying the database for the artificial auditing record; S52, arranging in reverse order according to the auditing time, selecting the latest artificial auditing result for inheritance; S53, setting the auditing state, the risk type after auditing and the risk level after auditing of the current detection result as the corresponding values of the inherited record; S54, marking the auditor as system, which is used to distinguish whether the partition is automatically audited by the system or manually audited by the human, and setting the auditing time as the current system time; S6, storing the detection result, the grouping hash, the unique identifier and the inherited auditing result in the database; S7, in the artificial auditing link, the data to be audited is displayed according to the grouping hash, and only one data in the same group is displayed, and the auditors batch audit the deduplicated data; For the detection results of the risk external link type, three batch review modes are supported during review, namely, review by main domain name, review by Host, and review by link. The review by main domain name is to review all external link detection results under the same main domain name together. The review by Host is to review all external link detection results under the same Host together. The review by link is to review according to the grouping hash, which is consistent with other detection types. S8, after the review is completed, the review state of all detection results under the grouping hash is updated.

2. The method for improving the efficiency of human review of text content of claim 1, wherein: The review state is set as the corresponding field value of the last manual review record. The post-review risk type is set as the corresponding field value of the last manual review record. The post-review risk level is set as the corresponding field value of the last manual review record.

3. The method for improving the efficiency of human review of text content of claim 1, wherein, The batch review step in the step S7 includes: S71, the reviewer queries the data to be reviewed according to the detection type; S72, the system displays the data to be reviewed after deduplication according to the grouping hash; S73, the reviewer selects one or more representative records for review operation; S74, the system marks all detection results in the group as reviewed according to the grouping hash of the selected records.

4. The method for improving the efficiency of human review of text content of claim 1, wherein: The detection results include but are not limited to one or more of sensitive information, private information, sensitive person, risk external link, and custom word.

5. The method for improving the efficiency of human review of text content of claim 1, wherein: The review state includes two states of "confirming that it is a problem that needs to be modified" and "confirming that it is not a problem that does not need to be modified".

6. The method for improving the efficiency of human review of text content of claim 1, wherein: The post-review risk type and risk level are dynamically set according to the specific detection type and review result.

Images