A method, system and device for multimedia content evaluation and quality inspection

Abstract

The present invention discloses a method, system and device for multimedia content evaluation and quality inspection. The present invention first creates an evaluation task according to the task configuration, wherein the task configuration includes the evaluation business party, the time range, quantity and conditions of the random inspection; then the created evaluation task is sent to the message queue; by monitoring the message queue, when it is found that there is a task in the message queue, the process production data is started, and the content that meets the task configuration information is extracted from the full amount of data of the evaluation business party or the audit record according to the task configuration information, and the snapshot content of the evaluation business party, the audit status when it was last audited, and the number of the evaluation business party are recorded in the evaluation table; after the evaluator completes the evaluation task, the audit status is recorded and an evaluation report is generated. By creating evaluation tasks of different dimensions, the present invention can improve the evaluation efficiency, truly and effectively reflect the audit quality, and comprehensively reflect the overall security of the platform content.

Description

Technical Field

[0001] The invention relates to a method, system and equipment for multimedia content evaluation and quality inspection, and belongs to the field of Internet applications. Background Art

[0002] In Internet content community products, it is increasingly important to ensure the compliance of online multimedia content (including text, pictures, videos, etc.). Once illegal and unlawful content is disseminated, the platform will face severe regulatory penalties, user complaints, public opinion risks, etc.

[0003] At present, there are many solutions for content review, but the quality inspection solutions for evaluating the review effect are very simple. The traditional solution is to manually export part of the content from the reviewed content for secondary review based on the reviewer dimension to evaluate the accuracy of the quality inspection review. This method has the following problems: First, the dimension is single, and it can only check the accidental injuries of the reviewers, but it is difficult to detect their omissions; second, it is difficult to retain and view the quality inspection results for a long time, and the work status of the quality inspectors cannot be ensured; third, for content products that users can update in real time, the extracted real-time content cannot truly and effectively reflect the quality of the review; fourth, it is impossible to simultaneously perform quality inspection on the content reviewed by the machine, and it is impossible to accurately reflect the overall security of the platform content. Summary of the invention

[0004] Purpose of the invention: In view of the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a method, system and equipment for multimedia content evaluation and quality inspection, which can improve the evaluation efficiency and the accuracy of evaluation and review personnel by creating evaluation tasks in different dimensions, truly and effectively reflect the review quality, and comprehensively reflect the overall security status of the platform content.

[0005] Technical solution: To achieve the above-mentioned purpose of the invention, the present invention provides a method for multimedia content evaluation and quality inspection, comprising the following steps:

[0006] Create an assessment task according to the task configuration; the task configuration includes the assessment business party, the time range, quantity and conditions of the random inspection; the conditions of the random inspection include at least the assessment type, and the assessment type is a full assessment or a reviewed assessment;

[0007] Send the created evaluation task to the message queue;

[0008] Monitor the message queue, and when a task is found in the message queue, start the process to produce data. Specifically, according to the task configuration information, extract the content that meets the task configuration information from the full data of the assessed business party or the audit record, and record it in the evaluation table for use by the assessor; the evaluation table records the snapshot content of the assessed business party, the audit status when it was last audited, and the number of the assessed business party;

[0009] After the evaluator completes the evaluation task, the audit status is recorded and compared with the audit status during the last audit to generate an evaluation report.

[0010] In a specific implementation scheme, the evaluation service party is an article, a user, a video or a comment.

[0011] Preferably, the sampling inspection conditions also include one or more of a specified business number, a specified keyword, a specified reviewer, and a specified review status.

[0012] Preferably, when the evaluation type is a full evaluation, the latest business content or the business content that hits the keyword is randomly extracted by configuring random extraction or specifying keywords; when the evaluation type is a reviewed evaluation, the reviewed business content is extracted by specifying reviewers and / or review status; the review status is passed or failed.

[0013] Preferably, when producing data, a scanner extracts data from a corresponding extraction source at a fixed rate according to task configuration and hands it over to an evaluation generator; the evaluation generator collects the data extracted by the scanner and writes the relevant content into an evaluation table.

[0014] Preferably, by comparing the evaluator's audit status with the first audit status of the auditor or machine audit, the accuracy rate, omission rate and false error rate of the assessment quality inspection are calculated, wherein the accuracy rate is the number of first audit statuses that are consistent with the evaluator's audit status / the total number of assessments; the omission rate is the number of first audit statuses that are passed but the evaluator's audit status is failed / the total number of assessments; the false error rate is the number of first audit statuses that are failed but the evaluator's audit status is passed / the total number of assessments.

[0015] Based on the same inventive concept, the present invention provides a system for multimedia content evaluation and quality inspection, comprising:

[0016] A task creation module is used to create an evaluation task according to the task configuration; the task configuration includes the evaluation business party, the time range, quantity and conditions of the random inspection; the conditions of the random inspection include at least the evaluation type, and the evaluation type is a full evaluation or a reviewed evaluation; and send the created evaluation task to the message queue;

[0017] The evaluation data production module is used to monitor the message queue, and when a task is found in the message queue, the process is started to produce data. Specifically, according to the task configuration information, the content that meets the task configuration information is extracted from the full data of the evaluation business party or the audit record, and recorded in the evaluation table for use by the evaluator; the evaluation table records the snapshot content of the evaluation business party, the audit status when it was last audited, and the number of the evaluation business party;

[0018] And the evaluation module is used to record the audit status after the evaluator completes the evaluation task, and generate an evaluation report by comparing it with the audit status at the last time it was audited.

[0019] Preferably, the evaluation data production module randomly extracts the latest business content or extracts the business content that hits the keywords according to the configuration information when the evaluation type is a full evaluation; when the evaluation type is a reviewed evaluation, the reviewed business content is extracted by specifying the reviewer and / or the review status; the review status is passed or failed.

[0020] Preferably, the evaluation module calculates the accuracy, omission rate and false error rate of the evaluation quality inspection by comparing the review status of the evaluator with the first review status of the auditor or machine audit, wherein the accuracy rate is the number of first review statuses that are consistent with the review status of the evaluator / the total number of evaluations; the omission rate is the number of first review statuses that are passed but the review status of the evaluator is failed / the total number of evaluations; the false error rate is the number of first review statuses that are failed but the review status of the evaluator is passed / the total number of evaluations.

[0021] Based on the same inventive concept, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the multimedia content evaluation and quality inspection method when loaded into the processor.

[0022] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0023] 1) This invention solves the problem of single dimension in traditional quality inspection schemes, and can create custom assessment tasks to improve assessment efficiency. Through tasks of different dimensions, the accuracy of auditors can be assessed with high quality, while ensuring the quality of assessors.

[0024] 2) The present invention divides the evaluation into full evaluation and reviewed evaluation. Through random sampling technology, it can accurately evaluate the accuracy of manual review and machine review (false positive rate, omission rate), and can comprehensively reflect the security issues of platform content.

[0025] 3) The present invention extracts snapshot content from the audit record, which solves the quality inspection problem of content products updated in real time, and can retain the audit results of the content snapshot updated by the user each time and can be randomly extracted.

[0026] 4) Based on the evaluation task of the present invention, a quality inspection work report can be automatically generated to visually reflect the quality inspection work effect and ensure the effectiveness of the quality inspection work. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0028] Figure 2 This is a screenshot of the statistical report page in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments.

[0030] In order to fully and accurately reflect the security compliance issues of platform content, such as Figure 1 As shown, the embodiment of the present invention discloses a method for multimedia content evaluation and quality inspection. The main middleware uses MySQL and Redis, and the implementation language is Golang. The specific steps are as follows:

[0031] S1: Create an assessment task according to the task configuration. The task configuration mainly includes the assessment business party, the time range, quantity and conditions of the sampling, etc. The specific instructions are as follows:

[0032] (1) Business parties evaluated: The business parties evaluated in this evaluation mainly refer to articles, users, videos, comments, etc.

[0033] (2) Time range of random inspection

[0034] (3) Number of random inspections

[0035] (4) Sampling conditions: Specify the evaluation type, which can be divided into full evaluation (general market) and reviewed evaluation (snapshot); or specify other conditions, such as business number, keyword, reviewer, review status, etc.

[0036] Taking the evaluation article as an example (the following examples are based on the article business dimension), the sampling conditions are explained.

[0037] Specify a business number. For example, if an article number is specified, obtain the article details by number to generate an assessment task.

[0038] Specify keywords. For example, if keywords are specified, the article library will be scanned sequentially. If the keywords are found, the article will be added to this evaluation task.

[0039] Specify the reviewer. For example, for the reviewed articles, the information of the relevant reviewers (including manual or machine) will be recorded. The reviewer can be given a number. By scanning the review record table, the relevant review records of the person can be found. The article number and the review status at that time (passed / failed) are saved, and this information is added to this evaluation task.

[0040] Specify the review status. For example, if the review status is specified, obtain the article information of the corresponding review status by scanning the review record table and add it to this assessment task.

[0041] Specify the evaluation type: Full evaluation: randomly select articles from the full article library to generate evaluation tasks; Reviewed evaluation: randomly select articles from the reviewed article records, and attach the review status of the articles to generate evaluation tasks.

[0042] In this embodiment, when the system is implemented, for article extraction evaluation, the corresponding data structure is as follows:

[0043] `id`int(10)unsigned NOT NULL AUTO_INCREMENT COMMENT'id',

[0044] `creator_id` int(10) unsigned NOT NULL DEFAULT'0' COMMENT'Creator',

[0045] `assess_status`tinyint(3)NOT NULL DEFAULT'0'COMMENT'Assessment task status: 0 normal; 1 creating; 2 deleted; 3 creation failed',

[0046] `remark`varchar(256)NOT NULL DEFAULT"COMMENT'Remarks',

[0047] `create_time` int(10) unsigned NOT NULL COMMENT' creation time',

[0048] `attr` varchar(2048) NOT NULL DEFAULT "COMMENT 'Evaluation attribute'

[0049] Evaluation attribute json sample:

[0050] {"is_snapshot":true,"reviewers":[5,78,305,320,328,329,331,332,334],"state":0,"s_time":1621327023,"e_time":1623919023,"total":100000}

[0051] Among them, "is_snapshot" marks the evaluation type, true means it is a reviewed evaluation; "reviewers" indicates the reviewer number related to this extraction; "state" indicates the review status of the extracted article original data; "s_time" and "e_time" respectively indicate the start and end creation time of the extracted article.

[0052] S2: Message queue monitoring task creation. After the task creator configures and submits the assessment task at the front desk of the operation center, the back end will record the configuration of the task and the specific operator information in MySQL after receiving the task, and send the primary key ID of the current MySQL record to the redis message queue. The back end monitors the redis message queue through the redis lpop instruction. Once the task queue is found to have data, it enters S3 to start a background process to produce data according to the assessment configuration recorded in MySQL.

[0053] S3: Produce evaluation data. According to the task configuration information, extract the content that meets the task configuration information from the full data of the evaluation business party or the audit record, and record it in the evaluation table for the use of the evaluator. The specific production process is: first, the scanner extracts data from the corresponding extraction source at a fixed rate according to the task configuration and gives it to the evaluation generator; then the evaluation generator collects the data extracted by the scanner and writes the relevant content into the evaluation table.

[0054] Specifically, the scanner will extract relevant data sources from the source according to the task configuration. Scanner data structure:

[0055]

[0056] Where da represents the database instance, cond represents the extraction condition, batchsize represents the number of extractions each time, firstid represents the extraction start id, curid represents the current extracted cursor value that saves the current scanned position, and lastid represents the extraction end id.

[0057] The scanner has two methods, Next() and Load(). Next() is mainly used to compare the scan cursor value with the extraction end id. If the scan cursor value > the extraction end id, the scan process is terminated; otherwise, the Load() method is called. The main function of Load() is to extract data from the MySQL data source according to the extraction conditions in a loop and hand it over to the evaluation generator.

[0058] Evaluation generator data structure:

[0059]

[0060] Where assessID indicates the number of this assessment task, record is the extraction source when the assessment type is snapshot, content is the extraction source when the assessment type is large disk, scanner indicates the scanner, which extracts data from the extraction source at a fixed rate, done indicates whether it is completed, and is used to notify the process to exit. running indicates the running status of the generator, which is used to interrupt the current operation. mu is a concurrent lock, which allows users to implement multiple scanners to concurrently and safely operate the database.

[0061] The evaluation generator has methods such as write(), buildStat(), writeStat(), importSnapshots(), importRandomArticles(), and importByKeywords(). The importSnapshots() method is suitable for snapshot evaluation (i.e., reviewed evaluation). The extraction source is the business review record and is sent to the downstream to call the write() method. The importRandomArticles() method is suitable for large-scale evaluation (i.e., full-volume evaluation). It can randomly extract business content based on dimensions such as time, machine service / personnel, channel, and content dissemination volume (reading volume, forwarding volume, etc.) and send it to the downstream to call the write() method. The importByKeywords() method is suitable for keyword extraction evaluation. The extraction logic is to extract qualified content according to filtering conditions such as timeline and status, and match the selected keywords one by one. If the keywords are hit, the content is collected and sent to the downstream. The write() method mainly collects data from the upstream (scanner), and records the data of relevant content (article snapshot content, review status of the article when it was last reviewed, and article content number) in the evaluation table for evaluators to perform evaluation tasks. The buildStat() method is used to initialize the evaluation report when the task is generated, and is used to record the original data of this task: mainly to record part of the data of the original audit status in the snapshot evaluation, that is, how many of the original status in this task are blocked (failed) and how many are passed. The writeStat() method is used to record the evaluation operation status when the auditor is performing the evaluation task, that is, how many of the evaluation are blocked (failed) and how many are passed.

[0062] S4: After the data is generated, the assessor can conduct the assessment task through the assessment background. After the task is completed, the statistical results can be viewed on the data statistics report page. Figure 2 As shown in the figure, in the statistical report of the article evaluation report, the first line is the original article review data, and the second line is the total amount of evaluation modification data, which refers to the operation of this evaluation data. These two lines can be used to compare errors and omissions, and all data can be clicked to jump to view the details. In addition, for a certain evaluation task, a chart can be displayed to track the work of the quality inspectors. It is normal for the number of quality inspectors to increase evenly over time. If the evaluation level does not change over time, it means that the work of the quality inspectors is abnormal.

[0063] By comparing the audit records at the time of capture (first audit status) with the audit records of the final assessor's operation, the accuracy, omission rate, and false positive rate of this assessment can be obtained. For example, the accuracy rate can be obtained by the number of first audit statuses that are consistent with the assessor's audit status / total number of assessments; the omission rate can be obtained by the number of first audit statuses that are passed and the assessor's audit status is not passed / total number of assessments; the false positive rate can be obtained by the number of first audit statuses that are not passed and the assessor's audit status is passed / total number of assessments.

[0064] Based on the method of the present invention, the quality of audit work can be evaluated through evaluation tasks of different dimensions to reflect the security issues of the platform content. For example, the snapshot content of the articles reviewed by auditor A within a period of time, such as 100 articles, can be randomly selected to evaluate their accuracy and false positive rate. 1,000 online market contents can be randomly selected within a period of time to see the overall omission rate and the security of the online environment. In addition, it can be combined with the quality inspection situation to analyze whether it is manual or machine detection omission, further analyze the data, improve the audit rules, and improve the overall security of the platform.

[0065] Based on the same inventive concept, a system for multimedia content evaluation and quality inspection provided by an embodiment of the present invention includes: a task creation module, which is used to create an evaluation task according to the task configuration; the task configuration includes the evaluation business party, the time range, quantity and conditions of the random inspection; an evaluation data production module, which is used to monitor the message queue, and when a task is found in the message queue, the process production data is started, and the content that meets the task configuration information is extracted from the full data of the evaluation business party or the audit record according to the task configuration information, and recorded in the evaluation table for use by the evaluator; and an evaluation module, which is used to record the audit status after the evaluator completes the evaluation task, and generate an evaluation report by comparing it with the audit status at the last time it was audited. The specific implementation of each module refers to the above method embodiment, which will not be repeated here.

[0066] Based on the same inventive concept, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the multimedia content evaluation and quality inspection method when loaded into the processor.

Claims

1. A method for evaluating and inspecting multimedia content, characterized in that: The steps include: Create an assessment task according to the task configuration; the task configuration includes the assessment business party, the time range, quantity and conditions of the random inspection; the random inspection conditions at least include the assessment type, and the assessment type is a full assessment and a reviewed assessment; the assessment business party is an article, video or comment; Send the created evaluation task to the message queue; Monitor the message queue, and when there is a task in the message queue, start the process to produce data. According to the task configuration information, extract the content that meets the task configuration information from the full data of the assessed business party and the audit records, and record it in the assessment table for use by the assessors; when producing data, the scanner extracts data from the corresponding extraction source at a fixed rate according to the task configuration and hands it over to the assessment generator; the assessment generator collects the data extracted by the scanner and writes the relevant content into the assessment table; the scanner data structure includes the database instance, extraction conditions, the number of each extraction, the extraction start id, the current extraction cursor value and the extraction end id. The methods include Next() and Load(). Next() is used to compare the scan cursor value with the extraction end id. If the scan cursor value > the extraction end id, the scanning process is terminated; otherwise, the Load() method is called. The Load() loop extracts data from the MySQL data source according to the extraction conditions and gives it to the assessment generator; the assessment generator data structure includes the assessment task number, extraction source, scanner, completion mark, running status and concurrent lock. The methods include write(), buildStat(), writeStat(), importSnapshots(), importRandomArticles() and importByKeywords(). The importSnapshots() method is applicable to the reviewed assessment. The extraction source is the business review record and is sent to the downstream to call the write() method. The importRandomArticles() method is applicable to the full assessment. The business content is randomly extracted and sent to the downstream to call the write() method. The importByKeywords() method is applicable to the keyword extraction assessment. The extraction logic is to extract the qualified content according to the screening conditions and match the selected keywords one by one. If the keyword is hit, the content is collected and sent to the downstream. The write() method collects the upstream data and records the data of the relevant content in the assessment table for the assessor to perform the assessment task; The buildStat() method is used to initialize the evaluation report when the task is generated and record the original data of the task. The writeStat() method is used to record the evaluation operation status when the auditor performs the evaluation task. The evaluation table records the snapshot content of the evaluated business party, the audit status when it was last audited, and the number of the evaluated business party. When the evaluation type is a full evaluation, the business content that hits the keyword is extracted by specifying keywords. When the evaluation type is an audited evaluation, the audited business content is extracted by specifying the auditor and / or the audit status. The audit status is passed or failed. After the evaluator completes the evaluation task, the audit status is recorded and compared with the audit status during the last audit to generate an evaluation report.

2. A method for multimedia content evaluation and quality inspection according to claim 1, characterized in that: By comparing the evaluator's audit status with the first audit status of the auditor or machine audit, the accuracy rate, omission rate and false error rate of the assessment quality inspection are calculated, where the accuracy rate is the number of first audit statuses that are consistent with the evaluator's audit status / total number of assessments; the omission rate is the number of first audit statuses that are passed but the evaluator's audit status is failed / total number of assessments; the false error rate is the number of first audit statuses that are failed but the evaluator's audit status is passed / total number of assessments.

3. A system for multimedia content evaluation and quality inspection, characterized in that: include: A task creation module is used to create an evaluation task according to the task configuration; the task configuration includes the evaluation business party, the time range, quantity and conditions of the random inspection; the random inspection conditions include at least the evaluation type, the evaluation type is a full evaluation and a reviewed evaluation, and the evaluation business party is an article, video or comment; and send the created evaluation task to the message queue; The evaluation data production module is used to monitor the message queue. When a task is found in the message queue, the process is started to produce data. According to the task configuration information, the content that meets the task configuration information is extracted from the full data of the evaluation business party and the audit record, and recorded in the evaluation table for use by the evaluator; when producing data, the scanner extracts data from the corresponding extraction source at a fixed rate according to the task configuration and hands it over to the evaluation generator; the evaluation generator collects the data extracted by the scanner and writes the relevant content into the evaluation table; the scanner data structure includes the database instance, extraction conditions, the number of each extraction, the extraction start id, the current extraction cursor value and the extraction end id. The methods include Next() and Load(). Next() is used to compare the scan cursor value with the extraction end id. If the scan cursor value > the extraction end id, the scanning process is terminated; otherwise, the Load() method is called. The Load() loop extracts data from the MySQL data source according to the extraction conditions and gives it to the assessment generator; the assessment generator data structure includes the assessment task number, extraction source, scanner, completion mark, running status and concurrent lock. The methods include write(), buildStat(), writeStat(), importSnapshots(), importRandomArticles() and importByKeywords(). The importSnapshots() method is applicable to the reviewed assessment. The extraction source is the business review record and is sent to the downstream to call the write() method. The importByKeywords() method is applicable to keyword extraction assessment. The extraction logic is to extract qualified content according to the screening conditions and match the selected keywords one by one. If the keywords are hit, the content is collected and sent to the downstream. The write() method collects upstream data and records the data of related content in the assessment table for the assessors to perform the assessment task. The buildStat() method is used to initialize the evaluation report when the task is generated and record the original data of the task. The writeStat() method is used to record the evaluation operation status when the auditor performs the evaluation task. The evaluation table records the snapshot content of the evaluated business party, the audit status when it was last audited, and the number of the evaluated business party. When the evaluation type is a full evaluation, the business content that hits the keyword is extracted by specifying keywords. When the evaluation type is an audited evaluation, the audited business content is extracted by specifying the auditor and / or the audit status. The audit status is passed or failed. And the evaluation module is used to record the audit status after the evaluator completes the evaluation task, and generate an evaluation report by comparing it with the audit status at the last time it was audited.

4. A multimedia content evaluation and quality inspection system according to claim 3, characterized in that: The evaluation module calculates the accuracy, omission rate and false error rate of the evaluation quality inspection by comparing the review status of the evaluator with the first review status of the reviewer or machine review, wherein the accuracy rate is the number of first review statuses that are consistent with the review status of the evaluator / the total number of evaluations; the omission rate is the number of first review statuses that are passed but the review status of the evaluator is failed / the total number of evaluations; the false error rate is the number of first review statuses that are failed but the review status of the evaluator is passed / the total number of evaluations.

5. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the method for evaluating and inspecting multimedia content according to claim 1 or 2 is implemented.

Images