A model-assisted news media thought chain sample semi-automatic construction method

Abstract

The present application relates to the technical field of data labeling and machine learning model training dataset construction, and specifically provides a model-assisted news media thinking chain sample semi-automatic construction method, aiming to solve the balance problem between the professional nature, quality controllability, interpretability and labeling efficiency of the existing news media data labeling scheme. The present application takes model-assisted semi-automation as the design concept, and divides the overall process into two stages: the first stage is model-assisted automatic labeling and manual quality checking, including field self-adaptive fine-tuning model preliminary labeling and labeling personnel correction cleaning and expert review; the second stage is large model interpretable review and expert final review, including multi-modal base large model review and thinking chain generation and expert final review locking, to obtain the final labeled sample set. Through the progressive cooperation of model and manual, the present application realizes the semi-automatic construction of news media professional dimension label labeling sample set, and provides interpretability for the labeling process through thinking chain reasoning.

Description

Technical Field

[0001] This invention belongs to the field of data annotation and machine learning model training dataset construction technology, specifically involving a model-assisted semi-automatic method for constructing news media thought chain samples. It is applicable to scenarios such as constructing professional training datasets for multimedia content such as news videos, news images, and news texts, training news content understanding models, and artificial intelligence applications. Background Technology

[0002] With the deepening application of deep learning and computer vision technologies in the news media field, building high-quality and reliable professional training datasets for news media has become a critical infrastructure requirement. Data annotation, as the core link in dataset construction, directly affects the performance and reliability of downstream models (such as multimodal analysis models, manuscript writing models, intelligent editing models, etc.).

[0003] Currently, data annotation for news media content mainly follows these technical approaches:

[0004] (1) General annotation framework method. This method uses annotation standards and tools for general fields to annotate news media data. The dimensions that this method focuses on usually include general attributes such as time, location, people, events, objects, actions, and atmosphere. However, it lacks the ability to annotate for news-specific dimensions, resulting in insufficient applicability of the annotation results in news-specific scenarios.

[0005] (2) Automated annotation method using general models. This method uses large-scale general models (such as visual-language models or multimodal large models) to perform end-to-end automatic annotation of news media data. This method relies on the generalization ability of the model and can complete the annotation of large-scale data with high efficiency. However, due to the limited semantic understanding of professional concepts in the news field by the model, the annotation results have a high error rate and uncertainty, especially in the recognition of fine-grained professional features, which is prone to systematic bias.

[0006] (3) Crowdsourced annotation method. Annotation tasks are distributed to a large number of non-professional annotators through a crowdsourcing platform, and quality is controlled by cross-validation mechanism. This method is low-cost and scalable, but the annotators lack professional knowledge in the news field, and the consistency and accuracy of annotation are difficult to guarantee.

[0007] (4) Domain expert full annotation method. All data is manually annotated by experts in journalism and communication or related fields. This method can ensure high professional accuracy, but it is costly, time-consuming, and has low annotation efficiency, making it difficult to meet the annotation needs of large-scale datasets.

[0008] Existing annotation schemes not only lack a systematic, multi-layered quality verification and traceability mechanism, making it impossible to transform annotation quality from static inspection to dynamic continuous optimization; but they also fail to utilize the logical reasoning capabilities of large language models for in-depth consistency review of annotation results, resulting in a lack of interpretability in the annotation process and an inability to form a reliable semi-automatic sample construction chain. Summary of the Invention

[0009] This invention provides a model-assisted semi-automatic method for constructing news media thought chain samples, aiming to solve the balance problem between professionalism, quality controllability, interpretability, and annotation efficiency in existing news media data annotation schemes.

[0010] The technical solution adopted in this invention is as follows:

[0011] This invention provides a model-assisted semi-automatic method for constructing news media thought chain samples, comprising:

[0012] A domain-adaptive fine-tuning model was used to initially label the news media sample set with multi-dimensional tags, resulting in a draft labeled sample set.

[0013] The draft annotation sample set is manually reviewed and corrected, and a sample cleaning operation is performed to obtain a valid annotation sample set, as well as a comparison record between the draft annotation sample set and the valid annotation sample set;

[0014] The quality of the valid labeled sample set is reviewed through an expert sampling review mechanism to obtain the labeling results; the labeling results include the labeled draft sample set, manual correction information, review conclusions and quality monitoring data;

[0015] Based on the aforementioned news media sample set, the annotation results are logically reviewed for factual accuracy, compliance, and consistency using the reasoning capabilities of a general large model, resulting in a thought chain reasoning text and a review report.

[0016] Based on the annotation results, the comparison records, the thought chain reasoning text, and the review report, experts will make a final review decision to determine the final annotated sample set.

[0017] Further solutions: The expert sampling review mechanism includes:

[0018] A hybrid sampling strategy is used to extract samples from the set of valid labeled samples to construct an expert review sample set;

[0019] Perform expert standard re-annotation on all samples in the expert review sample set to obtain the expert re-annotated sample set;

[0020] The expert-re-annotated sample set and the effective-annotated sample set are compared dimension by dimension under multi-dimensional labels to calculate the annotation consistency rate.

[0021] A quality monitoring report is generated based on the labeling consistency rate; the quality monitoring report includes the labeling consistency rate of each dimension of labels, information analysis of inconsistent labeling samples, and labeling quality trend assessment.

[0022] Based on the above scheme, by performing targeted sampling verification on the effective labeled sample set, a gold standard (the expert relabeled sample set) is established and the labeling consistency rate is calculated. The labeling quality is evaluated with quantitative indicators, so that quality monitoring is closely coupled with the labeling process, and real-time quantitative evaluation of labeling quality is achieved.

[0023] A further proposed solution: the hybrid sampling strategy consists of two parts: disputed sample sampling and random supplementary sampling.

[0024] The content of the disputed sample sampling is as follows: based on the comparison record, samples that are inconsistent with the annotation of the annotation draft sample set are selected from the set of valid annotated samples;

[0025] The random supplementary sampling involves: selecting samples from the valid labeled sample set that are consistent with the labeled draft sample set based on the comparison record, and randomly sampling them according to a preset ratio.

[0026] A further solution: The annotation consistency rate is defined as the proportion of the number of samples in the expert re-annotated sample set that are consistent with the annotations in the effective annotation sample set to the total number of samples in the expert re-annotated sample set.

[0027] A further solution: The expert sampling review mechanism also includes: providing feedback based on the quality monitoring report to manually re-label samples whose labeling consistency rate does not meet the standard.

[0028] Based on the above scheme, the quality of labeling can be optimized through the feedback from the quality monitoring report.

[0029] Further measures include: performing logical review on the annotation results, including:

[0030] Construct logical review prompts; the logical review prompts include role settings, task descriptions, the news media sample set, the annotation results, the dimensions of logical review, and the output format in the form of a thought chain; wherein, the dimensions of logical review include at least: annotation consistency review between dimension tags, annotation matching degree review, annotation granularity and boundary review, and annotation definition compliance review;

[0031] Based on the logical review prompts, the general model performs logical review on the annotation results according to the dimensions of the logical review.

[0032] According to the output format, the general large model outputs thought chain reasoning text; wherein, the content of the thought chain reasoning text includes at least: dimension label analysis, statement of reasoning basis, explanation of inconsistencies in labeling, and analysis of the reasons for inconsistencies in labeling;

[0033] The review conclusion is obtained based on the thought chain reasoning text, and the thought chain reasoning text and the review conclusion are integrated to obtain the structured review report.

[0034] Based on the above scheme, firstly, when performing logical review on the annotation results, the dimensions of the logical review are set so that the output thought chain reasoning text can closely conform to the annotation specifications in the news field for targeted reasoning, which can improve the professionalism and credibility of the review conclusion.

[0035] Secondly, for the thought chain reasoning text, the analysis of the dimensional tags and the statement of reasoning basis make the review logic visible to the public, facilitating subsequent expert final review; through explanations and analysis of inconsistencies in annotations, the essence of annotation defects can be revealed, reducing repeated disputes. The thought chain reasoning text makes the logical review process of the general model interpretable, and also makes the annotated samples interpretable and reliable.

[0036] Further options: The aforementioned final expert ruling includes:

[0037] For samples marked as disputed, the final review experts will comprehensively review the marking results, the comparison records, the thought chain reasoning text, and the review report, and make a decision to maintain the original marking, correct the marking, or mark it as indeterminate, and record the basis for the decision.

[0038] After confirming that all the disputed samples have been processed, the final labeled sample set is locked and archived; wherein, the content of the final labeled sample set includes: the final labeled label, labeler identification, quality monitoring report, thought chain reasoning text, and record of the basis for the ruling for each sample in the news media sample set.

[0039] Further proposal: Samples marked as disputed must include at least one of the following:

[0040] The general large model mentioned in the audit conclusion was determined to be a sample that failed or was questionable;

[0041] When manually reviewing and correcting the labeled draft sample set, samples with a skip rate higher than a preset threshold are skipped.

[0042] In addition, samples whose labels on any dimension are inconsistent between the expert-relabeled sample set and the effective labeled sample set.

[0043] A further solution: When using a domain-adaptive fine-tuning model to initially label the news media sample set with multi-dimensional labels, the dimensional labels include, but are not limited to: shot size, shooting angle, composition method, camera movement method, and the start and end times of the camera movement.

[0044] Based on the above scheme, the present invention first uses the domain adaptive fine-tuning model to perform initial automated annotation, providing a structured starting point for subsequent manual review and correction, and improving the overall process efficiency.

[0045] Further solutions: The manual review and correction of the annotation draft sample set includes: reviewing whether the dimension labels of each sample in the annotation draft sample set are correct, correcting incorrect label values, and supplementing missing dimension labels;

[0046] The sample cleaning operation includes: removing samples from the news media sample set that do not meet the labeling requirements, marking samples from the news media sample set that have abnormal sources, and standardizing the format and naming conventions of the news media sample set.

[0047] The beneficial effects of this invention are as follows:

[0048] (1) This invention can be divided into two stages: the first stage takes the automatic annotation of the adaptive fine-tuning model of the domain as the starting point for efficiency, and completes the effective annotation and quality verification through manual correction and cleaning and expert sampling review mechanism; the second stage uses the general large model to conduct interpretable logic review and generate thought chain reasoning text, and finally locks it by expert review.

[0049] This invention designs a progressive architecture that combines model assistance and human guidance, forming a closed-loop link between the two stages: "model-assisted initialization - human quality verification - large model interpretability review - expert authoritative confirmation". It highlights the core concept of model-assisted semi-automation and achieves organic synergy between machine efficiency, human breadth, expert depth, large model logic and human authority.

[0050] (2) This invention utilizes the reasoning ability of the general large model to perform interpretable logical review, providing a complete thought chain review record for each labeled news media sample, so that the constructed training dataset (i.e. the final labeled sample set) has interpretability and reliability, fully reflecting the core value of model-assisted semi-automation.

[0051] (3) The comparison record of the annotation draft sample set and the effective annotation sample set in this invention, the annotation results and the thought chain reasoning text can realize full-link traceability from the domain adaptive fine-tuning model automatic annotation to manual review and correction, expert quality review, and then to the general large model thought chain review, which facilitates the final review decision by experts. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of a model-assisted semi-automatic method for constructing a news media thinking chain sample according to the present invention;

[0054] Figure 2 This is a schematic diagram of the interface of the annotation platform in this invention;

[0055] Figure 3 This is a flowchart of the expert sampling review mechanism in this invention;

[0056] Figure 4 This is a flowchart of the logical review of the annotation results by the multimodal base large language model in this invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the protection scope of the present invention.

[0058] like Figure 1 As shown, this embodiment provides a model-assisted semi-automatic method for constructing news media thought chain samples. Based on the core design concept of model-assisted semi-automation, the method divides the construction process of professional training samples for news media into two sequentially executed, progressively advancing stages:

[0059] Phase 1:

[0060] S100. Use a domain-adaptive fine-tuning model to perform initial labeling of the news media sample set with multi-dimensional labels to obtain a labeled draft sample set;

[0061] S200. Manually review and revise the annotation draft sample set, perform sample cleaning operation, obtain a valid annotation sample set, and a comparison record between the annotation draft sample set and the valid annotation sample set;

[0062] S300. The quality of the effective labeled sample set is reviewed through an expert sampling review mechanism to obtain the labeled results; the labeled results include the labeled draft sample set, manual correction information, review conclusions and quality monitoring data;

[0063] Phase Two:

[0064] S400. Based on the news media sample set, the annotation results are logically reviewed for factual accuracy, compliance, and consistency through the reasoning capabilities of a general large model to obtain the thought chain reasoning text and review report.

[0065] S500. Based on the annotation results, the comparison records, the thought chain reasoning text, and the review report, experts make a final review decision and lock in the final annotated sample set.

[0066] A more specific example of step S100 in the first stage is as follows:

[0067] The domain-adaptive fine-tuning model is obtained by fine-tuning on a large-scale multimodal corpus (including news videos, news images and news text data), and the domain-adaptive fine-tuning model has the basic ability to recognize news professional features.

[0068] The domain-adaptive fine-tuning model receives a news media sample set (including video frame sequences, images, or text) and outputs a label prediction result for each sample in the news media sample set according to a preset news professional dimension label system.

[0069] The label prediction results are organized by sample to obtain the label draft sample set.

[0070] Preferably, the dimension labels include, but are not limited to: shot type (label values: long shot, full shot, medium shot, close-up, extreme close-up, etc.), shooting angle (label values: front, back, oblique, etc.), composition method (label values: symmetrical composition, rule of thirds composition, central composition, etc.), camera movement method (label values: push, pull, pan, tilt, fixed, etc.) and corresponding start and end times.

[0071] Each label prediction result includes the dimension label name, predicted label value, and domain adaptive fine-tuning model version information. For video samples, the dimension label for camera movement also includes the start and end timestamps of each camera movement. The output format of each label prediction result is a structured data format (such as JSON).

[0072] The annotation draft sample set is the output of step S100. Each record in the annotation draft sample set includes the identifier of the received news media sample, the annotation dimension label predicted by the domain adaptive fine-tuning model, and the version information of the domain adaptive fine-tuning model.

[0073] In optional cases, the domain-adaptive fine-tuning model can be replaced by a visual understanding model, a multimodal large model, or a dedicated news media analysis model that also has the basic ability to identify the professional characteristics of news, as long as it can perform initial labeling on the news media sample set.

[0074] The following is an example of the output of the domain adaptive fine-tuning model's prediction result for the camera movement dimension label of a video sample; the output example is in structured JSON format, containing the label value predicted by the domain adaptive fine-tuning model, timestamp information, and the version identifier of the domain adaptive fine-tuning model.

[0075] {

[0076] "predictions": [

[0077] {

[0078] "ranges": [ {"start": 1,"end": 166} ],

[0079] "choices": ["fixed"],

[0080] "model_version": "V1.1"

[0081] }

[0082] ],

[0083] "data": {

[0084] "video": " / data / localfiles / ?d=MediaResourceDataset / videodata / 1.mp4"

[0085] }

[0086] }

[0087] In the above output example, the "predictions" array is the set of prediction results obtained by the domain adaptive fine-tuning model after recognizing the camera movement mode of the video sample; each prediction result contains the "ranges" array (start and end timestamps of the camera movement mode in the video, in seconds), the "choices" array (label values ​​of the camera movement mode), and "model_version" (version of the domain adaptive fine-tuning model); "data" records the path of the labeled video sample.

[0088] A more specific example of step S200 in the first stage is as follows:

[0089] The manual review and correction of the annotation draft sample set output in step S100 includes: reviewing whether the dimension labels of each sample in the annotation draft sample set are correct, correcting incorrect label values, and supplementing missing dimension labels.

[0090] The sample cleaning operation includes: removing samples from the news media sample set that do not meet the labeling requirements (e.g., blurry images, damaged videos, or incomplete text), marking samples from the news media sample set that have abnormal sources, and standardizing the format and naming conventions of the news media sample set.

[0091] It should be noted that the personnel involved in step S200 are professional annotators who have received systematic training in journalism and communication studies or related fields. Before reviewing the sample set of the draft annotations, the annotators must pass annotation specification training and consistency testing to ensure a consistent understanding of the annotation standards for news professional dimension tags. The annotation specifications clearly define the annotation standards for each dimension tag, the rules for handling boundary cases, and the annotation quality requirements.

[0092] The output annotation draft sample set is imported into the annotation platform, the interface of which is as follows: Figure 2 As shown, the interface of the annotation platform displays: a sample from the annotation draft sample set, and below the sample are the label values ​​of each dimension of the domain adaptive fine-tuning model annotation. Annotators can directly modify the sample in the annotation draft sample set on the annotation platform interface. The annotation platform automatically records each modification operation, forming a modification log.

[0093] Based on the above scheme, during the process of reviewing and revising the sample set of the annotation draft, the annotation platform provides the annotation personnel with a "skip annotation" option to handle situations where the annotation personnel cannot determine the label value (e.g., the sample quality is insufficient to support the judgment, the sample content exceeds the scope covered by the annotation specification, etc.).

[0094] For sample i, define its skip rate. for:

[0095] ;

[0096] Among them, the This indicates the number of annotators who selected the "Skip Annotation" option for sample i; The skip rate represents the total number of annotators involved in annotating sample i; The value range is [0, 1].

[0097] Set a dispute threshold and discard threshold And satisfy ; and The specific value can be set according to the difficulty of the annotation task, the sample quality requirements, and the size of the annotation team;

[0098] The sample classification rules are as follows:

[0099] when If the sample is deemed a normal sample, the labeling results will proceed to the next step.

[0100] when If the sample is determined to be a disputed sample, it will be included in the expert quality review sampling pool in step S300 for key review.

[0101] when If the sample is deemed invalid, it is removed from the set of valid labeled samples and the reason for removal is marked.

[0102] The annotation platform automatically generates the comparison records, recording the differences between the annotation results of the domain adaptive fine-tuning model in step S100 and the results corrected by professionals for each sample. The comparison records include: sample identifier, dimension label, label value of the domain adaptive fine-tuning model annotation, manually corrected label value, modification time, and the identifier of the person making the modification. These comparison records serve as the basic data for annotation quality traceability and also as the basis for the mixed sampling strategy in the expert sampling review in step S300.

[0103] A more specific example of step S300 in the first stage is as follows:

[0104] The annotation platform has a built-in expert sampling review mechanism. After the annotators review and clean all the annotated draft samples in the annotated draft sample set, the expert sampling review mechanism is activated to quantitatively review the annotation quality of the annotators (i.e. the annotation quality of the effective annotation sample set).

[0105] Specifically, such as Figure 3 As shown, the steps of the expert sampling review mechanism include:

[0106] S301. Using a hybrid sampling strategy, samples are drawn from the set of valid labeled samples to construct an expert review sample set;

[0107] As a preferred option, the hybrid sampling strategy can consist of two parts: disputed sample sampling and random supplementary sampling.

[0108] The content of the disputed sample sampling is as follows: based on the comparison record, samples that are inconsistent with the annotation of the annotation draft sample set are selected from the set of valid annotated samples; it may also include disputed samples based on the skip rate.

[0109] The random supplementary sampling involves selecting samples from the valid labeled sample set that are consistent with the labeled draft sample set based on the comparison record, and then randomly sampling them at a preset ratio (5%–20%).

[0110] In addition, the ratio between the disputed sample and the random supplementary sample can be adjusted according to the actual application scenario (e.g., 1:1 or 2:1, etc.), and other sampling dimensions can also be added (e.g., stratified sampling by sample source, stratified sampling by labeled personnel, etc.).

[0111] S302. Perform expert standard re-annotation on all samples in the expert review sample set to obtain an expert re-annotated sample set; wherein, the experts are experts in journalism and communication or related fields, with many years of practical experience in journalism or a background in journalism research; when re-annotating the samples in the expert review sample set, the experts are not affected by the previous annotation results (domain adaptive fine-tuning model annotation results and annotation results of the annotators), and make independent judgments based solely on the sample content and annotation specifications;

[0112] S303. Compare the expert re-annotated sample set with the effective annotated sample set under multi-dimensional labels, and calculate the annotation consistency rate.

[0113] Preferably, the annotation consistency rate is defined as the proportion of the number of samples in the expert-re-annotated sample set that are consistent with the annotations in the validly annotated sample set to the total number of samples in the expert-re-annotated sample set. The threshold for the annotation consistency rate can be set according to actual conditions, for example, 0.87. When the annotation consistency rate of a sample in the validly annotated sample set is greater than 0.87, it means that the sample can enter the subsequent process; otherwise, it is recorded as having an unsatisfactory annotation consistency rate and needs to be re-annotated.

[0114] To further eliminate the bias caused by random consistency, a weighted Kappa coefficient is used to quantitatively evaluate the degree of consistency in annotation among different annotation subjects (annotators and experts):

[0115] Weighted Kappa coefficient The value range of is [-1, 1]. A higher numerical value indicates stronger labeling consistency. In specific implementations, the following general interpretation standard can be adopted:

[0116] 0.0~0.20: Slightly consistent;

[0117] 0.21~0.40: Generally consistent;

[0118] 0.41~0.60: moderately consistent;

[0119] 0.61~0.80: Highly consistent;

[0120] 0.81~1.00: Almost identical.

[0121] For any dimension label, the weighted Kappa coefficient The calculation formula is:

[0122] ;

[0123] Where i and j are different label values; k represents the total number of label values ​​for this dimension; This represents the weight between label value i and label value j, when the following conditions are met. ,and This means there is no penalty for the same label value; This indicates the actual number of samples labeled as 'i' by one of the two labeling entities: labelers and experts; the... This represents the expected number of samples assuming that the two annotation subjects, annotators and experts, are completely independent (random).

[0124] S304. Generate a quality monitoring report based on the labeling consistency rate; the quality monitoring report includes the labeling consistency rate of each dimension of labels, information analysis of inconsistent labeling samples, and labeling quality trend assessment.

[0125] Furthermore, in step S304, the quality monitoring report is fed back to the annotators in step S200, and the annotators are required to review and re-annotate the samples whose annotation consistency rate does not meet the standard, so as to realize real-time quantitative evaluation and dynamic continuous optimization of annotation quality.

[0126] In addition to being fed back to the labelers, the quality monitoring report can also be fed back to step S100, serving as training data for the iterative optimization of the domain adaptive fine-tuning model, and also being fed back to step S500 in the second stage to provide quality reference for the final expert review.

[0127] The following is an example of the annotation results (in structured JSON format) after the camera movement dimension of a video sample has been corrected by the annotators and completed expert sampling quality review. The annotation results include the annotated draft sample set, manual correction information, review conclusions, and quality monitoring data (annotation consistency rate in the monitoring report).

[0128] {

[0129] "annotations": [

[0130] {

[0131] "completed_by": 14,

[0132] "result":

[0133] {"value": {"ranges": [{"start": 1,"end": 204}],"choices": ["fixed"]}}

[0134] ,

[0135] "updated_at": "2025-12-24T02:07:43.414247Z",

[0136] "prediction": {

[0137] "result":

[0138] {"value": {"ranges": [{"start": 1,"end": 166}],"choices":["fixed"]}}

[0139] ,

[0140] "model_version": "V1.1"

[0141] },

[0142] "expert_review": {

[0143] "consistency_rate": 0.92,"review_conclusion": "passed"

[0144] }

[0145] }

[0146] ,

[0147] "data": {

[0148] "video": " / data / local-files / ?d=MediaResourceDataset / videodata / 1.mp4"

[0149] }

[0150] }

[0151] In the above example of annotation results, the "annotations" array is a list of annotation results corrected by the annotators for the camera movement dimension. Each annotation result includes "completed_by" (annotator identifier), "result" array (label value and timestamp of the corrected camera movement by the annotator), "updated_at" (correction timestamp), "prediction" array (initial annotation results of the domain adaptive fine-tuning model for the camera movement), and "expert_review" (expert quality review, including annotation consistency rate and review conclusion). "data" records the path of the video sample. Furthermore, if the same video sample has multiple annotators, the "annotations" array will contain multiple records.

[0152] In one alternative case, a quality review iterative loop is set between step S200 and step S300; for example: in step S300, samples whose annotation consistency rate does not meet the standard are returned to the annotators in step S200 for re-annotation to obtain the current valid annotation sample set, and the expert sampling review mechanism in step S300 is restarted to conduct a quality review of the current valid annotation sample set until the annotation consistency rate of all samples meets the standard.

[0153] A more specific example of step S400 in the second stage is as follows:

[0154] In this embodiment, the general large model can be a multimodal base large language model, or other large language models or multimodal large models with thinking chain reasoning capabilities. It can also be replaced by a combination of multiple models (such as using different models to review different dimension tags separately, and then combining the logical review results of each model).

[0155] Furthermore, such as Figure 4 As shown, the steps for the multimodal base large language model to logically review the annotation results include:

[0156] S401. Construct logical audit prompts; the logical audit prompts include:

[0157] (1) Role setting, used to instruct the multimodal base large language model to perform logic auditing tasks in the role of a journalism logic auditor;

[0158] (2) Task description, which requires the multimodal base language model to check the factuality, compliance and consistency of the annotation results from multiple logical review dimensions;

[0159] (3) The news media sample set is used to provide basic information and content description of the news media sample set;

[0160] (4) The annotation results are used to provide the dimension labels and corresponding label values ​​of all samples in the news media sample set;

[0161] (5) The dimensions of the logical review can cover logical reviews related to factuality, compliance, and consistency, including:

[0162] Consistency review of labeling between dimensions: Check whether there are contradictions between different dimension labels of the same sample; for example, if the shot is labeled as "long shot" but the description of the main subject of the picture states that the person occupies most of the area of ​​the picture, then there is an inconsistency between the shot and the description.

[0163] Label and content matching review: Verify whether the label value matches the actual content of the sample; for example, if the camera movement method is labeled as "fixed shot", it is necessary to verify whether there is camera movement in the video clip;

[0164] Label granularity and boundary review: Check whether the granularity of the dimension labels is appropriate and whether the label boundaries (such as the start and end times of the camera movement) are accurate;

[0165] And compliance review of labeling definitions: verify whether the labeling of the samples conforms to the labeling specifications for the definitions of labels for each dimension;

[0166] (6) Define the output format in the form of a thought chain, which is used to require the reasoning process to be output step by step in the form of a thought chain.

[0167] S402. Based on the logical review prompts, the multimodal base language model performs logical review on the annotation results based on the dimensions of the logical review; and outputs the thought chain reasoning text according to the output format; wherein the content of the thought chain reasoning text includes at least: dimension label analysis, statement of reasoning basis, explanation of annotation inconsistencies, and analysis of the reasons for annotation inconsistencies;

[0168] In addition, the content of the thought chain reasoning text also includes the judgment result. When the multimodal base large language model does not detect any contradictions in the label dimension or label value (labeling inconsistency), the judgment result in the thought chain reasoning text is correct; when the multimodal base large language model detects any contradictions in the label dimension or label value (labeling inconsistency), the judgment result in the thought chain reasoning text is incorrect.

[0169] S403. Obtain the review conclusion based on the thought chain reasoning text, and integrate the thought chain reasoning text with the review conclusion to obtain the structured review report.

[0170] Preferably, the review conclusion is classified according to the following criteria: when no contradictions are found after all dimension tags have been reviewed, "pass" is output; when there are slight inconsistencies but not enough to negate the overall correctness of the labeling, "doubtful" is output with a specific explanation of the problem; when there are clear logical contradictions or obvious labeling errors, "fail" is output with a detailed reasoning process.

[0171] The audit report includes, but is not limited to: the sample identifier being audited, the audit time, the list of dimension tags, the thought chain reasoning text or a summary of the thought reasoning text, the audit conclusion for each dimension tag (pass / questionable / fail), and the multimodal base large language model identifier; if the audit conclusion is questionable or fail, the audit report should also include a detailed description of the problems or contradictions found.

[0172] The following two examples further illustrate the output of the thought chain reasoning text in step S402 and the review report in step S403:

[0173] Example 1: An example of the output of the thought chain reasoning text of the multimodal base large language model described in step S402, including two scenarios:

[0174] Scenario A: The judgment result is correct. The output of the thought chain reasoning text is as follows:

[0175] "I'm certain this photo is a close-up because: the main subjects are two phones on the table (a purple iPhone with its back facing up, and a dark iPhone standing upright on a stand), which occupy the majority of the visual space, with clear details (such as the camera module, the text 'iPhone Air' on the screen, and the buttons on the frame). While elements in the background, such as the promotional sign, telephone line, and stand base, are present, they are secondary and slightly blurred, which fits the characteristics of a close-up composition: focusing on the subject, compressing space, and highlighting local details. In addition, the entire image does not include the full body of the person or a panoramic view of the environment, nor is it a close-up shot, so it is accurately classified as a close-up."

[0176] Scenario B: The thought process logic is incorrect. The output text is as follows:

[0177] "This conclusion is incorrect. In fact, the composition of this image is not symmetrical. Symmetrical composition requires a mirror-like visual balance on either side (or vertically) of the central axis, with a high degree of correspondence in the distribution, shape, color, and spatial relationships of elements. However, in this image:"

[0178] 1. The left side of the image features large glass windows, pedestrians, and greenery, while the right side shows store counters, promotional signs, and displays of various mobile phones, creating a completely different visual experience between the left and right sides.

[0179] 2. The central focal point, "iPhone 17 Officially Available for Sale," is positioned slightly to the right, rather than being centered and symmetrical.

[0180] 3. The mobile phones are arranged linearly from left to right along the counter, with no mirror image relationship in terms of quantity or position;

[0181] 4. The pillars, plants, and distant customers in the background are all asymmetrically distributed;

[0182] 5. The overall design is a typical commercial display scene, employing a composition based on the "rule of thirds" or "visual flow" to guide the eye, rather than pursuing balance and symmetry.

[0183] Therefore, this image does not belong to symmetrical composition.

[0184] Example 2: After the multimodal base large language model performs logical review of the camera movement dimension label of a video sample, the output example of the review report (integrating the thought chain reasoning text and review conclusion) in step S403 will serve as the precursor to the final labeled sample set and will be passed to step S500 for final review by the expert:

[0185] {

[0186] "video": " / data / tiandalin / dataset / MediaResourceDataset / videodata_test / 120.mp4",

[0187] "Thinking": "\n\nFrom 0.0 seconds to 13.2 seconds, the image remains stable, with the camera focused on a group of students being interviewed. The positions of the background buildings and people do not change significantly, which is a typical fixed-shot shot. During this phase, the camera does not move or switch perspectives.\n\nFrom 13.2 seconds to 15.0 seconds, the camera gradually and smoothly moves to the left. Students who were originally on the right side of the screen (such as those wearing white camouflage jackets) gradually move out of the field of view, while students on the left wearing burgundy school uniforms enter a clearer foreground. The architectural patterns in the background also change as the camera sweeps horizontally. This continuous horizontal sweeping motion conforms to the definition of a 'panning shot'—that is, the camera position remains unchanged but the shooting range is rotated horizontally or vertically, thereby changing the composition and position of the subject.\n\nTherefore, judging from the content of the image and the movement trajectory, the first part of the video is a fixed shot, and the second part is a panning shot, with the dividing point around 13.2 seconds. The conclusion is correct and conforms to the laws of visual expression.\nIn summary, I have determined that the camera movement methods contained in this video and the start and end times of each camera movement are:

[0188] {0.0s - 13.2s: Fixed, 13.2s - 15.0s: Shaking}

[0189] "label": "{0.0s - 13.2s: Fixed, 13.2s - 15.0s: Shaking}",

[0190] "audit_conclusion": "Passed",

[0191] "audit_time": "2025-12-24T03:15:00Z",

[0192] "model_name": "VL-MAX"

[0193] }

[0194] In Example 2 above, "video" records the path of the sample being reviewed; "thinking" is the thought chain reasoning text; "label" is the final verified label value result for the dimension label of the camera movement method; "audit_conclusion" is the review conclusion; "audit_time" is the review time; and "model_name" is the identifier of the multimodal base large language model.

[0195] A more specific example of step S500 in the second stage is as follows:

[0196] The steps involved in the expert final ruling include:

[0197] S501. The final review experts obtain the following aggregated information on the annotation platform: (1) the effective annotation sample set corrected by the annotators in step S200 of the first stage; (2) the quality monitoring report and annotation consistency rate data generated by the expert review in step S300 of the first stage; (3) the thought chain reasoning text output by the multimodal base large model in step S400 of the second stage and the sorted review report; (4) the comparison records in step S200 of the first stage. The annotation platform displays the above information in a unified interface, allowing the final review experts to view it item by item according to the dimension tags of the samples.

[0198] S502. For samples marked as disputed, the final review experts shall review the annotation results, the comparison records, the thought chain reasoning text, and the review report, and make a decision to maintain the original annotation, correct the annotation, or mark it as indeterminate, and record the basis of the decision.

[0199] Among them, the samples marked as disputed include at least one of the following:

[0200] The review conclusion identifies samples that are deemed unacceptable or questionable by the multimodal base language model. For example, when the multimodal base language model performs a logical review on a sample, if the scene dimension label is deemed unacceptable, then the sample is marked as a disputed sample.

[0201] When manually reviewing and correcting the labeled draft sample set, samples with a skip rate higher than a preset threshold, preferably, when the skip rate is within a certain range... If the sample falls within the specified range, it is marked as a disputed sample.

[0202] In addition, samples whose labels on any dimension are inconsistent between the expert-relabeled sample set and the effective labeled sample set.

[0203] S503. After confirming that all the disputed samples have been processed, lock and archive them to generate the final labeled sample set. The final labeled sample set includes: the final labeled tag, labeler identification, quality monitoring report, thought chain reasoning text, and record of the basis for the ruling for each sample in the news media sample set.

[0204] Furthermore, the quality management documents generated throughout the entire process of the model-assisted semi-automatic construction method for news media thinking chain samples of the present invention are archived, including: the comparison record in the first stage step S200, the quality monitoring report in the first stage step S300, the multimodal base large model thinking chain reasoning text in the second stage step S400, and the adjudication basis record in the second stage step S500, forming a complete full life cycle archive of sample annotation.

[0205] This invention is not limited to the above-described optional embodiments. Anyone can derive other various forms of products under the guidance of this invention. However, regardless of any changes made in their shape or structure, any technical solution that falls within the scope of the claims of this invention shall be protected by this invention.

Claims

1. A model-assisted semi-automatic method for constructing news media thought chain samples, characterized in that, include: A domain-adaptive fine-tuning model was used to initially label the news media sample set with multi-dimensional tags, resulting in a draft labeled sample set. The news media sample set includes video frame sequences, images, or text; the multi-dimensional tags include shot type, shooting angle, composition method, camera movement method, and corresponding start and end times; The draft annotation sample set is manually reviewed and corrected, and a sample cleaning operation is performed to obtain a valid annotation sample set, as well as a comparison record between the draft annotation sample set and the valid annotation sample set; The quality of the effective labeled sample set is reviewed through an expert sampling review mechanism to obtain the labeling results. The annotation results include the annotation draft sample set, manual correction information, review conclusions, and quality monitoring data; Based on the aforementioned news media sample set, the annotation results are logically reviewed for factual accuracy, compliance, and consistency using the reasoning capabilities of a general-purpose model. This process yields a thought chain reasoning text and a review report. The steps include: constructing logical review prompts; using these prompts, the general-purpose model performs a logical review of the annotation results based on dimensions including label consistency, label-content matching, label granularity and boundary checks, and label definition compliance; outputting the thought chain reasoning text; obtaining a review conclusion based on the thought chain reasoning text; and integrating the thought chain reasoning text with the review conclusion to obtain a structured review report. Based on the annotation results, the comparison records, the thought chain reasoning text, and the review report, experts will make a final review decision to determine the final annotated sample set. The expert sampling review mechanism includes: A hybrid sampling strategy is employed to extract samples from the set of valid labeled samples to construct an expert review sample set. The hybrid sampling strategy consists of two parts: disputed sample sampling and random supplementary sampling. The disputed sample sampling involves selecting samples from the set of valid labeled samples that are inconsistent with the labels in the draft labeling sample set, based on the comparison records. The random supplementary sampling involves selecting samples from the set of valid labeled samples that are consistent with the labels in the draft labeling sample set, based on the comparison records, and then randomly sampling them according to a preset ratio. Perform expert standard re-annotation on all samples in the expert review sample set to obtain the expert re-annotated sample set; The expert-re-annotated sample set and the effective-annotated sample set are compared dimension by dimension under multi-dimensional labels to calculate the annotation consistency rate. A quality monitoring report is generated based on the aforementioned consistency rate of annotations.

2. The model-assisted semi-automatic method for constructing news media thought chain samples according to claim 1, characterized in that, The quality monitoring report includes the labeling consistency rate of each dimension, information analysis of inconsistent labeling samples, and labeling quality trend assessment.

3. The model-assisted semi-automatic method for constructing news media thought chain samples according to claim 1, characterized in that, The labeling consistency rate is defined as the proportion of the number of samples in the expert-relabeled sample set that are consistent with the labeling in the effective labeled sample set to the total number of samples in the expert-relabeled sample set.

4. The model-assisted semi-automatic method for constructing news media thought chain samples according to claim 1, characterized in that, The expert sampling review mechanism also includes: providing feedback based on the quality monitoring report to manually re-label samples whose labeling consistency rate does not meet the standard.

5. The model-assisted semi-automatic method for constructing news media thought chain samples according to claim 1, characterized in that, The logical review prompts include role settings, task descriptions, the news media sample set, the annotation results, the dimensions of the logical review, and the output format that defines the thought chain. According to the output format, the general large model outputs the thought chain reasoning text; wherein, the content of the thought chain reasoning text includes at least: dimension label analysis, statement of reasoning basis, explanation of inconsistencies in labeling, and analysis of the reasons for inconsistencies in labeling.

6. The model-assisted semi-automatic method for constructing news media thought chain samples according to claim 5, characterized in that, The expert final ruling includes: For samples marked as disputed, the final review experts will comprehensively review the marking results, the comparison records, the thought chain reasoning text, and the review report, and make a decision to maintain the original marking, correct the marking, or mark it as indeterminate, and record the basis for the decision. After confirming that all the disputed samples have been processed, the final labeled sample set is locked and archived; wherein, the content of the final labeled sample set includes: the final labeled label, labeled personnel identification, quality monitoring report, thought chain reasoning text, and record of the basis for the ruling for each sample in the news media sample set.

7. The model-assisted semi-automatic method for constructing news media thought chain samples according to claim 6, characterized in that, Samples marked as disputed include at least one of the following: The general large model mentioned in the audit conclusion was determined to be a sample that failed or was questionable; When manually reviewing and correcting the labeled draft sample set, samples with a skip rate higher than a preset threshold are skipped. In addition, samples whose labels on any dimension are inconsistent between the expert-relabeled sample set and the effective labeled sample set.

8. The model-assisted semi-automatic method for constructing news media thought chain samples according to claim 1, characterized in that, When using a domain-adaptive fine-tuning model to initially label a news media sample set with multi-dimensional labels, the dimensional labels include, but are not limited to: shot size, shooting angle, composition method, camera movement method, and the start and end times of the camera movement.

9. The model-assisted semi-automatic method for constructing news media thought chain samples according to claim 8, characterized in that, The manual review and correction of the annotation draft sample set includes: reviewing whether the dimension labels of each sample in the annotation draft sample set are correct, correcting incorrect label values, and supplementing missing dimension labels; The sample cleaning operation includes: removing samples from the news media sample set that do not meet the labeling requirements, marking samples from the news media sample set that have abnormal sources, and standardizing the format and naming conventions of the news media sample set.

Images