Automatic construction system of multi-modal data set based on thought chain and self-reflection

Abstract

The application relates to the technical field of artificial intelligence and discloses a multi-modal data set automatic construction system based on thought chains and self-reflection, which comprises data preprocessing, Prompt generation and evaluation, reasoning data generation, multi-dimensional quality evaluation, automatic correction, model fine-tuning and closed-loop control modules. The system first extracts samples and generates qualified question Prompts; subsequently, a reasoning data generation module calls a pre-trained multi-modal large model to generate structured data containing reasoning chains and formal outputs; a multi-dimensional quality evaluation module quantitatively scores the data, and high-quality data is used to update the core pre-trained multi-modal large model in the model fine-tuning module. The application can automatically construct a high-quality data set containing a deep reasoning process, and through a self-correction and closed-loop evolution mechanism, solves the problems of high labeling cost and uncontrollable data quality in the prior art, and realizes the collaborative evolution of data and model capability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an automatic construction system for multimodal datasets based on thought chains and self-reflection. Background Technology

[0002] The development of artificial intelligence technologies, especially large multimodal models, is highly dependent on the scale and quality of training datasets. Currently, the main way to obtain such datasets is through manual annotation. As a labor-intensive process, this method not only requires a high investment of manpower and time, resulting in low execution efficiency, but also has poor scalability, making it difficult to meet the training needs of modern large models for massive amounts of data.

[0003] To overcome the limitations of manual annotation, the industry has begun exploring automated data generation technologies. However, existing automated methods have significant shortcomings in ensuring data quality. The data produced by these methods varies in quality, especially for tasks requiring deep reasoning. The generated data often lacks a complete and rigorous logical reasoning process or contains factual errors, directly affecting the training effect of the model.

[0004] Furthermore, existing automation solutions are typically open, linear processes. The data generation models upon which these systems rely have fixed capabilities after deployment, unable to learn from their own generated data or iterate their capabilities based on evaluation results. This creates a disconnect between data production and model capability improvement, limiting the continuous evolution of the data generation system's performance. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an automatic multimodal dataset construction system based on thought chain and self-reflection. This system solves the problems of high cost and low efficiency in dataset construction due to reliance on manual annotation, uncontrollable quality of automatically generated data and lack of evaluation and correction mechanisms, and the inability of the system itself to form a closed loop for continuous optimization.

[0006] To achieve the above objectives, this invention provides the following technical solution: an automatic construction system for multimodal datasets based on thought chains and self-reflection, comprising:

[0007] The data preprocessing module extracts samples from a full multimodal dataset and receives the preliminary annotation information corresponding to the samples.

[0008] The Prompt generation and evaluation module connects to the data preprocessing module. Based on samples and preliminary annotation information, this module generates question prompts and includes a Prompt evaluation unit. The evaluation unit inputs the generated question prompts into a pre-trained large language model, driving the model to independently evaluate them from four preset perspectives: user, developer, domain expert, and security. Based on the majority vote, it outputs a qualified question prompt.

[0009] The inference data generation module connects to the prompt generation and evaluation module. It receives a qualified prompt and corresponding samples, and invokes a pre-trained multimodal large model to generate structured data. The module uses a specific instruction template to guide the model, forcing its output to include two parts: an inference chain guided by a preset inference chain start marker, and a formal output guided by a preset formal output start marker. The module extracts content by parsing these two markers, thereby generating structured data containing the inference chain and formal output. The module also performs self-reflection and immediate correction steps: it submits a verification task to a pre-trained large language model to check for logical contradictions or inconsistencies between the inference chain and the formal output; if an inconsistency is detected, it uses this inconsistency information as feedback and resubmits it along with the original input to the pre-trained multimodal large model for immediate correction and regeneration.

[0010] The multi-dimensional quality assessment module connects the inference data generation module and the data preprocessing module. This module quantitatively evaluates the generated structured data. It uses an independent pre-trained large language model as the evaluator, scoring the data across three dimensions: logical coherence, factual accuracy, and inference completeness, yielding scores for logical coherence, factual accuracy, and inference completeness, respectively. Subsequently, the multi-dimensional quality assessment module calculates a comprehensive quality score, which is then compared to a quality threshold to classify the data as high-quality or low-quality.

[0011] The automated correction module connects to the multi-dimensional quality assessment module. This module handles structured data marked as low-quality. It integrates the scores of low-scoring dimensions with the scoring criteria provided by the evaluator into a structured correction instruction. This instruction, along with the original question prompt and sample data, is resubmitted to the pre-trained multi-modal polymorphic large model to drive targeted regeneration. The automated correction module also sets an upper limit on the number of corrections; if the quality score of a single data point remains below the quality threshold after reaching this limit, the data is marked as a correction failure and discarded.

[0012] The model fine-tuning module connects to the multi-dimensional quality assessment module. It utilizes all the structured data labeled as high-quality to perform fine-tuning operations on the aforementioned pre-trained multimodal large model. The fine-tuning process is triggered when the accumulated data reaches a batch processing threshold or a predetermined time period. Methods such as efficient parameter fine-tuning can be employed to update only a portion of the model's parameters, thereby efficiently embedding the knowledge and reasoning capabilities inherent in the generated high-quality data into the model itself.

[0013] The closed-loop control module is connected to the model fine-tuning module and the data preprocessing module. After the model fine-tuning module completes a model update, the closed-loop control module receives a model update completion signal, obtains the identifier of the new version model, and updates the system configuration so that subsequent generation tasks can call the updated model. The closed-loop control module then selects new unprocessed data from the full dataset and starts a new round of automated annotation process, beginning with the Prompt generation and evaluation module.

[0014] This invention provides an automatic system for constructing multimodal datasets based on thought chains and self-reflection. It offers the following advantages:

[0015] 1. This invention integrates modules such as data preprocessing, prompt generation, and inference data generation to automate the dataset construction process, replacing the traditional manual annotation step, thereby improving the efficiency of dataset construction and reducing costs.

[0016] 2. This invention sets up a multi-dimensional quality assessment module and an automated correction module to quantitatively evaluate and iteratively correct the generated structured data, thereby identifying and correcting data that lacks logical coherence, factual accuracy, or reasoning completeness, ensuring that the final output dataset meets high-quality standards.

[0017] 3. This invention, through the cooperation of a model fine-tuning module and a closed-loop control module, constitutes a system for the co-evolution of data and the model. The model fine-tuning module updates the core model using the generated high-quality data, while the closed-loop control module applies the updated model to subsequent data generation tasks. This allows the overall performance of the system to continuously improve as the amount of data processed increases. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the system functional modules of the present invention;

[0019] Figure 2 This is a detailed flowchart of the data preprocessing and preliminary annotation process of the present invention;

[0020] Figure 3 This is a detailed flowchart of the multi-view Prompt generation and evaluation of the present invention;

[0021] Figure 4 This is a detailed flowchart of the inference chain and the generation of the formal output of the present invention;

[0022] Figure 5 This is a detailed flowchart of the multi-dimensional quality assessment and quantification of the present invention;

[0023] Figure 6 This is a detailed flowchart of the automated correction and reconstruction of low-quality data in this invention;

[0024] Figure 7 This is a detailed flowchart of the continuous fine-tuning of the multimodal large model of the present invention;

[0025] Figure 8 This is a detailed flowchart of the closed-loop control and automated iteration of the present invention.

[0026] The module includes: 10. Data preprocessing module; 20. Prompt generation and evaluation module; 30. Inference data generation module; 40. Multi-dimensional quality assessment module; 50. Automated correction module; 60. Model fine-tuning module; and 70. Closed-loop control module. Detailed Implementation

[0027] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] See attached document Figure 1 This invention provides an automatic multimodal dataset construction system based on thought chains and self-reflection. The system is deployed in a computing device including a processor, memory, and network interface. Specifically, the system may include: a data preprocessing module 10, a prompt generation and evaluation module 20, an inference data generation module 30, a multi-dimensional quality evaluation module 40, an automatic correction module 50, a model fine-tuning module 60, and a closed-loop control module 70.

[0029] The data preprocessing module 10 is used to extract samples from the full multimodal dataset. This module implements a strategy that combines stratified sampling and random sampling. First, it determines the sampling ratio based on the proportion of each modality's data in the full dataset, and then performs stratified sampling within each modality according to a preset classification structure.

[0030] Data preprocessing module 10 determines the number of samples to be drawn using the following formula. :

[0031] ;

[0032] in, This represents the final number of samples drawn. This represents the total number of data points in the full multimodal dataset. This module also receives and stores preliminary annotation information in text format corresponding to the extracted samples. This information includes task category, key objects, and preliminary inference conclusions.

[0033] The Prompt generation and evaluation module 20 has its input connected to the output of the data preprocessing module 10. The Prompt generation and evaluation module 20 receives sample data and corresponding preliminary annotation information, calls a pre-trained multimodal large model, and automatically generates one or more sets of question prompts for each sample.

[0034] The Prompt generation and evaluation module 20 also includes a Prompt evaluation unit. This unit inputs each generated question Prompt into a pre-trained large language model. The model evaluates the Prompt from four preset perspectives: user, developer, domain expert, and security, and outputs a pass / fail result. The Prompt generation and evaluation module 20 outputs pass-through Prompts to the next module and triggers optimization and regeneration processes for fail-through Prompts.

[0035] The inference data generation module 30 has its input connected to the output of the Prompt generation and evaluation module 20. The inference data generation module 30 receives qualified question prompts and corresponding sample data, calls a pre-trained multimodal large model, and uses the prompt as a guide to generate structured data containing two parts: an inference chain and a formal output. The inference chain is a textual description of a series of logical reasoning steps, and the formal output is the final conclusion for downstream tasks.

[0036] The multi-dimensional quality assessment module 40 has its inputs connected to the outputs of the inference data generation module 30 and the data preprocessing module 10, respectively. The multi-dimensional quality assessment module 40 receives the generated inference chain, the final output, and the initial annotation information, and calls an independent pre-trained large language model to calculate quality index scores in three dimensions: logical coherence score. Accuracy score of facts and reasoning completeness score .

[0037] The multi-dimensional quality assessment module 40 calculates the overall quality score of this structured data based on the following formula. :

[0038] ;

[0039] in, For the overall quality score; , , The scores are respectively based on logical coherence, factual accuracy, and complete reasoning; , , These are preset weighting coefficients that can be adjusted according to the task type, and satisfy the following conditions: This module will calculate the score. The data is compared with a preset quality threshold, and the data is marked as high quality or low quality based on the comparison result.

[0040] An automated correction module 50 has its input connected to the output of a multi-dimensional quality assessment module 40. This module receives data marked as low quality and its scores for each dimension. , , The system includes the data and the scoring criteria provided by the evaluation model. The automated correction module 50 uses this information as correction instructions and feeds them back to the pre-trained multimodal large model called by the inference data generation module 30, driving the pre-trained multimodal large model to specifically regenerate the inference chain and the final output. The corrected data will then be sent back to the multi-dimensional quality evaluation module 40 for re-evaluation.

[0041] The model fine-tuning module 60 has its input connected to the output of the multi-dimensional quality assessment module 40. The model fine-tuning module 60 receives all data labeled as high-quality and organizes this data into a high-quality dataset. Using this dataset, the model fine-tuning module 60 performs fine-tuning operations on the system's core pre-trained multimodal large model (i.e., the model invoked by the Prompt generation and evaluation module 20 and the inference data generation module 30) to update its model parameters.

[0042] The closed-loop control module 70, connected to the model fine-tuning module and the data preprocessing module, is used to schedule and control the data flow and workflow of the entire system. In the initial stage, the closed-loop control module 70 triggers the data preprocessing module 10 to start working, and then sequentially drives subsequent modules to perform operations. After the model fine-tuning module 60 completes a model update, the closed-loop control module 70 uses the updated model to initiate a new round of automated annotation process for the new unlabeled data. This process starts from the Prompt generation and evaluation module 20, thus forming a complete closed loop of data generation and model optimization. Although the attached diagram mainly shows the closed-loop control module connecting the model fine-tuning module and the data preprocessing module to form a process loop, in the actual system architecture, this module can achieve global update control of the model versions called by each module in the system by updating the central configuration file or model registry.

[0043] The following section will describe each functional module in the system in detail:

[0044] The data preprocessing module 10 is used to perform sample extraction and preliminary labeling data management. The input to this module is the metadata and storage path of the full multimodal dataset. Its internal workflow is as follows: First, it parses the dataset's metadata to identify the modality type of each data entry (e.g., identified as image, text, video, etc. by file extension or metadata tag) and preset classification information; next, it calculates the proportion of each modality and classification in the full dataset; then, based on the determined number of samples... The module employs a stratified sampling algorithm to extract a subset of samples with the same distribution characteristics from the full dataset. It also provides an interface to receive and store preliminary annotation information in a structured text format (e.g., JSON or XML) that corresponds one-to-one with this subset of samples. This information includes three fields: task category, list of key objects, and preliminary inference conclusions.

[0045] The Prompt generation and evaluation module 20 is responsible for generating high-quality question prompts for each sample. The module receives sample data and its initial annotation information from the data preprocessing module 10. During the generation phase, the module integrates the sample data (e.g., images are encoded as tensors) and the initial annotation information in text form into a structured input, which is then submitted to a pre-trained multimodal large model along with a pre-defined instruction template. This template instructs the model to generate multiple question prompts covering different semantic perspectives.

[0046] During the evaluation phase, the Prompt evaluation unit within the Prompt generation and evaluation module 20 processes each generated question Prompt. This unit constructs an independent evaluation task for each preset evaluation perspective (user, developer, domain expert, security). Each task includes a role definition, evaluation instructions, and the Prompt to be evaluated, and is sent to a pre-trained large language model. For example, for the evaluation task from the developer perspective, the input submitted to the large language model is: Role: Developer. Task: Evaluate the logic and structural rationality of the following Prompt. Result: Pass. Prompt: [Prompt to be evaluated]. This unit collects the judgment results (pass or fail) from all four perspectives and makes a final judgment based on the majority voting principle (e.g., a Prompt receiving three or more pass judgments is considered as overall pass). Prompts judged as fail are sent back to the generation phase along with the negative evaluation evidence they received for iterative optimization.

[0047] The inference data generation module 30 generates structured data containing the inference process and the final conclusion. This module receives qualified question prompts and corresponding raw sample data from the Prompt generation and evaluation module 20. The inference data generation module 30 integrates these two pieces of information, attaches a specific instruction template, and submits it to the pre-trained multimodal large model. This instruction template explicitly requires that the model's output must contain two parts: the inference chain and the final output. After the model executes, the inference data generation module 30 parses the returned text stream, extracts the content according to predefined delimiters (such as the inference chain and the final output), and stores it in two independent data fields, forming a complete structured data record.

[0048] A multi-dimensional quality assessment module 40 is used to quantitatively score the generated structured data. The input to this module includes the prompt, the generated reasoning chain, the final output, and the original preliminary annotation information. Internally, this module integrates an independent pre-trained large language model as the evaluator. For each piece of data to be evaluated, this module constructs three independent scoring tasks, corresponding to the dimensions of logical coherence (L), factual accuracy (F), and reasoning completeness (C). Each scoring task includes explicit scoring criteria and requires the evaluator to output a score between 0 and 100. For example, for the logical coherence scoring task, the instruction submitted to the evaluator is: Score the logical coherence of the reasoning chain (0-100 points) according to the following scoring criteria: [Specific scoring criteria details]. Data: [Prompt, reasoning chain, etc.]. Score.

[0049] In obtaining , , After scoring the three dimensions, the multi-dimensional quality assessment module 40 assesses the scores based on preset weighting coefficients. , , Calculate the overall quality score These weighting coefficients are stored as configurable parameters in this module, allowing adjustments based on different task types to emphasize different quality dimensions. This module will calculate... The value is compared with a preset quality threshold, and a high-quality or low-quality label is added to the data.

[0050] The automatic correction module 50 is used to process data marked as low quality. This module receives low-quality data and its corresponding scores for each dimension. , , The module integrates this information with the text scoring criteria provided by the evaluator into a structured correction instruction that explicitly identifies the dimension in which the original generated content is deficient. Subsequently, this correction instruction, the original question prompt, and the sample data are submitted to the pre-trained multimodal large model, instructing it to regenerate the inference chain and the final output. This correction process includes a counter to record the number of corrections for a single data point. If the number of corrections exceeds a preset limit (e.g., 3 times) and the quality score still does not meet the standard, the data is discarded.

[0051] The model fine-tuning module 60 is used to update the parameters of the pre-trained multimodal large model at the core of the system using the generated high-quality data. This module continuously collects all data records marked as high-quality by the multi-dimensional quality assessment module 40. When the accumulated data volume reaches the preset batch size or the predetermined fine-tuning period, the model fine-tuning module 60 initiates the fine-tuning process. The first step of the process is to convert the collected high-quality data records into the format required for model training (e.g., each record is converted into a JSON object containing the input and target output). Next, the module loads the current version of the pre-trained multimodal large model and executes a fine-tuning algorithm, such as the LoRA method in Parameter Efficient Fine-Tune (PEFT), using the prepared dataset. After fine-tuning is complete, the module saves the updated model parameters as a new version and records its performance validation metrics.

[0052] The closed-loop control module 70, as the system's scheduling core, manages the entire data generation and model optimization workflow. This module operates as a state machine, monitoring and driving the execution of other modules. During system initialization, this module triggers the data preprocessing module 10. Once a module completes its task, the closed-loop control module 70 passes its output to the next designated module. Its core function is demonstrated by receiving a model update completion signal from the model fine-tuning module 60. Upon receiving this signal, the closed-loop control module 70 obtains the identifier of the new model version and updates the configurations of the Prompt generation and evaluation module 20 and the inference data generation module 30, enabling them to call this updated model in subsequent tasks. Subsequently, the closed-loop control module 70 checks for the existence of new unlabeled datasets. If found, it initiates a new round of automated labeling, starting with the Prompt generation and evaluation module 20, thus completing a closed-loop iteration between data and model.

[0053] See attached document Figure 2 The data preprocessing and initial labeling stages provide initial, representative seed data for subsequent automated processes.

[0054] The first stage involves data preprocessing and preliminary labeling, which includes the following steps:

[0055] Step S110: Sample set acquisition based on stratified sampling. This step extracts a sample subset from a full multimodal dataset. First, the system traverses the entire dataset, classifying each data point into a predefined modality category (such as image, text, audio, video, or image-text pairs) based on its metadata information (e.g., file extension, MIME type, or metadata tags). Simultaneously, if the metadata contains other classification information (e.g., data source, scene tags, etc.), secondary classification is performed within each modality category.

[0056] Next, the system counts the quantity and proportion of each modality category and secondary category in the entire dataset. Then, the system calculates the total number of samples to be extracted using the following formula. :

[0057] ;

[0058] in, This represents the final number of samples drawn. This represents the total number of data points in the full multimodal dataset.

[0059] Finally, based on the calculated proportions of each category, the system uses a stratified sampling algorithm to proportionally extract a corresponding number of samples from the full dataset, ensuring that the generated sample subsets are consistent with the full dataset in terms of modal distribution and category structure. The extracted sample data and their file paths are stored for use in subsequent steps.

[0060] Step S120: Execution of preliminary annotation and data structuring. This step involves performing preliminary annotation, either manually or semi-automatically, on each data point in the sample subset obtained in step S110. The annotation work does not rely on complex graphical annotation tools, but is accomplished by creating a structured text file (e.g., JSON format) for each sample data point.

[0061] Before performing annotation, the system first requires one or more "Annotation Documents". Each "Annotation Document" corresponds to a specific task category (e.g., wildfire identification). The document generation logic is as follows: input a task category name, output a structured text containing a list of key objects and a preliminary reasoning conclusion template. The list of key objects defines the target entities that need to be focused on under this task category (e.g., forest, flame, smoke, buildings). The preliminary reasoning conclusion template defines the conclusion sentence structure that can be drawn based on the relationships between these key objects (e.g., <object A> is close to <object B>, there is a <risk type> risk).

[0062] Annotators, based on the task category of the sample to be annotated, refer to the corresponding annotation document. For each sample data entry, the annotator fills in the following fields in the JSON file they create:

[0063] : The name of the task category to which the current data belongs.

[0064] : A list or dictionary of boolean type used to identify whether each key object defined in the annotation document exists in the current sample data.

[0065] For example{ }

[0066] This is a string that the annotator uses to generate an objective preliminary inference conclusion based on the sample data and the inference template in the annotation document. For example: The image shows flames and thick smoke on the right side of the forest, with no buildings visible, indicating a risk of the fire spreading.

[0067] After the annotation is completed, these JSON files containing preliminary annotation information will be associated with the original sample data (e.g., by filename or unique ID) to form a seed dataset, which will then be passed to the next stage.

[0068] See attached document Figure 3 The multi-perspective Prompt generation and evaluation phase aims to automatically generate and validate high-quality question prompts for each sample.

[0069] The second stage involves multi-perspective Prompt generation and evaluation, which includes the following steps:

[0070] Step S210: Automated generation of the Prompt. This step receives the sample data from the previous stage and its corresponding structured text file containing preliminary annotation information. For each sample, the system first encodes the multimodal data. For example, if the sample is an image, it is converted into a tensor representation acceptable to the model; if it is text, it is segmented and embedded.

[0071] Next, the system integrates the encoded sample data with preliminary textual annotations (task category, key objects, and preliminary inference conclusions) and combines this with a pre-defined instruction template to form a complete input sequence. This instruction template instructs the pre-trained multimodal large model to generate multiple different prompts based on all provided information, from different semantic levels and inference paths. For example, the instruction template might include the following: Based on the following [image] and preliminary analysis: Task Category: [Task Category]; Key Objects: [List of Key Objects]; Preliminary Conclusion: [Preliminary Inference Conclusion]. Please generate five different questions, covering different dimensions such as fact-finding, causal analysis, outcome prediction, and counterfactual reasoning.

[0072] The system submits the input sequence to a pre-trained multimodal large model and receives the text output from the model containing multiple prompts. The system parses this output and treats each prompt as an independent candidate.

[0073] Step S220: Implementation of the multi-role Prompt evaluation mechanism. This step verifies the quality of each candidate prompt generated in step S210. The system constructs an independent evaluation task for each preset evaluation perspective (user, developer, domain expert, security).

[0074] The data structure for each evaluation task is a request containing the following fields:

[0075] : A string that defines the role of the current evaluation task. For example, if you are a developer, your task is to evaluate the logic and structural rationality of the problem.

[0076] : A string providing specific evaluation instructions and output format requirements, such as: Please determine whether the following Prompt is logically clear, unambiguous, and executable. Only answer "yes" or "no," along with a brief reason.

[0077] : Prompt for candidate questions to be evaluated.

[0078] The system sends these four independent evaluation task requests in parallel or sequentially to a pre-trained large language model. The large language model responds to each task separately, returning a pass / fail judgment result and the reason.

[0079] Step S230: Evaluation Result Judgment and Optimization Loop. The system collects four judgment results for the same candidate Prompt. Based on a preset majority voting rule, such as a Prompt receiving three or more qualified judgments being considered as finally qualified, a comprehensive judgment is made.

[0080] If a candidate Prompt is deemed ultimately qualified, it is passed to the next stage along with its corresponding original sample data.

[0081] If a candidate Prompt is ultimately deemed unqualified, the system will collect all reasons returned by the evaluation tasks that were deemed unqualified. Then, the system will package these reasons along with the original candidate Prompt, along with an optimization instruction (e.g., "Rewrite this Prompt based on the following negative feedback: [List of negative feedback]"), and resubmit it to a pre-trained large language model or a dedicated text rewriting model to generate an optimized new version of the Prompt. This new version of the Prompt will then re-enter step S220 for evaluation. This optimization loop has a maximum number of attempts; if it still fails the evaluation after exceeding this number, the candidate Prompt will be discarded.

[0082] See attached document Figure 4 The reasoning chain and formal output generation phases aim to generate structured data containing detailed reasoning processes and final conclusions.

[0083] The third stage involves generating the inference chain and the final output, which includes the following steps:

[0084] Step S310: Guidance and Control of the Generation Process. This step receives the final qualified prompt from the previous stage, along with its corresponding raw sample data. The system integrates these two pieces of information and includes a structured instruction template to form a complete input sequence for submission to the pre-trained multimodal large model.

[0085] This instruction template is a key component that mandates the model's output to follow a predefined format containing specific delimiters. For example, the template might look like this: For the following [sample data] and question [Prompt], generate an answer in the following format: First, starting with the inference chain: provide a detailed, step-by-step logical reasoning process, with each step on a separate line. Then, starting with the final output: provide a concise and direct final answer.

[0086] By including this instruction template, the system ensures the parsability of the model output and explicitly separates the reasoning process from the conclusion.

[0087] Step S320: Structured parsing of the inference chain and the final output. The system submits the input sequence constructed in step S310 to the pre-trained multimodal large model and receives the text stream generated by the model. The system processes the returned text stream, first locating the two predefined delimiters: the inference chain and the final output.

[0088] Once the delimiter is located, the system performs the following operations:

[0089] Extract all content from the inference chain: after the delimiter to the actual output: before the delimiter, and store it in a file named... The data field contains a string representing the logical reasoning steps.

[0090] Extract the actual output: everything after the delimiter, and store it in a file named... The data field contains a string representing the final conclusion.

[0091] Through this parsing process, the system generates a response containing the prompt for each input question. and Structured data records with two core fields.

[0092] Step S330: Self-reflection and immediate correction. Before passing the structured data record to the next stage, the system performs an optional, lightweight self-reflection check. This check is achieved by submitting a rapid validation task to a pre-trained large language model.

[0093] The input to this verification task includes the generated... and And a verification instruction, such as: check for any obvious logical contradictions or inconsistencies between the following inference chain and the actual output. Answer only yes or no.

[0094] If the model returns "yes," it indicates that an inconsistency has been detected. The system can then trigger an immediate correction loop: the detected inconsistency information is used as feedback and resubmitted along with the original input to the pre-trained multimodal large model, requesting it to correct and regenerate. If the model returns "no," or this step is configured to be skipped, the structured data record is considered to have been successfully generated and is passed to the next stage for a comprehensive quality assessment.

[0095] See attached document Figure 5 The multi-dimensional quality assessment and quantification phase aims to conduct a comprehensive and quantifiable quality check on the generated structured data.

[0096] The fourth stage involves multi-dimensional quality assessment and quantification, which includes the following steps:

[0097] Step S410: Assessment Task Construction and Input Preparation. This step receives a structured data record from the previous stage, containing the prompt, inference chain, and formal output. Simultaneously, this step also obtains preliminary annotation information from Stage 1 corresponding to this record (including task category, key objects, and preliminary inference conclusions).

[0098] The system constructs three independent scoring tasks for each data record to be evaluated, corresponding to the following three quality dimensions:

[0099] Logical coherence: Evaluate whether the logical relationships between the steps within the reasoning chain are reasonable and whether there are any contradictions.

[0100] Factual accuracy: Evaluate whether the official output is consistent with the objective facts presented by the initial annotation information or sample data.

[0101] Reasoning completeness: Evaluate whether the reasoning chain includes all the key intermediate steps necessary to arrive at the formal output.

[0102] Each scoring task's data structure includes the following:

[0103] Scoring Dimensions Definition: Clearly indicate whether the current task is to assess logical coherence, factual accuracy, or reasoning completeness.

[0104] Detailed scoring criteria: Provides a specific and actionable set of scoring rules to guide the scoring of the evaluation model. For example, the scoring criteria for factual accuracy can be refined as follows: if the conclusion is completely consistent with the initial annotation, 90-100 points are awarded; if it is partially consistent, 60-89 points are awarded; if it is completely inconsistent, 0-59 points are awarded.

[0105] Data to be evaluated includes the prompt, inference chain, formal output, and preliminary annotation information.

[0106] Step S420: Call the evaluation model for independent scoring. The system submits the three scoring tasks constructed in step S410 one by one to an independent, pre-specified pre-trained large language model (this model is called the evaluator). The evaluator is configured to return an integer score between 0 and 100 and a brief scoring statement for each task request.

[0107] For example, for a logical coherence scoring task, the system sends the following request to the evaluator: Task: Logical coherence scoring. Scoring criteria: [Detailed criteria]. Data: [Data to be evaluated]. Please output a score between 0 and 100 and a one-sentence reason for the score.

[0108] The system collects three scores returned by the evaluator and records them as logical coherence scores. Accuracy score of facts and reasoning completeness score .

[0109] Step S430: Calculation and Determination of Overall Quality Score. This step calculates an overall quality score based on the obtained scores from the three dimensions. First, the system retrieves the corresponding weight coefficients from a preset weight configuration based on the task category of the current data record. , , This weight configuration predefines different weight combinations for different task categories to suit the quality emphasis of different tasks. For example, for the visual inspection task category, the weight coefficient can be set to... , , For reasoning question-answering tasks, the weighting coefficient can be set as follows: , , For decision-making and planning task categories, the weighting coefficient can be set as follows: , , .

[0110] The system calculates the overall quality score based on the following formula. :

[0111] ;

[0112] in, For the overall quality score; , , The scores are respectively based on logical coherence, factual accuracy, and complete reasoning; , , These are preset weighting coefficients that can be adjusted according to the task type, and satisfy the following conditions: .

[0113] Calculate After the value is calculated, the system compares it with a preset quality threshold (e.g., 75 points). If the value is greater than or equal to the threshold, the system adds a high-quality tag to the data record. If the value is below the threshold, a low-quality flag is added.

[0114] Ultimately, this data record, along with its scores for each dimension, overall score, scoring criteria, and quality indicators, is transmitted to subsequent processes.

[0115] See attached document Figure 6 The automated correction and reconstruction phase for low-quality data executes a feedback-driven iterative correction process on data records that have been evaluated and determined to be unqualified.

[0116] The fifth stage involves the automated correction and reconstruction of low-quality data, which includes the following steps:

[0117] Step S510: Correcting the generation and formatting of feedback information. This step receives data records marked as low quality from the previous stage. The data structure of this record includes: original sample data, question prompt, inference chain, formal output, and scores for each dimension. , , ), Overall score And the text-based scoring criteria generated by the evaluator.

[0118] Based on the received information, the system automatically generates a structured correction instruction. This instruction is either a text string or a data object, and its content format is predefined, for example: Correction Task: Regenerate the inference chain and formal output based on the following evaluation results. Logical coherence score: [ Evaluation basis: [corresponding scoring basis text]; Factual accuracy score: [ Evaluation criteria: [corresponding scoring criteria text]; Reasoning completeness score: [ Evaluation criteria: [corresponding scoring criteria text]. This instruction clearly quantifies and textually describes the dimensions with low scores and their reasons.

[0119] Step S520: Automated correction execution based on feedback. The system integrates the structured correction instructions generated in step S510, the original prompt, and the original sample data to form a new input sequence.

[0120] The input sequence is submitted to the same pre-trained multimodal large model used in Phase 3 to generate the inference chain and final output. Upon receiving this input sequence containing explicit correction feedback, the model generates a new set of inference chains and final outputs. The system parses this output and uses it to overwrite the invalid inference chain and final output fields in the original data record, forming an updated data record awaiting review.

[0121] Step S530: Implementation of multi-round correction and data discard strategy. To avoid infinite correction loops, the system associates a correction count counter with each data record, with an initial value of 0. After performing the correction operation in step S520, the system increments the correction count counter corresponding to that data record by 1.

[0122] The updated data records were resent to Phase Four (Multi-dimensional Quality Assessment and Quantification) for re-scoring. The newly calculated overall quality score... If the quality threshold is greater than or equal to the preset quality threshold, the quality flag of the data record is updated to high quality, and its correction process ends, ready to enter the next stage.

[0123] If the newly calculated overall quality score If the data record is still below the quality threshold, the system checks the value of its correction count counter. If the value is less than the preset maximum number of corrections (e.g., 3 times), the data record will return to step S510 to start a new round of correction. If the value of the correction count counter has reached the maximum number of corrections, the data record will be marked as a correction failure by the system and ultimately discarded, and will not proceed to any subsequent processes.

[0124] See attached document Figure 7 The continuous fine-tuning phase of the multimodal large model is used to update the parameters of the pre-trained multimodal large model at the core of the system using the high-quality data records generated and validated in the aforementioned phases.

[0125] The sixth stage involves continuous fine-tuning of the multimodal large model, specifically including the following steps:

[0126] Step S610: Construction and preparation of high-quality dataset. This step first aggregates all data records marked as high-quality in Phase 4 from a temporary storage area. Next, the system performs format conversion on each data record, transforming it into a structured data unit (e.g., a JSON object) that meets the requirements of model fine-tuning and contains input text and target output text.

[0127] In this JSON object, the input text consists of the original query prompt, while the target output text is formed by concatenating the contents of the inference chain and the actual output fields from the data record, separated by a specific delimiter. In this way, the system processes all high-quality data records into a high-quality dataset that can be directly read by the model for supervised fine-tuning.

[0128] Step S620: Selection and execution of fine-tuning strategy. This step is triggered when the size of the high-quality dataset built in step S610 reaches a preset batch processing threshold (e.g., accumulating to 1000 records), or when the time interval since the last fine-tuning reaches a preset period (e.g., 24 hours).

[0129] The system first loads the current version of the pre-trained multimodal large model. Then, based on the system's preset configuration, it selects and executes a fine-tuning strategy. These strategies include, but are not limited to:

[0130] Full parameter fine-tuning: Under this strategy, the training algorithm will update all parameters of the model using a high-quality dataset.

[0131] Efficient parameter fine-tuning: For example, a low-rank adaptation method is used, which injects trainable low-rank matrices between the existing layers of the model. During fine-tuning, only the parameters of these newly added matrices are updated, while most of the parameters of the original model are frozen.

[0132] The selection of fine-tuning strategies and related hyperparameters (such as learning rate, training epochs, and rank r in LoRA) are all managed as configurable items of the system.

[0133] Step S630: Model performance verification and version management. After the fine-tuning process is completed, the system saves the updated model parameters (or adapter parameters in LoRA) as a new model file with a unique version number or timestamp.

[0134] In an optional validation step, the system can use a reserved, high-quality validation set that was not involved in this fine-tuning to evaluate the performance of the new model version. The system records its performance by calculating the loss value of the new model on that validation set or a task-specific evaluation metric (e.g., text generation perplexity or task-specific accuracy).

[0135] After saving and optional verification, the identifier of the new version model (e.g., its storage path and version number) will be passed to the next stage of the closed-loop control flow for the system to call in subsequent data generation tasks.

[0136] See attached document Figure 8 The closed-loop control and automation iteration phase is responsible for scheduling the overall workflow of the system and applying the results of model updates to subsequent data generation tasks to form a continuously running loop.

[0137] The seventh stage is closed-loop control and automation iteration, which specifically includes the following steps:

[0138] Step S710: Receive the model update signal and update the system configuration. This step is triggered when Phase Six (continuous fine-tuning of the multimodal large model) is completed. After completing one model fine-tuning, Phase Six generates a completion signal containing a new model version identifier (e.g., the storage path and version number of the model parameter file).

[0139] Upon receiving this completion signal, the closed-loop control module 70 immediately executes a system configuration update operation. Specifically, this operation involves accessing a central configuration file or a configuration object in memory, locating the parameter specifying the path to the core pre-trained multimodal large model, and modifying its value to the newly received model version identifier. This configuration parameter is read by the Prompt generation module in phase two and the inference data generation module 30 in phase three. This update operation ensures that all subsequent tasks calling this core model will use the latest fine-tuned version.

[0140] Step S720: Trigger a new round of automated annotation process. After the model configuration update in step S710 is completed, the closed-loop control module 70 queries the full multimodal dataset to identify data records that have not been processed, i.e., not selected as initial samples and not used in any previous automated iterations.

[0141] If the query results contain one or more unprocessed data records, the closed-loop control module 70 selects a preset number of batches and initiates a new round of automated data generation and annotation. If there is no unprocessed data, the system enters standby mode until new data is added to the full dataset.

[0142] Step S730: Perform automated iteration. Unlike the initial system run which starts from Phase 1, the new round of processes triggered by step S720 will begin directly from Phase 2 (multi-perspective Prompt generation and evaluation).

[0143] The closed-loop control module 70 takes the newly selected batch of unprocessed data as input and directly passes it to stage two. Subsequently, the system will sequentially execute stage two, stage three (generation of inference chain and formal output), stage four (multi-dimensional quality assessment and quantification), stage five (automatic correction and reconstruction of low-quality data), and stage six (continuous fine-tuning of multimodal large model). In stages two and three of this iteration, the core model called by the system is the model that has been updated to the latest version in step S710.

[0144] The final output of this iteration is another batch of high-quality data records, and an updated version of the model obtained by further fine-tuning these data records. Once phase six of this iteration is complete, the system will return to step S710 to begin the next closed-loop iteration. This process constitutes a complete closed loop of collaborative data and model updates, which can be repeated until all data in the full dataset has been processed.

[0145] In order to implement this invention, the pre-trained multimodal large model and the pre-trained large language model are described in detail below to clarify their functional requirements and interrelationships.

[0146] Regarding the core generative model (pre-trained multimodal large model):

[0147] The pre-trained multimodal large model invoked in the inference data generation module of this invention is the core generation engine of the system. This model is not arbitrarily selected; it must possess specific technical capabilities: the ability to receive and jointly process inputs from at least two modalities (e.g., images and text), and generate coherent text containing inference logic based on text instructions.

[0148] In one specific embodiment, the model can be a Transformer-based model pre-trained on massive amounts of image and text data pairs. Its core function is to analyze and reason based on multimodal inputs and produce results in a structured form that includes inference chains and formal outputs.

[0149] Regarding assistive function models (pre-trained large language models):

[0150] In the self-reflection steps of the Prompt generation and evaluation module, the multi-dimensional quality evaluation module, and the inference data generation module, this invention calls upon pre-trained large language models to perform auxiliary tasks. These models each have their own functional requirements and focuses:

[0151] In the Prompt generation and evaluation module, the model used needs to have strong instruction following and role-playing capabilities so that it can evaluate the quality of the question Prompt from multiple preset perspectives such as users and developers.

[0152] In the multi-dimensional quality assessment module, this model is used as an evaluator. Therefore, it needs to have excellent text analysis and critical thinking skills, be able to quantitatively score the logic, factuality and completeness of the input reasoning chain and the formal output, and provide specific evaluation criteria.

[0153] In the self-reflection and immediate correction steps, the model focuses on logical consistency detection, that is, judging whether there is a contradiction between the reasoning process and the final conclusion.

[0154] Regarding the selection and relationship of model instances:

[0155] In practice, the pre-trained large language models used for different auxiliary steps in the aforementioned assistive function model can be the same model instance with sufficient generality, or they can be multiple different model instances selected for different tasks.

[0156] Option 1 (Single Model Instance): A large-scale general-purpose language model (e.g., a model with tens of billions of parameters) can be deployed and driven to complete different tasks such as evaluation, scoring, and logic detection by designing different instructions.

[0157] Option 2 (Multiple Model Instances): To optimize costs or pursue optimal performance for specific tasks, different, more targeted models can be selected for different auxiliary steps. For example, a model specifically optimized for logical reasoning can be chosen for the quality assessment module.

[0158] The core generative model (pre-trained multimodal large model) and the auxiliary function model (pre-trained large language model) mentioned above are, in principle, different models because they deal with fundamentally different input types and core tasks.

Claims

1. An automatic construction system for multimodal datasets based on thought chains and self-reflection, characterized in that: include: The data preprocessing module is used to extract samples from the full multimodal dataset and receive preliminary annotation information corresponding to the samples; The Prompt generation and evaluation module is connected to the data preprocessing module and is used to generate and evaluate the question prompts based on the samples and preliminary annotation information, so as to output qualified question prompts. The inference data generation module, connected to the Prompt generation and evaluation module, is used to call a pre-trained multimodal large model to generate structured data containing inference chains and formal outputs based on the qualified question Prompt and samples. A multi-dimensional quality assessment module, connected to the inference data generation module and the data preprocessing module, is used to calculate a comprehensive quality score based on the inference chain, the formal output and the preliminary annotation information, and to mark the structured data as high quality or low quality based on the comprehensive quality score. An automated correction module, connected to the multi-dimensional quality assessment module, is used to receive structured data marked as low quality and drive the inference data generation module to correct and regenerate based on the assessment results of the low-quality structured data. The model fine-tuning module, connected to the multi-dimensional quality assessment module, is used to perform fine-tuning operations on the pre-trained multimodal large model using structured data labeled as high quality. A closed-loop control module, connected to the model fine-tuning module and the data preprocessing module, is used to drive the data preprocessing module to start a new round of automated annotation process on the new data after the model fine-tuning module completes the fine-tuning operation, using the updated pre-trained multimodal large model.

2. The automatic construction system for multimodal datasets based on thought chains and self-reflection according to claim 1, characterized in that, The Prompt generation and evaluation module includes a Prompt evaluation unit, which is configured to input each generated question Prompt into a pre-trained large language model and drive the pre-trained large language model to evaluate the question Prompt from four preset perspectives: user, developer, domain expert, and security, and output a qualified or unqualified judgment result.

3. The automatic construction system for multimodal datasets based on thought chains and self-reflection according to claim 1, characterized in that, The inference data generation module is configured to: use an instruction template to guide the pre-trained multimodal large model to generate a text stream containing an inference chain and a formal output based on the qualified question prompt and samples, wherein the inference chain is guided by a preset inference chain start marker and the formal output is guided by a preset formal output start marker; the inference data generation module then parses the two start markers in the text stream to extract the inference chain and the formal output respectively, thereby generating the structured data.

4. The automatic construction system for multimodal datasets based on thought chains and self-reflection according to claim 1, characterized in that, The multi-dimensional quality assessment module is configured to call an independent pre-trained large language model as an evaluator to calculate quality index scores in three dimensions: logical coherence score, factual accuracy score, and reasoning completeness score.

5. The automatic construction system for multimodal datasets based on thought chains and self-reflection according to claim 4, characterized in that, The multi-dimensional quality assessment module obtains the comprehensive quality score by calculating the weighted sum of the logical coherence score, factual accuracy score, and reasoning completeness score; wherein, the weighting coefficients used for weighting are preset and can be adjusted according to the task type, and the sum of the weighting coefficients is 1.

6. The automatic construction system for multimodal datasets based on thought chains and self-reflection according to claim 4, characterized in that, The automated correction module is configured to integrate the scores of each dimension corresponding to the low-quality data and the scoring criteria given by the evaluator into a structured correction instruction, and submit the correction instruction, the original question prompt, and the sample data together to the pre-trained multimodal large model to drive the pre-trained multimodal large model to regenerate the inference chain and the formal output.

7. The automatic construction system for multimodal datasets based on thought chains and self-reflection according to claim 6, characterized in that, The automated correction module also includes a correction count counter to record the number of corrections for a single data entry; If the number of corrections exceeds the preset upper limit, and the data quality score is still lower than the quality threshold, the data will be marked as a correction failure and ultimately discarded.

8. The automatic construction system for multimodal datasets based on thought chains and self-reflection according to claim 1, characterized in that, The model fine-tuning module is configured to: start the fine-tuning process and execute the parameter efficient fine-tuning method to update the pre-trained multimodal large model when the number of high-quality data records collected reaches a preset batch processing threshold or the time interval since the last fine-tuning reaches a predetermined period.

9. The automatic construction system for multimodal datasets based on thought chains and self-reflection according to claim 1, characterized in that, The closed-loop control module is configured to: after receiving the model update completion signal from the model fine-tuning module, obtain the identifier of the new version model, update the central configuration file so that the Prompt generation and evaluation module and the inference data generation module call the updated model, and select unprocessed data batches to start the automated iteration starting from phase two.

10. The automatic construction system for multimodal datasets based on thought chains and self-reflection according to claim 1, characterized in that, The reasoning data generation module is also configured to perform a self-reflection and immediate correction step after generating the structured data; The self-reflection and immediate correction steps include: submitting a verification task to a pre-trained large language model to check whether there are logical contradictions or inconsistencies between the inference chain and the formal output; If an inconsistency is detected, the inconsistency information is used as feedback and resubmitted along with the original input to the pre-trained multimodal large model for immediate correction and regeneration.

Images