Task processing method and device based on multi-source data synthesis, equipment and medium

By using a unified configuration file and a multi-source data synthesis mechanism, the problem of inconsistent multi-source data synthesis in existing technologies has been solved, achieving efficient and stable multi-source dataset generation and improved task processing performance.

CN121997941APending Publication Date: 2026-05-08SHUCHUANG ARC LIGHT (SHENZHEN) TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
SHUCHUANG ARC LIGHT (SHENZHEN) TECHNOLOGY CO LTD
Filing Date
2026-02-02
Publication Date
2026-05-08

AI Technical Summary

Technical Problem

Existing data synthesis schemes suffer from problems such as fragmented construction links of multi-source data, insufficient utilization of context, strong heterogeneity of open datasets, unstable output of teacher models, lack of universal adaptation interfaces for multimodal data synthesis, and difficulty in generating domain knowledge boundaries according to instructions, resulting in poor task processing performance.

Method used

By acquiring a unified configuration file, processing local corpora based on an evidence-constrained synthesis mechanism, using an open dataset detection and filtering mechanism and a distillation data synthesis mechanism, a high-quality multi-source dataset is generated, and the training sample set is used to train the model to perform the target task.

Benefits of technology

It enables high-quality data synthesis from multiple data sources, improves the controllability, reproducibility, and consistency of task processing, reduces cross-domain migration costs, and enhances the efficiency of data synthesis and training effectiveness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121997941A_ABST
    Figure CN121997941A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of artificial intelligence, and provides a task processing method and device based on multi-source data synthesis, equipment and a medium, which can obtain a unified configuration file, generate a multi-source data synthesis task according to the unified configuration file, and ensure controllability and reproducibility. Executing a local corpus synthesis task based on an evidence constraint synthesis mechanism to obtain a local data set, executing an open data synthesis task based on an open data set detection screening mechanism to obtain an open data set, and executing a distillation data synthesis task based on a distillation type data synthesis mechanism to obtain a distillation data set; high-quality data synthesis of multiple data sources is realized; and combining the local data set, the open data set and the distillation data set to train the model, and executing the target task by using the trained model, thereby assisting in improving the task execution effect based on the synthesized high-quality multi-source data set.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a task processing method, apparatus, device, and medium based on multi-source data synthesis. Background Technology

[0002] In recent years, Large Language Models (LLMs) and multimodal large models have made significant progress in tasks such as text generation, dialogue, and visual understanding, and have been explored for application in vertical fields such as healthcare, finance, law, and education. To enable models to possess domain-specific knowledge and task capabilities, it is typically necessary to construct high-quality training data that matches the target task and conduct processes such as supervised fine-tuning, alignment training, or post-model training. However, in many industry scenarios, the availability of truly labeled data is limited by factors such as acquisition costs, copyright and privacy compliance restrictions, insufficient coverage, and delayed updates, making it difficult to meet the needs of model iteration for data scale and diversity.

[0003] To alleviate the problems of insufficient real data and long-tail coverage, data synthesis techniques have been widely researched and applied. However, existing data synthesis schemes still have the following shortcomings: (1) The data construction links from multiple sources are fragmented, lacking a unified process abstraction and consistent sample specifications.

[0004] Existing solutions typically employ different scripts or toolchains for local corpora, open datasets, and teacher-generated data, resulting in inconsistent data structures, field naming, and metadata recording methods. This makes it difficult to reuse and manage synthesized data within the same framework. Furthermore, subsequent quality control, filtering, export, and training integration require repeated adaptation, leading to high engineering costs and potential for inconsistencies in configuration.

[0005] (2) In local corpus-driven synthesis, insufficient use of context or evidence or weak constraint mechanisms can easily lead to inconsistencies in facts and homogenization of samples.

[0006] When relying solely on generative models for data construction or making insufficient use of local corpora, the generated content is prone to deviating from local private knowledge and factual details, producing erroneous content or "illusions" that are inconsistent with the corpus. On the other hand, when there is a lack of diversified sampling, deduplication, and coverage control of evidence fragments, the generated samples tend to concentrate on a small number of high-frequency patterns or expressions, resulting in excessively high sample similarity, high repetition, and insufficient diversity, which affects the training coverage and generalization effect.

[0007] (3) The datasets of open data platforms are highly heterogeneous, and field mapping and task matching are highly dependent on manual work, resulting in high quality screening costs.

[0008] Publicly available datasets vary significantly in field naming, structural organization, sample format, and labeling systems. Many datasets do not naturally conform to the input or output definitions of the target task. Existing construction methods often require manual interpretation of the dataset's structured metadata definitions and the writing of mapping and cleaning logic, which is inefficient and difficult to scale. Furthermore, they are prone to introducing data that is mismatched with the task or contains excessive noise, affecting the final training results.

[0009] (4) The teacher model distillation synthesis has problems with unstable output and format drift, and lacks reproducible generation constraints and consistency verification.

[0010] The data generated by the teacher model is usually random, and there may be issues such as output format not meeting requirements, inconsistent answer labeling, redundant or missing reasoning processes. In addition, the quality of the synthesized data fluctuates greatly, making it difficult to stably support subsequent training and comparison experiments.

[0011] (5) Multimodal data synthesis and training data organization lack universal adaptation interfaces, and the cost of extending to new modalities is high.

[0012] Existing solutions typically customize processing logic for specific modalities (such as images), making it difficult to reuse it for other modalities such as audio. At the same time, the binding methods between modal data and contextual information (subtitles, transcription, metadata) are not consistent, which requires rewriting a large number of processes when expanding to new modal tasks, affecting the efficiency of engineering iteration and system scalability.

[0013] (6) The controllability of data synthesis for different domains or tasks is insufficient, and the domain knowledge boundary and data form are difficult to be generated stably according to instructions.

[0014] Existing synthetic data solutions are often customized for a single domain or a single task template. This leads to the need to redesign prompts, rules, and post-processing procedures when switching to new domains (such as mathematics, finance, and healthcare) or new task formats (such as requiring CoT (Chain of Thought) step-by-step reasoning, answer labeling, and multi-turn dialogue structures). The generated results are unstable and difficult to reproduce. At the same time, cross-domain mixing, style drift, or samples inconsistent with the target domain are prone to occur, affecting the usability of the data and the training effect.

[0015] Because of the aforementioned shortcomings in the synthesized samples, the effectiveness of task processing will be indirectly affected. Summary of the Invention

[0016] In view of the above, it is necessary to provide a task processing method, apparatus, device and medium based on multi-source data synthesis, which can improve the task processing effect by improving the quality of multi-source data synthesis samples.

[0017] A task processing method based on multi-source data synthesis, the method comprising: A unified configuration file is obtained, and a multi-source data synthesis task is generated based on the configuration file; wherein, the multi-source data synthesis task includes a local corpus synthesis task, an open data synthesis task, and a distilled data synthesis task; The local dataset is obtained by performing the local corpus synthesis task based on the evidence-constrained synthesis mechanism, the open dataset is obtained by performing the open data synthesis task based on the open dataset detection and filtering mechanism, and the distilled dataset is obtained by performing the distilled data synthesis task based on the distilled data synthesis mechanism. The local dataset, the open dataset, and the distillation dataset are combined to obtain the training sample set; The initial model is trained using the aforementioned training sample set to obtain the target model; In response to the execution command of the target task, acquire the data to be processed; The target model is used to process the data to be processed to obtain the target task processing result.

[0018] A task processing device based on multi-source data synthesis, the task processing device based on multi-source data synthesis includes: A generation unit is used to obtain a unified configuration file and generate a multi-source data synthesis task based on the configuration file; wherein, the multi-source data synthesis task includes a local corpus synthesis task, an open data synthesis task, and a distilled data synthesis task; An execution unit is configured to perform the local corpus synthesis task based on an evidence-constrained synthesis mechanism to obtain a local dataset, perform the open data synthesis task based on an open dataset detection and filtering mechanism to obtain an open dataset, and perform the distilled data synthesis task based on a distilled data synthesis mechanism to obtain a distilled dataset. A combination unit is used to combine the local dataset, the open dataset, and the distillation dataset to obtain a training sample set; The training unit is used to train an initial model using the training sample set to obtain a target model; The acquisition unit is used to acquire data to be processed in response to the execution command of the target task; The processing unit is used to process the data to be processed using the target model to obtain the target task processing result.

[0019] A computer device, the computer device comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the task processing method based on multi-source data synthesis.

[0020] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the task processing method based on multi-source data synthesis.

[0021] As can be seen from the above technical solutions, the present invention can obtain a unified configuration file and generate a multi-source data synthesis task based on the unified configuration file, ensuring controllability and reproducibility; it performs a local dataset by performing a local corpus synthesis task based on an evidence-constrained synthesis mechanism, performs an open dataset by performing an open dataset synthesis task based on an open dataset detection and filtering mechanism, and performs a distilled dataset by performing a distilled data synthesis task based on a distilled data synthesis mechanism, thereby achieving high-quality data synthesis from multiple data sources; it combines the local dataset, the open dataset, and the distilled dataset to train a model, and uses the trained model to perform the target task, thereby improving the task execution effect based on the synthesized high-quality multi-source dataset. Attached Figure Description

[0022] Figure 1 This is a flowchart of a preferred embodiment of the task processing method based on multi-source data synthesis of the present invention.

[0023] Figure 2 This is a functional block diagram of a preferred embodiment of the task processing device based on multi-source data synthesis of the present invention.

[0024] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the task processing method based on multi-source data synthesis according to the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the task processing method based on multi-source data synthesis according to the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0027] The task processing method based on multi-source data synthesis is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0028] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0029] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0030] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0031] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0032] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0033] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0034] S10, obtain a unified configuration file and generate a multi-source data synthesis task based on the configuration file; wherein, the multi-source data synthesis task includes a local corpus synthesis task, an open data synthesis task, and a distilled data synthesis task (corresponding to the three data synthesis paths of Local, Web, and Distill, respectively).

[0035] In this embodiment, the configuration file may include, but is not limited to: task instructions, domain boundaries, input-output structure constraints (including CoT (Chain of Thought), answer labels, multi-turn structures, etc.), modality types, data source paths, quality control parameters, data evaluation and rewriting strategy parameters, training and evaluation related parameters, synthesis path selection and parameters, model and inference parameters, output directory and export format, etc.

[0036] The task instructions are natural language instructions used to define the dataset task type, content requirements, and generation goals, serving as one of the core constraints for synthesis and distillation.

[0037] The configuration file can reflect the specifications of the executable tasks.

[0038] By configuring a unified configuration file, synthesis tasks from multiple data sources can be uniformly configured, ensuring controllability and reproducibility. Furthermore, by configuring unified expression task instructions, domain boundaries, and input / output structure constraints (including but not limited to CoT stepwise reasoning, answer labeling, and multi-turn dialogue structures), the same engine can stably switch to data synthesis for different domains and task formats by changing configurations. Compared to existing methods that rely on manually modifying prompts or scripts, this reduces cross-domain migration costs and improves generation consistency.

[0039] In this embodiment, one of the tasks can be selected for execution according to different scenarios or needs.

[0040] In this embodiment, sample specifications can also be standardized, such as a standardized minimum sample structure that includes input and output fields. For multimodal samples, modal fields (e.g., references or identifiers for images and audio) and metadata fields (source identifiers, field mappings, scores, batch and version information, evidence fragment identifiers, etc.) can be extended on this basis.

[0041] This embodiment uses a unified dataset object to manage the sample set, supporting filtering, hierarchical export, and versioned records.

[0042] In this embodiment, by executing the multi-source data synthesis task, the data scale can be expanded, the task style can be improved, the data diversity can be enhanced, and real data can be replaced or supplemented in special scenarios such as privacy sensitivity.

[0043] S11, the local corpus synthesis task is performed based on the evidence-constrained synthesis mechanism to obtain a local dataset, the open data synthesis task is performed based on the open dataset detection and filtering mechanism to obtain an open dataset, and the distillation data synthesis task is performed based on the distillation data synthesis mechanism to obtain a distillation dataset.

[0044] In this embodiment, the process of performing the local corpus synthesis task based on the evidence-constrained synthesis mechanism to obtain the local dataset includes: Traverse the local document directory to obtain the documents to be processed; The document to be processed is subjected to structured parsing to obtain the parsed text; The parsed text is divided into multiple corpus units, and the multiple corpus units are written into the local corpus in the form of a structured file; Obtain task instructions, optional examples, and domain boundaries from the configuration file, and generate a search intent based on the task instructions, optional examples, and domain boundaries; Based on the search intent, a retrieval is performed in the local corpus to obtain a set of reference fragments; The input / output structure constraints, answer marking rules, and example patterns are obtained from the configuration file, and a joint hint is constructed based on the task instructions, the input / output structure constraints, the answer marking rules, the example patterns, and the reference fragment set. The generative model is retrieved, and data synthesis is performed based on the joint prompts using the generative model; The local dataset is constructed by obtaining the output data of the generated model.

[0045] The document to be processed may include documents such as PDFs. For example, the MinerU parser can be used to perform structured parsing on PDF documents.

[0046] Of course, different document parsing tools can be used to achieve equivalent parsing of the document.

[0047] The parsed text may include available text content and optional intermediate structure information. The intermediate structure information may include document hierarchy, formatting and style information, layout and position information, etc.

[0048] The system can perform recall based on any combination of keyword retrieval, semantic vector retrieval, sparse and dense fusion retrieval, and their re-ranking. Reference fragment sets can be obtained using Top-K, threshold filtering, deduplication, and diversified sampling strategies. For example, the BM25 (Best Matching 25) recall algorithm or other recall strategies can be used to achieve equivalent relevant fragment recall. Introducing reference fragments from the retrieval recall as generation constraints, and controlling the evidence coverage through deduplication and diversified sampling, ensures that the generated data is closer to local private knowledge and factual details, while reducing the problems of excessively high sample similarity, high repetition, and insufficient diversity caused by weak constraint generation. Compared with pure generation without evidence constraints, it reduces the risk of erroneous content inconsistent with local corpora. Evidence constraints combined with diversified sampling improve factual consistency and alleviate homogenization.

[0049] The generative model can be a large language model or other model with data generation capabilities.

[0050] In this embodiment, the step of using the generative model to perform data synthesis based on the joint prompts includes: The combined prompts are split into multiple batches, and data synthesis for each batch is performed in parallel using multi-threading; Record intermediate results and progress information during data synthesis; The intermediate results and progress information are used to support breakpoint resume and repeated execution data synthesis.

[0051] The above embodiments can improve data synthesis efficiency and support breakpoint resume and repeated execution of data synthesis, further enhancing the stability of data synthesis.

[0052] In this embodiment, the output data of the generated model can also be subjected to structured parsing and format verification to form a sample that meets the unified sample specification.

[0053] In the above embodiments, an integrated synthesis method is proposed for local unstructured corpora (including but not limited to PDFs, scanned document parsing results, and structured exported chapter texts). First, the local document is parsed to obtain text content and structured intermediate information that can be used for downstream retrieval and constraint generation. Then, the parsed text is segmented into blocks to generate a set of searchable corpus units, and the segmentation results are persistently saved in a general structured format to support breakpoint resumption and reuse. Next, combining task instructions, input / output format constraints, and optional examples, a large language model is used to automatically extract domain keywords, and the domain information is used as keyword enhancement signals, ensuring that the retrieval intent simultaneously includes "task semantic requirements." The system first identifies the "user-specified domain boundary" and then retrieves task-related reference fragments from the local corpus based on a retrieval and recall algorithm. This retrieval and recall algorithm includes, but is not limited to, keyword-based or sparse representation-based retrieval, semantic retrieval based on dense vector representation, hybrid retrieval combining sparse and dense representations, and learning and ranking or re-ranking of the retrieved results. Top-K, threshold filtering, deduplication, randomization, or stratified sampling strategies can be set to control coverage and diversity. Finally, a joint prompt is constructed consisting of "task instructions + input format constraints + output format constraints (including answer extraction rules) + example pattern + retrieval evidence fragments," and the generation model is called in batches to generate structured samples. By introducing searchable local evidence fragments as generation constraints and performing deduplication and diversified sampling on the evidence fragments, the generation is transformed from generation that relies solely on model parameterized knowledge to generation that is constrained by local corpus evidence and controlled by sampling. This improves the consistency of sample domain, the reliability of facts, and the compliance of format, while significantly reducing the common problems of sample homogenization, excessive similarity, and excessive repetition when generating without corpus or with weak constraints, and reducing the proportion of illusion samples that are inconsistent with the local corpus.

[0054] In this embodiment, the process of obtaining the open dataset by performing the open data synthesis task based on the open dataset detection and filtering mechanism includes: Generate a set of search keywords based on the task instructions and the domain boundaries; By specifying an open data platform interface, a search is performed based on the set of search keywords to obtain a candidate dataset including multiple subsets; Obtain available configuration and data partitioning information from the candidate dataset; Based on the available configuration and the data partitioning information, a preset number of sample rows are extracted from the candidate dataset as each probe sample; Identify the set of fields for each probe sample, and determine the input and output fields of each probe sample through field mapping; Based on the task instructions and the input and output fields of each probe sample, a consistency and quality score is performed on each probe sample to obtain a quantitative quality value for each probe sample. The detection samples whose quality quantification value is higher than a preset threshold are obtained as each candidate sample, and the subset corresponding to each candidate sample is obtained as each candidate subset according to the data partitioning information; Configure the extraction quota for each candidate subset; Samples are drawn from the corresponding candidate subset according to the extraction quota; The extracted samples are format-converted according to the input-output structure constraints, and the converted data is used to construct the open dataset.

[0055] For example, the specified open data platform can be HuggingFace (an open-source dataset ecosystem and toolkit) or other open data platforms.

[0056] Of course, other equivalent candidate dataset retrieval algorithms can also be used in this embodiment, which will not be elaborated here.

[0057] In particular, when determining the input and output fields of each probe sample through field mapping, if it contains multiple modes, the modality field can also be determined simultaneously.

[0058] The quality quantification value can be used to reflect the availability and quality of the dataset. For example, a multi-dimensional quality assessment of sample rows can be performed based on a large language model, including: first, identifying the input and output fields in the sample rows that correspond to the task (adding modal fields if it is multimodal), then inputting the "input-output" sample and the task instructions into the quality assessment model, and scoring them on five dimensions: relevance, correctness, usefulness, clarity, and difficulty, thereby obtaining the quality quantification value of each probe sample.

[0059] The preset threshold is used to assist in the initial screening of high-quality samples.

[0060] The extraction quota can be set according to the configuration conditions satisfied by each candidate subset.

[0061] In addition, after extracting samples from the corresponding candidate subset according to the extraction quota, metadata such as source dataset identifier, split, configuration, field mapping relationship and score can be recorded simultaneously to facilitate evidence traceability.

[0062] The combination of lightweight probing with field self-adaptation and score-based filtering improves usability and reduces manual intervention. Candidate datasets are probed within the web path, and input / output field mappings are automatically identified. Then, filtering and quota extraction are performed based on task consistency and quality scores. Compared to existing methods that involve manually selecting datasets, writing field mappings, and applying cleaning rules, this approach improves dataset utilization efficiency and reduces filtering costs.

[0063] In this embodiment, the step of extracting a preset number of sample rows from the candidate dataset as each probe sample according to the available configuration and the data partitioning information includes: Available data is filtered from the candidate dataset based on the available configuration; According to the data partitioning information, sample rows are extracted from each subset corresponding to the available data as each probe sample; The sum of the sample sizes drawn from each subset is equal to the preset number.

[0064] The available configurations can include parameter settings supported by the dataset, such as whether streaming is supported, field types, modal types (text, image), and licensing agreements. These available configurations can help quickly determine whether the dataset is compatible with current technology and compliance requirements, avoiding format incompatibility or permission issues during subsequent processing.

[0065] The data partitioning information can be used to indicate pre-divided subsets of the dataset, such as training, validation, and test subsets. This data partitioning information clarifies the purpose of different subsets, allowing for targeted extraction based on task requirements. It ensures separation of data sources for subsequent model training, validation, and evaluation, preventing data leakage, and provides crucial information for metadata records, enhancing traceability.

[0066] The preset quantity corresponds to a small number of samples, sufficient for lightweight probing and analysis, to reduce computational load. For example, the dataset split and configuration can be obtained through the HuggingFace dataset service interface, and the first batch of sample rows can be read. Alternatively, other equivalent methods can be used to obtain sample rows.

[0067] Here, "split" refers to the partitioning of data, meaning that the dataset is pre-divided into different subsets (such as training set, validation set, and test set). It is a first-level dimension of the HuggingFace dataset, which helps to accurately read the dataset subset for a specific purpose, such as loading only the test set for evaluation.

[0068] Here, "config" refers to the dataset configuration, a secondary dimension designed by HuggingFace for multi-version, multi-scenario datasets. It is used to distinguish different variations of the same dataset, avoiding the need to store separate datasets for different variations. The config allows for quick switching between the required versions, reducing redundancy.

[0069] In the above embodiments, automatic data construction can be achieved without manually adapting the structured metadata definition schema of each dataset. First, a set of search keywords is generated based on task instructions and domain information for dataset retrieval on public data platforms (such as HuggingFace). Then, a lightweight probe is performed on each candidate dataset, that is, a small number of sample rows are extracted from the candidate dataset and its available field set is identified. Next, a large language model is used to automatically determine the input and output fields that best match the task in the candidate field set, realizing self-adaptive mapping of field naming and structural differences in heterogeneous datasets. Further, consistency scoring and quality scoring (such as combined scoring of dimensions such as relevance, correctness, helpfulness, clarity, and difficulty) are performed on the probe samples based on task instructions, and the scores are used as the basis for dataset selection and sampling strategies. Subsequently, extraction quotas are allocated to high-scoring datasets in sequence or by weight, and sufficient samples are extracted from high-scoring datasets until the target sample size is met. Finally, the extracted samples are format converted and normalized to meet the input and output format constraints of the target task, and metadata such as source dataset identifiers, probe scores, and field mapping relationships can be retained to support traceability and subsequent auditing. This embodiment solves the problems of synthesis efficiency and usability caused by heterogeneous, noisy, inconsistent fields, and uncontrollable quality of Web source datasets through an open dataset detection and screening mechanism of "field self-adaptation + quality detection scoring + quota allocation". It realizes automatic screening of multiple datasets and aggregation of high-quality samples.

[0070] In this embodiment, the process of obtaining a distillation dataset by performing the distillation data synthesis task based on the distillation-based data synthesis mechanism includes: Based on the task instructions, the input / output structure constraints, and the optional examples, prompt words are generated. The strong teacher model is invoked to generate a high-quality sample generation pattern based on the prompt words; Obtain format constraints and diversity constraints, and construct distillation generation prompts based on the format constraints, the diversity constraints, and the high-quality sample generation mode; The distillation-generated prompts are input into the strong teacher model in batches using multi-threading to obtain structured samples; The structured samples are subjected to structured parsing, format verification, and consistency stabilization processing, and the distillation dataset is constructed based on the processed data.

[0071] The strong teacher model refers to a large language model with powerful performance and excellent results. The strong teacher model can accurately understand task requirements and output high-quality demonstration samples.

[0072] The high-quality sample generation mode allows the strong teacher model to learn and summarize the general patterns of samples under the task from a small number of examples and task constraints, so as to extract features such as question structure, expression style, reasoning process presentation and answer format.

[0073] The diversity constraints are used to avoid paraphrasing examples, avoid repetitive patterns, etc.

[0074] The distillation dataset can be used for student model training or post-model training to achieve knowledge transfer and capability distillation from the strong teacher model to the training data.

[0075] Teacher model distillation transfers capabilities into structured training samples, enhancing cold-start and long-tail coverage. This embodiment uses a strong teacher model as the knowledge source to extract generation patterns and generate structured samples in batches for post-training of the student model, achieving knowledge transfer from the teacher model to the training data. Compared to relying solely on real-label data or single-source data augmentation, it can cover long-tail tasks and complex reasoning styles more quickly.

[0076] In the above embodiments, instruction distillation-style data synthesis was performed using a strong teacher model as the knowledge source. Given task instructions, input / output format constraints, and optional examples, the teacher model first extracts the generation pattern of high-quality samples. Then, under the joint constraints of the generation pattern and format constraints, structured samples (containing at least input and output fields) are generated in batches, and the generated results are subjected to structured parsing and basic verification.

[0077] This embodiment abstracts the "task definition - data source - evidence / context - generation - structured output" into a unified synthesis paradigm and provides three alternative synthesis paths within the same pipeline framework: local corpus-driven synthesis (Local), web dataset-selected synthesis (Web), and instruction distillation synthesis (Distill). Regardless of the data source and synthesis strategy used, it outputs structured training samples that meet the unified sample specifications (containing at least input and output fields, and can be extended to include modal fields such as images and audio, as well as metadata fields), thereby achieving standardization, automation, and reproducibility of multi-source data synthesis strategies.

[0078] This embodiment also introduces a unified sample specification and consistent output interface on the three synthesis paths: regardless of the internal processing details used by Local / Web / Distill, it outputs isomorphic training sample objects (containing at least input and output fields, and can be expanded to include modal fields and metadata fields), truly possessing pipeline synthesis capabilities with replaceable multi-source data synthesis strategies and unchanged downstream processing links.

[0079] This embodiment provides a controllable data generation mechanism for different task instructions and knowledge boundaries in different domains. It can generate training samples in different domains (such as mathematics, finance, and medicine) and different data forms (including but not limited to CoT stepwise reasoning, answer labeling, and multi-turn dialogue structures) on demand within a unified framework. This solves the problems of existing solutions relying on a large amount of manual rewriting of scripts or prompts, poor generation stability, and style drift when transferring data across domains.

[0080] This embodiment solves the problems of existing technologies that require separate development for local corpora, open datasets and teacher model distillation data, inconsistent sample specifications, and redundant input for field adaptation and format conversion. It achieves replaceable multi-source strategies and consistent downstream processing links.

[0081] In this embodiment, the three data synthesis paths—Local, Web, and Distill—can be unified under the same paradigm and output sample specification: synthesis constrained by local corpus evidence, synthesis filtered from open platform datasets, and distillation synthesis from teacher models can be replaced as needed. Compared with the existing methods of separate development, maintenance, and inconsistent sample structures, this reduces redundant development and adaptation work, and lowers the cost of heterogeneous data access and maintenance.

[0082] S12, combine the local dataset, the open dataset, and the distillation dataset to obtain the training sample set.

[0083] By combining the local dataset, the open dataset, and the distillation dataset, a high-quality training sample set from multiple data sources can be obtained.

[0084] S13, Use the training sample set to train the initial model to obtain the target model.

[0085] In this embodiment, the training sample set can be further split into a training set and a validation set. The initial model is trained using the training set, and the training effect is verified using the validation set, thereby obtaining the target model.

[0086] Training the initial model with high-quality multi-source datasets can effectively improve the model's training performance and generalization ability, thereby helping to better perform the corresponding tasks.

[0087] In this embodiment, the initial model may include artificial intelligence models such as prediction models, classification models, and inference models used to perform different tasks.

[0088] S14, in response to the execution instruction for the target task, acquire the data to be processed.

[0089] In this embodiment, when the data to be processed is detected to be uploaded to the designated platform, the execution instruction is triggered.

[0090] In this embodiment, the target task may include prediction tasks, reasoning tasks, classification tasks, etc.

[0091] S15, the target model is used to process the data to be processed to obtain the target task processing result.

[0092] In this embodiment, the data to be processed can be input into the target model, and the target task processing result can be determined based on the output of the target model.

[0093] In this embodiment, as applications evolve towards multimodal approaches, training data is gradually expanding from plain text to modalities such as images and audio, as well as joint samples with textual contextual information. For example, in tasks such as image question answering and audio understanding, in addition to the modal data itself, it is also necessary to construct corresponding questions, answers, and necessary contextual information (such as surrounding text, subtitles, or transcription and structured tags) to meet the training requirements of multimodal models. Therefore, data synthesis and data organization processes oriented towards multiple sources, multiple tasks, and multiple modalities have become an important foundational link in the engineering implementation of large-scale models, and are gradually showing a trend towards process-oriented, automated, and configurable technological development.

[0094] Specifically, the method further includes: Acquire modal data and contextual information corresponding to the modal data, and establish a mapping relationship between the modal data and the contextual information; The multimodal generation model is invoked to generate structured target samples based on the mapping relationship, and the modality field references of each target sample are marked. The labeled target samples are added to the training sample set.

[0095] The modal data may include images, audio, etc.

[0096] The contextual information may include surrounding text of the document, subtitles or transcriptions, metadata tags, etc.

[0097] The target sample is adapted to the data structure corresponding to the training sample set, such as including input and output fields.

[0098] This embodiment can define a unified sample structure and modality adaptation interface for different modalities (including, but not limited to, modality data acquisition, preprocessing or parsing, contextual information binding, model input organization and result verification, etc.).

[0099] The above embodiments enable the acquisition of modal data (such as images, audio files or their metadata, transcribed text, etc.) from local or network sources. Contextual information related to the modal content (such as surrounding text, subtitles or transcriptions, tags and structured metadata, etc.) is then bound to the modal data and input into the corresponding generative model to generate structured question-and-answer or instruction data. This mechanism can be extended to more modal tasks by adding modal adapters without changing the main pipeline structure. It solves the problems of high cost for extending to new modalities and inconsistent binding methods between modal data and contextual information in existing technologies, supporting the extension to multimodal tasks such as images and audio while maintaining the stability of the main process and generating trainable data.

[0100] As can be seen from the above technical solutions, the present invention can obtain a unified configuration file and generate a multi-source data synthesis task based on the unified configuration file, ensuring controllability and reproducibility; it performs a local dataset by performing a local corpus synthesis task based on an evidence-constrained synthesis mechanism, performs an open dataset by performing an open dataset synthesis task based on an open dataset detection and filtering mechanism, and performs a distilled dataset by performing a distilled data synthesis task based on a distilled data synthesis mechanism, thereby achieving high-quality data synthesis from multiple data sources; it combines the local dataset, the open dataset, and the distilled dataset to train a model, and uses the trained model to perform the target task, thereby improving the task execution effect based on the synthesized high-quality multi-source dataset.

[0101] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the task processing device based on multi-source data synthesis according to the present invention. The task processing device 11 based on multi-source data synthesis includes a generation unit 110, an execution unit 111, a combination unit 112, a training unit 113, an acquisition unit 114, and a processing unit 115. The module / unit referred to in this invention is a series of computer program segments that can be executed by a processor and perform a fixed function, and which are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0102] The generation unit 110 is used to obtain a unified configuration file and generate a multi-source data synthesis task according to the configuration file; wherein, the multi-source data synthesis task includes a local corpus synthesis task, an open data synthesis task, and a distilled data synthesis task. The execution unit 111 is used to perform the local corpus synthesis task based on the evidence constraint synthesis mechanism to obtain a local dataset, to perform the open data synthesis task based on the open dataset detection and filtering mechanism to obtain an open dataset, and to perform the distillation data synthesis task based on the distillation data synthesis mechanism to obtain a distillation dataset. The combination unit 112 is used to combine the local dataset, the open dataset, and the distillation dataset to obtain a training sample set; The training unit 113 is used to train an initial model using the training sample set to obtain a target model; The acquisition unit 114 is used to acquire data to be processed in response to the execution instruction of the target task; The processing unit 115 is used to process the data to be processed using the target model to obtain the target task processing result.

[0103] As can be seen from the above technical solutions, the present invention can obtain a unified configuration file and generate a multi-source data synthesis task based on the unified configuration file, ensuring controllability and reproducibility; it performs a local dataset by performing a local corpus synthesis task based on an evidence-constrained synthesis mechanism, performs an open dataset by performing an open dataset synthesis task based on an open dataset detection and filtering mechanism, and performs a distilled dataset by performing a distilled data synthesis task based on a distilled data synthesis mechanism, thereby achieving high-quality data synthesis from multiple data sources; it combines the local dataset, the open dataset, and the distilled dataset to train a model, and uses the trained model to perform the target task, thereby improving the task execution effect based on the synthesized high-quality multi-source dataset.

[0104] like Figure 3 The diagram shown is a schematic representation of the structure of a computer device that implements a task processing method based on multi-source data synthesis according to a preferred embodiment of the present invention.

[0105] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a task processing program based on multi-source data synthesis.

[0106] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0107] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0108] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a task processing program based on multi-source data synthesis, but also to temporarily store data that has been output or will be output.

[0109] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing task processing programs based on multi-source data synthesis) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0110] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various embodiments of the task processing method based on multi-source data synthesis described above, for example... Figure 1 The steps are shown.

[0111] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a generation unit 110, an execution unit 111, a combination unit 112, a training unit 113, an acquisition unit 114, and a processing unit 115.

[0112] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, a computer device, or a network device, etc.) or processor to execute portions of the task processing method based on multi-source data synthesis described in the various embodiments of the present invention.

[0113] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0114] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0115] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0116] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0117] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0118] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0119] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the computer device 1 and other computer devices.

[0120] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0121] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0122] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0123] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a task processing method based on multi-source data synthesis, and the processor 13 can execute the multiple instructions to achieve: A unified configuration file is obtained, and a multi-source data synthesis task is generated based on the configuration file; wherein, the multi-source data synthesis task includes a local corpus synthesis task, an open data synthesis task, and a distilled data synthesis task; The local dataset is obtained by performing the local corpus synthesis task based on the evidence-constrained synthesis mechanism, the open dataset is obtained by performing the open data synthesis task based on the open dataset detection and filtering mechanism, and the distilled dataset is obtained by performing the distilled data synthesis task based on the distilled data synthesis mechanism. The local dataset, the open dataset, and the distillation dataset are combined to obtain the training sample set; The initial model is trained using the aforementioned training sample set to obtain the target model; In response to the execution command of the target task, acquire the data to be processed; The target model is used to process the data to be processed to obtain the target task processing result.

[0124] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0125] It should be noted that all the data involved in this case was legally obtained.

[0126] If any AI models, software tools, or components not belonging to this company appear in the embodiments of this invention, they are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this invention has been obtained by an entity authorized (with the knowledge and consent) or fully authorized by all parties through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0127] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0128] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0129] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0131] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0132] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0133] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A task processing method based on multi-source data synthesis, characterized in that, The task processing method based on multi-source data synthesis includes: A unified configuration file is obtained, and a multi-source data synthesis task is generated based on the configuration file; wherein, the multi-source data synthesis task includes a local corpus synthesis task, an open data synthesis task, and a distilled data synthesis task; The local dataset is obtained by performing the local corpus synthesis task based on the evidence-constrained synthesis mechanism, the open dataset is obtained by performing the open data synthesis task based on the open dataset detection and filtering mechanism, and the distilled dataset is obtained by performing the distilled data synthesis task based on the distilled data synthesis mechanism. The local dataset, the open dataset, and the distillation dataset are combined to obtain the training sample set; The initial model is trained using the aforementioned training sample set to obtain the target model; In response to the execution command of the target task, acquire the data to be processed; The target model is used to process the data to be processed to obtain the target task processing result.

2. The task processing method based on multi-source data synthesis as described in claim 1, characterized in that, The local dataset obtained by performing the local corpus synthesis task based on the evidence-constrained synthesis mechanism includes: Traverse the local document directory to obtain the documents to be processed; The document to be processed is subjected to structured parsing to obtain the parsed text; The parsed text is divided into multiple corpus units, and the multiple corpus units are written into the local corpus in the form of a structured file; Obtain task instructions, optional examples, and domain boundaries from the configuration file, and generate a search intent based on the task instructions, optional examples, and domain boundaries; Based on the search intent, a retrieval is performed in the local corpus to obtain a set of reference fragments; The input / output structure constraints, answer marking rules, and example patterns are obtained from the configuration file, and a joint hint is constructed based on the task instructions, the input / output structure constraints, the answer marking rules, the example patterns, and the reference fragment set. The generative model is retrieved, and data synthesis is performed based on the joint prompts using the generative model; The local dataset is constructed by obtaining the output data of the generated model.

3. The task processing method based on multi-source data synthesis as described in claim 2, characterized in that, The process of using the generative model to synthesize data based on the joint prompts includes: The combined prompts are split into multiple batches, and data synthesis for each batch is performed in parallel using multi-threading; Record intermediate results and progress information during data synthesis; The intermediate results and progress information are used to support breakpoint resume and repeated execution data synthesis.

4. The task processing method based on multi-source data synthesis as described in claim 2, characterized in that, The open dataset obtained by performing the open data synthesis task based on the open dataset detection and filtering mechanism includes: A set of search keywords is generated based on the task instructions and the domain boundaries; By specifying an open data platform interface, a search is performed based on the set of search keywords to obtain a candidate dataset including multiple subsets; Obtain available configurations and data partitioning information from the candidate dataset; Based on the available configuration and the data partitioning information, a preset number of sample rows are extracted from the candidate dataset as each probe sample; Identify the set of fields for each probe sample, and determine the input and output fields of each probe sample through field mapping; Based on the task instructions and the input and output fields of each probe sample, a consistency and quality score is performed on each probe sample to obtain a quantitative quality value for each probe sample. The detection samples whose quality quantification value is higher than a preset threshold are obtained as each candidate sample, and the subset corresponding to each candidate sample is obtained as each candidate subset according to the data partitioning information; Configure the extraction quota for each candidate subset; Samples are drawn from the corresponding candidate subset according to the extraction quota; The extracted samples are format-converted according to the input-output structure constraints, and the converted data is used to construct the open dataset.

5. The task processing method based on multi-source data synthesis as described in claim 4, characterized in that, The step of extracting a preset number of sample rows from the candidate dataset as each probe sample based on the available configuration and the data partitioning information includes: Available data is filtered from the candidate dataset based on the available configuration; According to the data partitioning information, sample rows are extracted from each subset corresponding to the available data as each probe sample; The sum of the sample sizes drawn from each subset is equal to the preset number.

6. The task processing method based on multi-source data synthesis as described in claim 2, characterized in that, The distillation dataset obtained by performing the distillation data synthesis task based on the distillation-based data synthesis mechanism includes: Based on the task instructions, the input / output structure constraints, and the optional examples, prompt words are generated. The strong teacher model is invoked to generate a high-quality sample generation pattern based on the prompt words; Obtain format constraints and diversity constraints, and construct distillation generation prompts based on the format constraints, the diversity constraints, and the high-quality sample generation mode; The distillation-generated prompts are input into the strong teacher model in batches using multi-threading to obtain structured samples; The structured samples are subjected to structured parsing, format verification, and consistency stabilization processing, and the distillation dataset is constructed based on the processed data.

7. The task processing method based on multi-source data synthesis as described in claim 1, characterized in that, The method further includes: Acquire modal data and contextual information corresponding to the modal data, and establish a mapping relationship between the modal data and the contextual information; The multimodal generation model is invoked to generate structured target samples based on the mapping relationship, and the modality field references of each target sample are marked. The labeled target samples are added to the training sample set.

8. A task processing device based on multi-source data synthesis, characterized in that, The task processing device based on multi-source data synthesis includes: A generation unit is used to obtain a unified configuration file and generate a multi-source data synthesis task based on the configuration file; wherein, the multi-source data synthesis task includes a local corpus synthesis task, an open data synthesis task, and a distilled data synthesis task; An execution unit is configured to perform the local corpus synthesis task based on an evidence-constrained synthesis mechanism to obtain a local dataset, perform the open data synthesis task based on an open dataset detection and filtering mechanism to obtain an open dataset, and perform the distilled data synthesis task based on a distilled data synthesis mechanism to obtain a distilled dataset. A combination unit is used to combine the local dataset, the open dataset, and the distillation dataset to obtain a training sample set; The training unit is used to train an initial model using the training sample set to obtain a target model; The acquisition unit is used to acquire data to be processed in response to the execution command of the target task; The processing unit is used to process the data to be processed using the target model to obtain the target task processing result.

9. A computer device, characterized in that, The computer device includes: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the task processing method based on multi-source data synthesis as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the task processing method based on multi-source data synthesis as described in any one of claims 1 to 7.