High-quality data set construction method and system for large model in vertical field

By segmenting and processing seed text data in vertical fields, generating and optimizing Q&A pairs, building a Zhicui big model, the problems of low efficiency, high cost and incomplete information extraction in the existing technology are solved, and efficient and accurate data set construction and information extraction are achieved.

CN119647595APending Publication Date: 2025-03-18NINGBO INST OF INFORMATION TECH APPL CHINESE ACAD OF SCI +1
0 Cites 5 Cited by

Patent Information

Application Number
CN202411723139.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-28
Publication Date
2025-03-18

AI Technical Summary

Technical Problem

The prior art is inefficient and costly when constructing high-quality vertical field datasets, and the information extraction is incomplete when processing long texts.

Method used

By selecting seed text data, splitting it into multiple text fragments, using the big model tool to generate seed Q&A pairs, and review and optimization. Then, the generated Q&A pair production instructions are used to generate a data set, fine-tune the base model, iteratively optimize the big model, and generate the Zhicui big model, which is used to automatically generate knowledge Q&A pairs.

Benefits of technology

It realizes efficient, accurate and low-cost construction of vertical field data sets, avoids a large amount of manual annotations, can fully extract long text information, and improves the performance of vertical field knowledge large models.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a vertical field large model-oriented high-quality data set construction method and system, and relates to the technical field of text processing, and the method comprises the steps: selecting seed text data, carrying out long text processing, segmenting the seed text data into a plurality of text segments, processing the text segments by using a large model tool to generate seed question and answer pairs, and constructing a seed question and answer pair; auditing and optimizing are carried out according to a preset auditing standard; making an instruction generation data set for the cue word, the text fragment and the seed question and answer pair according to a preset organization rule, performing fine adjustment on a pre-selected base model by using the instruction generation data set, and performing iterative optimization on the large model by testing the performance of the large model after fine adjustment and adjusting and optimizing parameters, thereby obtaining an intelligent extraction large model IELM; and based on the intelligent extraction big model, processing the text data by utilizing a preset script code, automatically generating knowledge question and answer pair data corresponding to the text data, and performing format conversion processing to obtain an instruction data set which is used for a training data set of the vertical domain knowledge big model.
Need to check novelty before this filing date? Find Prior Art