System and method for making high-quality large model training data
By designing the production system of high -quality model training data, the problem of data standardization, low processing efficiency, and data quality control in existing technologies have been solved, and efficient and high -quality data preparation has been achieved. It provides large -scale model training. Stable data support.
Patent Information
- Application Number
- CN202411559630.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2025-01-21
AI Technical Summary
In the prior art, data standardization, low processing efficiency and difficult data quality control have made it difficult to prepare high-quality data required for large-scale model training efficiently.
A production system for high-quality large model training data is designed, including a data set management module, a data annotation module, a sensitive vocabulary module, a data processing module and a data analysis module. Through unified data format, visual data pipeline arrangement, evaluation indicators and Feedback loop to improve data processing efficiency and quality.
Realize the efficiency and quality control of data processing, reduce the time cost of data preparation, and ensure the high -quality data set required for large model training.
Smart Images
Figure CN119337938A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of large model data technology, and in particular to a system and method for producing high-quality large model training data. Background Art
[0002] In the field of artificial intelligence, especially in the large-scale model training process for tasks such as natural language processing, image recognition, and speech recognition, the quality and diversity of data play a vital role in the performance of the final model. With the development of deep learning technology, the demand for training data for large models is increasing, which is reflected in the quantity, quality, and diversity of data. However, in the existing technology, there are still the following problems about how to efficiently prepare high-quality large-scale training data sets that meet specific needs: 1. Data standardization issues: Currently, many publicly available data sets lack unified standards or clearly defined data format specifications, making it difficult to directly merge and use data from different sources; 2. Inefficient data processing: The process of converting raw data into a form that can be used for training often involves multiple steps, including cleaning, labeling, and feature extraction, which may lead to a significant increase in time costs; 3. Data quality issues: Due to the lack of systematic evaluation indicators and effective feedback loop mechanisms, it is difficult to maintain long-term stable data quality. Summary of the invention
[0003] The purpose of the present invention is to propose a system and method for producing high-quality large model training data in response to the above-mentioned technical problems.
[0004] A system for producing high-quality large-model training data, including a data set management module, a data annotation module, a sensitive word library module, a data processing module, and a data analysis module; The data set management module defines data types, stores data in a versioned manner, imports data incrementally, exports data, and publishes data sets; The data annotation module annotates different data set versions; The sensitive word library module defines nine categories of sensitive word libraries, each of which has built-in sensitive words and supports custom word libraries; The data processing module provides a visual and draggable data pipeline function to clean, enhance, filter and arrange the operation sequence of the data, and generate a new data set version after the processing is completed; The data analysis module provides data evaluation indicators, conducts a comprehensive inspection of the data, and takes measures to solve potential problems; The large model is Qwen1.5-14B-Chat; The Qwen1.5-14B-Chat function is optimized for dialogue and chat scenarios. Based on the Transformer architecture, it provides high-quality natural language understanding and generation capabilities and is suitable for a variety of application scenarios.
[0005] Further, a system for producing high-quality large-model training data, the data types include Prompt+Response, Prompt+Multi-Response, Plain Text, Promp Set, Prompt-chosen-rejected; The versioned storage is implemented through MongoDB database and MinIO objects; The nine categories of sensitive word libraries include but are not limited to illegal words related to guns and explosives, pornographic words, and URLs.
[0006] Further, a system for producing high-quality large-model training data, wherein the data processing module includes a data cleaning submodule, a data enhancement submodule, and a vocabulary filtering submodule; The data cleaning module has preset cleaning operators, uses the BERT model to generate a language model, and uses the JIEBA model to perform Chinese word segmentation; The cleaning operators include but are not limited to normalizing spaces, converting traditional Chinese to simplified Chinese, and text repetition rate; The data enhancement submodule uses the large model prompt technology to enhance the data to achieve data generalization effect; The word library filtering submodule filters the sensitive words defined by the sensitive word library module.
[0007] Further, a system for producing high-quality large-model training data, the Qwen1.5-14B-Chat includes a parameter module, an optimization dialogue module, a knowledge module, a language module, and a security module; The parameter module includes 14 billion parameters, which can capture complex language patterns and subtle semantic differences, improving the model's ability to understand and generate long texts. The optimization module is professionally trained to handle multiple rounds of conversations, maintain contextual coherence, understand and respond to user questions or statements, and maintain the fluency and relevance of the conversation; The training data of the knowledge module covers a wide range of subject areas, including general common sense and professional knowledge. Through pre-training and fine-tuning, the large model can display knowledge background on different topics. The language module supports multiple languages, and is mainly optimized for Chinese and English; The security module follows strict data privacy protection principles, implements security and compliance requirements, and takes measures to avoid generating harmful, false or biased content.
[0008] Further, a system for producing high-quality large-model training data, wherein the JIEBA model includes a dict.txt dictionary and a custom dictionary; The dict.txt dictionary includes more than 20,000 words, listing the number of occurrences and part of speech of each word; The terms include but are not limited to JIEBA word segmentation, which belongs to the probabilistic language model word segmentation. The task is to find a segmentation scheme S from all the results obtained by full segmentation, so that the probability P(S) of the scheme S is maximized; The custom dictionary expands the vocabulary and improves the accuracy of word segmentation; The JIEBA model supports traditional Chinese word segmentation and English word segmentation to meet the needs of different scenarios.
[0009] A method for preparing high-quality large model training data includes the following sub-steps: S1: Define metadata; S2: import data; S21: Import local data; The local data includes json, xlsx, and txt formats; S22: Import existing data sets on the platform and reuse historical data; S3: Convert data format; S31: Convert the imported plain text data into training format; S32: converting the imported question pair data into a format for storage in a text database; S4: stores data; S5: Process data; S6, Analyze data.
[0010] Furthermore, a method for producing high-quality large model training data, wherein S1 comprises the following sub-steps: S11: define data type information; S12: define data import method; S13: define data import template; S14: define sensitive word library type; S15: Define sensitive words in the sensitive word library.
[0011] Furthermore, a method for producing high-quality large model training data, said S4 comprises the following sub-steps: S41: All data is stored in a shared file system to be shared in the distributed system; S42: The question pair data is stored in a document database; S43: All data storage is stored in a version management manner.
[0012] Furthermore, a method for producing high-quality large model training data, said S5 comprises the following sub-steps: S51: Perform data cleaning on specific versions of the dataset; S52: Perform data enhancement on specific versions of the dataset; S53: Filter data of a specific version of the data set; S54: Select S51 to S53 for visual arrangement of the specific version of the data set; S55: After the visual arrangement processing is completed, new data set version data is generated.
[0013] The beneficial effects of the present invention are as follows: through a system and method for producing high-quality large-model training data, large-model training data standards and specifications are defined, data templates and unified data formats are provided; visual drag-and-drop data pipeline orchestration management is provided to improve data processing efficiency; evaluation indicators are improved, data is comprehensively checked, and core data assets of the enterprise are precipitated; front-end and back-end programs and algorithms are included, processing details are encapsulated, and usage complexity is reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a structural diagram of a system for producing high-quality large-model training data.
[0015] Figure 2 It is a flowchart of a method for producing high-quality large model training data. DETAILED DESCRIPTION
[0016] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described with reference to the accompanying drawings.
[0017] As attached Figure 1 As shown, a system for producing high-quality large-model training data includes a data set management module, a data annotation module, a sensitive word library module, a data processing module, and a data analysis module; The data set management module defines data types, stores data in a versioned manner, imports data incrementally, exports data, and publishes data sets; The data annotation module annotates different data set versions; The sensitive word library module defines nine categories of sensitive word libraries, each of which has built-in sensitive words and supports custom word libraries; The data processing module provides a visual and draggable data pipeline function to clean, enhance, filter and arrange the operation sequence of the data, and generate a new data set version after the processing is completed; The data analysis module provides data evaluation indicators, conducts a comprehensive inspection of the data, and takes measures to solve potential problems; The large model is Qwen1.5-14B-Chat; The Qwen1.5-14B-Chat function is optimized for dialogue and chat scenarios. Based on the Transformer architecture, it provides high-quality natural language understanding and generation capabilities and is suitable for a variety of application scenarios.
[0018] Further, a system for producing high-quality large-model training data, the data types include Prompt+Response, Prompt+Multi-Response, Plain Text, Promp Set, Prompt-chosen-rejected; The versioned storage is implemented through MongoDB database and MinIO objects; The nine categories of sensitive word libraries include but are not limited to illegal words related to guns and explosives, pornographic words, and URLs.
[0019] Further, a system for producing high-quality large-model training data, wherein the data processing module includes a data cleaning submodule, a data enhancement submodule, and a vocabulary filtering submodule; The data cleaning module has preset cleaning operators, uses the BERT model to generate a language model, and uses the JIEBA model to perform Chinese word segmentation; The cleaning operators include but are not limited to normalizing spaces, converting traditional Chinese to simplified Chinese, and text repetition rate; The data enhancement submodule uses the large model prompt technology to enhance the data to achieve data generalization effect; The word library filtering submodule filters the sensitive words defined by the sensitive word library module.
[0020] Further, a system for producing high-quality large-model training data, the Qwen1.5-14B-Chat includes a parameter module, an optimization dialogue module, a knowledge module, a language module, and a security module; The parameter module includes 14 billion parameters, which can capture complex language patterns and subtle semantic differences, improving the model's ability to understand and generate long texts. The optimization module is professionally trained to handle multiple rounds of conversations, maintain contextual coherence, understand and respond to user questions or statements, and maintain the fluency and relevance of the conversation; The training data of the knowledge module covers a wide range of subject areas, including general common sense and professional knowledge. Through pre-training and fine-tuning, the large model can display knowledge background on different topics. The language module supports multiple languages, and is mainly optimized for Chinese and English; The security module follows strict data privacy protection principles, implements security and compliance requirements, and takes measures to avoid generating harmful, false or biased content.
[0021] Further, a system for producing high-quality large-model training data, wherein the JIEBA model includes a dict.txt dictionary and a custom dictionary; The dict.txt dictionary includes more than 20,000 words, listing the number of occurrences and part of speech of each word; The terms include but are not limited to JIEBA word segmentation, which belongs to the probabilistic language model word segmentation. The task is to find a segmentation scheme S from all the results obtained by full segmentation, so that the probability P(S) of the scheme S is maximized; The custom dictionary expands the vocabulary and improves the accuracy of word segmentation; The JIEBA model supports traditional Chinese word segmentation and English word segmentation to meet the needs of different scenarios.
[0022] A method for preparing high-quality large model training data includes the following sub-steps: S1: Define metadata; S2: import data; S21: Import local data; The local data includes json, xlsx, and txt formats; S22: Import existing data sets on the platform and reuse historical data; S3: Convert data format; S31: Convert the imported plain text data into training format; S32: converting the imported question pair data into a format for storage in a text database; S4: stores data; S5: Process data; S6, Analyze data.
[0023] Furthermore, a method for producing high-quality large model training data, wherein S1 comprises the following sub-steps: S11: define data type information; S12: define data import method; S13: define data import template; S14: define sensitive word library type; S15: Define sensitive words in the sensitive word library.
[0024] Furthermore, a method for producing high-quality large model training data, said S4 comprises the following sub-steps: S41: All data is stored in a shared file system to be shared in the distributed system; S42: The question pair data is stored in a document database; S43: All data storage is stored in a version management manner.
[0025] Furthermore, a method for producing high-quality large model training data, said S5 comprises the following sub-steps: S51: Perform data cleaning on specific versions of the dataset; S52: Perform data enhancement on specific versions of the dataset; S53: Filter data of a specific version of the data set; S54: Select S51 to S53 for visual arrangement of the specific version of the data set; S55: After the visual arrangement processing is completed, new data set version data is generated.
[0026] Specific embodiment 1 Bert model: The BERT model is based on the Transformer architecture, a neural network model with bidirectional depth. It applies bidirectional training of transformers to language modeling. In contrast to previous efforts to view text sequences from left to right or to combine left-to-right and right-to-left training, the BERT model uses a new technology called Masked Language Modeling, which allows bidirectional training in models that were not possible before. In its vanilla form, the transformer includes two independent mechanisms - an encoder that reads text input and a decoder that produces task predictions. Since the goal of the BERT model is to generate a language model, only the encoder mechanism is required.
[0027] Specific embodiment 2 MongoDB database: MongoDB is an open source, document-based NoSQL database that uses a JSON-like format called BSON, or Binary JSON, to store data. MongoDB is a non-relational NoSQL database that is particularly suitable for processing large amounts of unstructured or semi-structured data.
[0028] Following are some key features of MongoDB: Document storage: MongoDB stores data in the form of documents. Each document is a collection of key-value pairs, similar to a JSON object. This model is very suitable for storing hierarchical data and allows the flexibility to add new fields without pre-defining the entire structure. Dynamic mode: Unlike traditional relational databases, MongoDB does not require a fixed table structure to be defined in advance, which means that different documents in the same collection, equivalent to a table in a relational database, can have completely different structures; High scalability: MongoDB supports horizontal expansion and can distribute data across multiple servers through sharding technology, thereby improving read and write performance and storage capacity; Rich query language: MongoDB provides powerful query capabilities, including complex aggregation operations and full-text search; Built-in replication and high availability: Data redundancy is achieved through replica sets, ensuring data security and high availability of the system; Support for multiple storage engines: Users can choose different storage engines according to their needs, such as WiredTiger, MMAPv1, etc.
[0029] Index support: To speed up queries, MongoDB allows you to create indexes for any field in a document, and even composite indexes for arrays and nested documents; Transaction support: Starting from version 4.0, MongoDB introduced multi-document transaction support, allowing atomic operations to be performed across multiple documents, collections, and even databases; Community and support: As an open source project, MongoDB has an active community that provides a large number of resources and tools, and there are also commercial versions that provide professional technical support and services.
[0030] Specific embodiment 3 MinIO object storage system: MinIO is a high-performance object storage system that is compatible with the Amazon S3 API, which means it can be seamlessly integrated with widely used cloud storage services. MinIO is designed for private cloud and edge computing environments, can handle large-scale data sets, and supports a variety of data protection mechanisms, such as Erasure Coding to ensure high availability and persistence of data.
[0031] The main features are: High performance: MinIO can achieve very high throughput and low latency on standard hardware, which is particularly suitable for large-scale data applications that require fast read and write speeds, such as machine learning and big data analysis; Compatible: Fully compatible with the Amazon S3 API, which means that all tools, applications, and services that use S3 can be directly connected to MinIO. This compatibility makes it easy to migrate existing S3-based workloads. Distributed architecture: MinIO can be deployed as a single node or a distributed multi-node cluster. In distributed mode, data redundancy is provided through erasure coding, ensuring that data is not lost even if multiple nodes fail. Security: Supports TLS / SSL encrypted transmission to ensure data security during transmission, and provides authentication and authorization mechanisms, including policy-based access control. Supports server-side encryption, which can encrypt data stored in MinIO; Easy to deploy and manage: MinIO can be easily deployed in containerized environments such as Kubernetes. It provides command line tools and graphical interfaces for management and monitoring; Cross-platform support: It supports multiple operating systems such as Linux, Windows, and macOS. It can also run on the ARM architecture, which is suitable for edge computing scenarios; Open Source: MinIO is open source software, released under the GNU Affero General Public License v3.0 (AGPLv3).
[0032] The open source community is active, with a large number of contributors and support resources; Rich plug-ins and tools: The community provides many tools and plug-ins for operations such as backup, synchronization, and migration. It supports monitoring tools such as Prometheus and Grafana to facilitate performance monitoring and alarm settings.
[0033] This solution defines the standards and specifications for large model training data, provides data templates and unified data formats, and provides visual, draggable data pipeline orchestration management to improve data processing efficiency. It improves evaluation indicators, conducts a comprehensive inspection of data, and precipitates the core data assets of the enterprise. It includes front-end and back-end programs and algorithms, encapsulates processing details, and reduces usage complexity.
[0034] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. A system for producing high-quality large-model training data, characterized in that: It includes data set management module, data annotation module, sensitive word library module, data processing module and data analysis module; The data set management module defines data types, stores data in a versioned manner, imports data incrementally, exports data, and publishes data sets; The data annotation module annotates different data set versions; The sensitive word library module defines nine categories of sensitive word libraries, each of which has built-in sensitive words and supports custom word libraries; The data processing module provides a visual and draggable data pipeline function to clean, enhance, filter and arrange the operation sequence of the data, and generate a new data set version after the processing is completed; The data analysis module provides data evaluation indicators, conducts a comprehensive inspection of the data, and takes measures to solve potential problems; The large model is Qwen1.5-14B-Chat; The Qwen1.5-14B-Chat function is optimized for dialogue and chat scenarios. Based on the Transformer architecture, it provides high-quality natural language understanding and generation capabilities and is suitable for a variety of application scenarios.
2. A system for producing high-quality large-model training data according to claim 1, characterized in that: The data types include Prompt+Response, Prompt+Multi-Response, Plain Text, Promp Set, Prompt-chosen-rejected; The versioned storage is implemented through MongoDB database and MinIO objects; The nine categories of sensitive word libraries include but are not limited to illegal words related to guns and explosives, pornographic words, and URLs.
3. A system for producing high-quality large-model training data according to claim 1, characterized in that: The data processing module includes a data cleaning submodule, a data enhancement submodule, and a vocabulary filtering submodule; The data cleaning module has preset cleaning operators, uses the BERT model to generate a language model, and uses the JIEBA model to perform Chinese word segmentation; The cleaning operators include but are not limited to normalizing spaces, converting traditional Chinese to simplified Chinese, and text repetition rate; The data enhancement submodule uses the large model prompt technology to enhance the data to achieve data generalization effect; The word library filtering submodule filters the sensitive words defined by the sensitive word library module.
4. A system for producing high-quality large-model training data according to claim 1, characterized in that: The Qwen1.5-14B-Chat includes a parameter module, an optimization dialogue module, a knowledge module, a language module, and a security module; The parameter module includes 14 billion parameters, which can capture complex language patterns and subtle semantic differences, improving the model's ability to understand and generate long texts. The optimization module is professionally trained to handle multiple rounds of conversations, maintain contextual coherence, understand and respond to user questions or statements, and maintain the fluency and relevance of the conversation; The training data of the knowledge module covers a wide range of subject areas, including general common sense and professional knowledge. Through pre-training and fine-tuning, the large model can display knowledge background on different topics. The language module supports multiple languages, and is mainly optimized for Chinese and English; The security module follows strict data privacy protection principles, implements security and compliance requirements, and takes measures to avoid generating harmful, false or biased content.
5. A system for producing high-quality large-model training data according to claim 3, characterized in that: The JIEBA model includes a dict.txt dictionary and a custom dictionary; The dict.txt dictionary includes more than 20,000 words, listing the number of occurrences and part of speech of each word; The terms include but are not limited to JIEBA word segmentation, which belongs to the probabilistic language model word segmentation. The task is to find a segmentation scheme S from all the results obtained by full segmentation, so that the probability P(S) of the scheme S is maximized; The custom dictionary expands the vocabulary and improves the accuracy of word segmentation; The JIEBA model supports traditional Chinese word segmentation and English word segmentation to meet the needs of different scenarios.
6. A method for producing high-quality large model training data, implemented based on a system for producing high-quality large model training data according to any one of claims 1 to 5, characterized in that: It includes the following sub-steps: S1: Define metadata; S2: import data; S21: Import local data; The local data includes json, xlsx, and txt formats; S22: Import existing data sets on the platform and reuse historical data; S3: Convert data format; S31: Convert the imported plain text data into training format; S32: converting the imported question pair data into a format for storing in a text database; S4: stores data; S5: Process data; S6, Analyze data.
7. The method for producing high-quality large model training data according to claim 6, characterized in that: The S1 comprises the following sub-steps: S11: define data type information; S12: define data import method; S13: define data import template; S14: define sensitive word library type; S15: Define sensitive words in the sensitive word library.
8. The method for producing high-quality large model training data according to claim 6, characterized in that: The S4 comprises the following sub-steps: S41: All data is stored in a shared file system to be shared in the distributed system; S42: The question pair data is stored in a document database; S43: All data storage is stored in a version management manner.
9. The method for producing high-quality large model training data according to claim 6, characterized in that: The S5 comprises the following sub-steps: S51: Perform data cleaning on specific versions of the dataset; S52: Perform data enhancement on specific versions of the dataset; S53: Filter data of a specific version of the data set; S54: Select S51 to S53 for visual arrangement of the specific version of the data set; S55: After the visual arrangement processing is completed, new data set version data is generated.