Data set deduplication method applied to AI language large model
Through efficient preprocessing and feature judgment, combined with deduplication strategies for different data types, the problem of non-repetitive but similar data redundant in AI language large model training datasets is solved, and efficient and accurate data deduplication is achieved, improving the performance and efficiency of the model.
Patent Information
- Application Number
- CN202411608976.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-12
- Publication Date
- 2025-03-04
AI Technical Summary
The prior art is difficult to effectively identify and eliminate non-repetitive but similar data redundant in AI language large model training datasets, resulting in poor deduplication effect.
Through efficient preprocessing, feature judgment and unified keyword mapping, different deduplication strategies are adopted for different data types, including long context encoding-based embedding models and tree graph dynamic programming algorithms for deduplication.
It significantly improves the accuracy and efficiency of data deduplication, ensures high quality and diversity of data sets, reduces redundant data, optimizes resource utilization, and improves the learning efficiency and generalization capabilities of the model.
Smart Images
Figure CN119557560A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing, and more specifically to a data set deduplication method applied to a large AI language model. Background Art
[0002] With the rapid development of artificial intelligence technology, especially the rise of large language models (LLMs), high-quality data has become one of the key factors in improving model performance. Large language models usually require a large amount of training data to learn language patterns, understand semantics, and generate high-quality text output. However, the quality of training data directly affects the final performance of the model, so it is particularly important to build a high-quality training dataset.
[0003] Existing large language model training datasets usually reach TB or even PB levels, and contain a large amount of duplicate data. These duplicate data not only waste valuable storage space, but also increase the complexity of data management and processing. In large-scale datasets, inefficient storage will lead to increased training costs and affect the efficient management of datasets.
[0004] In addition, the AI industry's demand for data types is becoming increasingly diverse, and the amount of training data is also increasing dramatically. Different types of training data sets (such as pre-training data sets, question-answer data sets, preference data sets, etc.) require different deduplication methods. However, existing deduplication technologies are mainly focused on pre-training data sets, and lack specialized deduplication technologies for other data sets such as question-answer data and preference data. This makes it difficult to achieve ideal deduplication effects and efficiency when processing these specific types of data sets.
[0005] In addition, the data deduplication technologies widely used in this field mainly include hash technologies such as Simhash local hashing and Minhash. Although these methods can quickly identify identical texts to a certain extent, they mainly rely on the literal similarity of the text and cannot effectively identify and eliminate redundancy in non-repetitive but similar data. For example, two pieces of text may have slightly different vocabulary choices but are very similar in semantics. Existing deduplication methods often find it difficult to identify this situation, resulting in poor deduplication results.
[0006] Therefore, how to design a data set deduplication method applied to AI language big models, effectively improve the quality of AI language big model training data sets, and then enhance the ability of AI language big models in understanding and generating natural language is a problem that technical personnel in this field urgently need to solve. Summary of the invention
[0007] In view of this, the present invention provides a data set deduplication method applied to the AI language large model, which ensures the high quality and diversity of the data set through efficient preprocessing, feature judgment and unified keyword mapping, and significantly improves the accuracy and efficiency of data deduplication. It provides purer, richer and more structured data support for the training of the AI language large model.
[0008] In order to achieve the above object, the present invention adopts the following technical solution:
[0009] The present invention provides a data set deduplication method applied to an AI language large model, comprising the following steps:
[0010] S1. Obtain text data from a data set used to train an AI language large model, and perform preprocessing to obtain preprocessed text data;
[0011] S2, performing text feature judgment on the preprocessed text data to determine the data type of the text data; the data type includes: a pre-training data set, a question-and-answer data set, and a preference data set;
[0012] S3. Perform keyword unification and standardized field mapping on text data of a determined data type, and perform corresponding data deduplication; the data deduplication includes: pre-training data deduplication, question and answer data deduplication, and preference data deduplication.
[0013] Furthermore, in S1, the preprocessing includes:
[0014] S11, performing intelligent document parsing on the text data to identify structural elements and semantic information in the text data; the structural elements include titles, paragraphs, lists and tables, and the semantic information includes entities, relationships and events;
[0015] S12, performing unified format conversion on the text data after the intelligent document is parsed; the unified format conversion supports converting PDF, Word, Excel, CSV into a unified JSON or XML format;
[0016] S13. Perform data cleaning and standardization on the text data after the uniform format conversion to obtain pre-processed text data.
[0017] Furthermore, in S2, the preprocessed text data is subjected to text feature determination, including:
[0018] S21, extracting key features from the preprocessed text data; the key features include keywords, phrases, text length, structural elements, metadata and semantic information;
[0019] S22. Based on the key features, classify the text data using predefined rules or pre-trained models.
[0020] Furthermore, in S3, keyword unification and standardized field mapping are performed on the text data of the determined data type, including:
[0021] For the pre-training dataset, the keywords are uniformly mapped to text;
[0022] For the question-answering dataset, the keywords are uniformly mapped to roles and contents.
[0023] For the preference dataset, keywords are uniformly mapped to roles, contents, and scores.
[0024] Furthermore, in S3, the pre-training data deduplication includes:
[0025] S311, based on the embedding model capable of long context encoding, combined with the block algorithm, the text data is divided into N blocks fixed-length text blocks, and obtain the text block set B selected ;
[0026] S312: the text block set B selected Sampling is performed; the sampling includes: when N blocks =1, select B selected ={1}; when N blocks ≥2, select B selected ={1, N blocks};
[0027] S313, converting the sampled text block into a text embedding vector with rich semantic information;
[0028] S314, combining with the vector library, performing cosine similarity calculation on the text embedding vector, judging whether the current text data is duplicate data based on the similarity threshold T, and removing the duplicate data.
[0029] Furthermore, in S311, the number of text blocks N blocks It is expressed as:
[0030]
[0031] Among them, T max Indicates the maximum context length that the embedding model can handle, in tokens; R token Indicates the conversion ratio of Chinese characters to tokens; L text Indicates the maximum length of Chinese text that the block algorithm can process, in characters; L input Indicates the total length of the input text data; L window Indicates the length of the sliding window.
[0032] Furthermore, in S314, judging whether the current text data is duplicate data based on the similarity threshold T includes:
[0033] When N blocks =1, there is a text embedding vector V' in the vector library, so that cos(V, V')>T, then the current text data is determined to be duplicate data;
[0034] When N blocks =2, there are text embedding vectors V1′ and V2′ in the vector library, such that cos(V1, V1′)>T, cos(V2, V2′)>T, and the pointIDs of the text embedding vectors V1′ and V2′ are the same, then the current text data is determined to be duplicate data;
[0035] When N blocks >2, there are text embedding vectors V1′ and V2′ in the vector library, such that cos(V1, V1′)>T, cos(V2, V2′)>T, and the pointIDs of text embedding vectors V1′ and V2′ are the same, then the current text data is determined to be duplicate data;
[0036] If there are text embedding vectors V1′ and V2′ in the vector library, such that cos(V1, V1′)>T, cos(V2, V2′)>T, and the pointIDs of text embedding vectors V1′ and V2′ are different, then select B. selected ={2}, and converted into a text embedding vector V3. If there is a text embedding vector V3′ in the vector library, such that cos(V3, V3′)>T, and the pointID of the text embedding vector V3′ is the same as that of the text embedding vector V1′ or the text embedding vector V2′, then the current text data is determined to be duplicate data.
[0037] Furthermore, in S3, deduplication of question and answer data and deduplication of preference data includes:
[0038] S321, construct a tree index based on a linked list structure; the fields in the linked list structure include: role, content, unique identifier lineID, parent node frontID, root node rootID, end flag end and score score;
[0039] S322: Based on the tree index, perform data deduplication through a tree graph dynamic programming algorithm; the data deduplication includes:
[0040] For the first element in the text data, determine whether the first element is duplicate data; if so, do not process it into the database; otherwise, process it into the database;
[0041] For the i-th element in the text data, i≥2;
[0042] If the i-1th element is not a duplicate data, it is put into storage; if the i-1th element is a duplicate data, it is determined whether the i-th element in the text data is a duplicate data; if so, it is not put into storage; otherwise, it is put into storage.
[0043] Furthermore, in the question and answer data deduplication,
[0044] Determining whether the first element is duplicate data includes: if the first element and the element DA in the element library have the same role, the same content, and the end flag end is false, then determining that the first element is duplicate data;
[0045] Determine whether the i-th element in the text data is duplicate data, including: if the i-th element has the same role, content, and end mark end as false in the element DB in the element library, then further determine whether the role, content, and end mark end of the parent node of the element DB are the same as those of the i-1-th element, and if so, determine that the i-th element is duplicate data;
[0046] The root node rootID of the i-th element is the unique identifier lineID of the first element, and the parent node frontID of the i-th element is the unique identifier lineID of the i-1-th element.
[0047] Furthermore, in the deduplication of the preference data,
[0048] Determining whether the first element is duplicate data includes: if the first element and the element DC in the element library have data with the same role and content, then determining that the first element is duplicate data;
[0049] Determine whether the i-th element in the text data is duplicate data, including: if the first element has the same role, content, parent node frontID and score as the element DD in the element library, then determine that the i-th element is duplicate data;
[0050] The root node rootID and the parent node frontID of the i-th element are the unique identifier lineID of the first element.
[0051] It can be seen from the above technical solution that compared with the prior art, the technical solution of the present invention has the following advantages:
[0052] Beneficial effects:
[0053] 1. This method effectively improves the quality of data by preprocessing, feature judgment, keyword unification, and standardized field mapping of text data in the AI language large model training data set. In particular, by adopting different deduplication strategies for different data types, such as pre-training data deduplication, question and answer data deduplication, and preference data deduplication, the diversity and non-redundancy of the data set are ensured, which helps to improve the learning efficiency and generalization ability of the model.
[0054] 2. It supports intelligent parsing and conversion of multiple text data formats, including common formats such as PDF, Word, Excel and CSV, and can automatically identify and convert them into unified JSON or XML formats, greatly enhancing the flexibility and compatibility of data processing. This multi-format support not only simplifies the data preparation process, but also broadens the data source channels, making model training more convenient and efficient.
[0055] 3. In the process of deduplication of pre-trained data, the embedding model based on long context coding combined with the block algorithm and cosine similarity calculation method can accurately identify and remove duplicate text data. Not only the text content itself is considered, but also the text structure and semantic information are combined, which effectively avoids the situation where data is misjudged as duplicate data due to similar content, and ensures the uniqueness and value of the data set.
[0056] 4. For the question-answering data set and preference data set, by constructing a tree index based on a linked list structure and using a tree graph dynamic programming algorithm to deduplicate data, not only the efficiency of data processing is improved, but also the coherence of the question-answering dialogue and the accuracy of user preferences can be better maintained. It can effectively reduce storage space usage while maintaining the clarity and logic of the data structure, providing a solid foundation for subsequent data analysis and model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0058] Figure 1 A flow chart of a method for deduplicating a data set applied to a large AI language model provided by an embodiment of the present invention;
[0059] Figure 2A schematic diagram of performing corresponding data deduplication provided by an embodiment of the present invention;
[0060] Figure 3 A schematic diagram of a preprocessing process provided by an embodiment of the present invention;
[0061] Figure 4 A schematic diagram of a process for determining text features of preprocessed text data provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0063] Embodiment 1:
[0064] like Figure 1 As shown, this embodiment provides a data set deduplication method applied to an AI language large model, comprising the following steps:
[0065] S1. Obtain text data from a data set used to train an AI language large model, and perform preprocessing to obtain preprocessed text data;
[0066] S2, performing text feature judgment on the preprocessed text data to determine the data type of the text data; the data type includes: a pre-training data set, a question-and-answer data set, and a preference data set;
[0067] S3, unify keywords and standardize field mapping for text data of determined data types, and perform corresponding data deduplication; Figure 2 As shown, the data deduplication includes: pre-training data deduplication, question and answer data deduplication and preference data deduplication.
[0068] This method achieves efficient deduplication of pre-trained datasets, question-answering datasets, and preference datasets by preprocessing text data, determining data types, unifying keywords, and standardizing field mapping. This method significantly improves the quality and consistency of datasets, reduces redundant data, and thus improves the training efficiency and performance of large AI language models, optimizes resource utilization, and reduces training costs.
[0069] The above steps and related features are further described in detail below:
[0070] In this embodiment S1, text data in a data set used to train an AI language large model is obtained and preprocessed to obtain preprocessed text data;
[0071] like Figure 3 As shown, the preprocessing includes:
[0072] S11, performing intelligent document parsing on the text data to identify structural elements and semantic information in the text data; the structural elements include titles, paragraphs, lists and tables, and the semantic information includes entities, relationships and events;
[0073] S12, performing unified format conversion on the text data after the intelligent document is parsed; the unified format conversion supports converting PDF, Word, Excel, CSV into a unified JSON or XML format;
[0074] S13. Perform data cleaning and standardization on the text data after the uniform format conversion to obtain pre-processed text data.
[0075] In this step, through intelligent document parsing, structural elements (such as titles, paragraphs, lists, and tables) and semantic information (such as entities, relationships, and events) in text data are effectively identified, enhancing the parsability and usability of the data. Subsequently, the unified format conversion converts text data from different sources (such as PDF, Word, Excel, CSV) into a unified JSON or XML format for subsequent processing and analysis. Finally, data cleaning and standardization further eliminate redundant and erroneous information, ensuring the quality and consistency of the pre-processed text data.
[0076] In this embodiment S2, the text feature judgment is performed on the preprocessed text data to determine the data type of the text data; the data type includes: a pre-training data set, a question-and-answer data set, and a preference data set;
[0077] like Figure 4 As shown, the preprocessed text data is subjected to text feature judgment, including:
[0078] S21, extracting key features from the preprocessed text data; the key features include keywords, phrases, text length, structural elements, metadata and semantic information;
[0079] Specifically, extract keywords that reflect the theme and core content of the text; identify key phrases that contain important concepts and expressions; count the length of the text, including the number of characters, words or sentences, to assess the size of the text; identify structured elements in the text, such as titles, paragraphs, lists and tables, to understand the organizational structure of the text; extract metadata information related to the text, such as author, date, source, etc., to assist further classification and processing; analyze the semantic content of the text, including entities, relationships and events, to understand the deep meaning of the text.
[0080] S22. Based on the key features, classify the text data using predefined rules or pre-trained models.
[0081] Predefined rules can be formulated based on specific business needs and data characteristics, for example, by combining keywords and phrases to determine whether a text belongs to a certain category. Alternatively, a trained model (such as a classifier) can be used to classify text data. These models can be trained through machine learning algorithms (such as neural networks) and can automatically identify the category of text based on the extracted key features.
[0082] Ultimately, it is possible to accurately determine whether the text data belongs to a pre-training dataset, a question-and-answer dataset, or a preference dataset. This feature extraction and classification method effectively improves the pertinence and accuracy of data deduplication, ensuring the efficiency and effectiveness of data deduplication.
[0083] In this embodiment S3, keywords of the text data of the determined data type are unified and standardized in field mapping, and corresponding data deduplication is performed; the data deduplication includes: pre-training data deduplication, question and answer data deduplication, and preference data deduplication.
[0084] In this step, keywords are unified and standardized field mapping is performed on text data of a determined data type, including:
[0085] For the pre-training dataset, the keywords are uniformly mapped to text;
[0086] For the question-answering dataset, the keywords are uniformly mapped to roles and contents.
[0087] For the preference dataset, keywords are uniformly mapped to roles, contents, and scores.
[0088] For pre-training data sets, keywords are uniformly mapped to text for easy subsequent unified processing; for question-answering data sets, they are mapped to roles and content to clearly distinguish questions from answers; for preference data sets, they are mapped to roles, content, and scores to fully cover user preference information. This simplifies the complexity of data processing and enhances the consistency and comparability of data, providing a solid foundation for subsequent data deduplication and analysis.
[0089] Furthermore, the pre-training data deduplication includes:
[0090] S311, based on the embedding model capable of long context encoding, combined with the block algorithm, the text data is divided into N blocks fixed-length text blocks, and obtain the text block set B selected ;
[0091] Number of text blocks N blocks It is expressed as:
[0092]
[0093] Among them, T max Indicates the maximum context length that the embedding model can handle, in tokens; R token Indicates the conversion ratio of Chinese characters to tokens; L text Indicates the maximum length of Chinese text that the block algorithm can process, in characters; L input Indicates the total length of the input text data; L window Indicates the length of the sliding window.
[0094] S312: the text block set B selected Sampling is performed; the sampling includes: when N blocks =1, select B selected ={1}; when N blocks ≥2, select B selected ={1, N blocks};
[0095] S313, converting the sampled text block into a text embedding vector with rich semantic information;
[0096] S314, combining with the vector library, performing cosine similarity calculation on the text embedding vector, judging whether the current text data is duplicate data based on the similarity threshold T, and removing the duplicate data.
[0097] Further, judging whether the current text data is duplicate data based on the similarity threshold T includes:
[0098] When N blocks =1, there is a text embedding vector V' in the vector library, so that cos(V, V')>T, then the current text data is determined to be duplicate data;
[0099] When N blocks =2, there are text embedding vectors V1′ and V2′ in the vector library, such that cos(V1, V1′)>T, cos(V2, V2′)>T, and the pointIDs of the text embedding vectors V1′ and V2′ are the same, then the current text data is determined to be duplicate data;
[0100] When N blocks>2, there are text embedding vectors V1′ and V2′ in the vector library, such that cos(V1, V1′)>T, cos(V2, V2′)>T, and the pointIDs of the text embedding vectors V1′ and V2′ are the same, then the current text data is determined to be duplicate data;
[0101] If there are text embedding vectors V1′ and V2′ in the vector library, such that cos(V1, V1′)>T, cos(V2, V2′)>T, and the pointIDs of text embedding vectors V1′ and V2′ are different, then select B. selected ={2}, and converted into a text embedding vector V3. If there is a text embedding vector V3′ in the vector library, such that cos(V3, V3′)>T, and the pointID of the text embedding vector V3′ is the same as that of the text embedding vector V1′ or the text embedding vector V2′, then the current text data is determined to be duplicate data.
[0102] The pre-trained data deduplication combines the embedding model based on long context coding with the block algorithm to divide the text data into fixed-length text blocks, and effectively identify and remove duplicate data through sampling, text embedding vectorization and cosine similarity calculation. It can not only accurately determine the repetitiveness of text data, but also greatly reduce the amount of calculation while maintaining high accuracy, improve processing efficiency, and thus significantly improve the quality and consistency of the data set, providing high-quality input data for the training of large AI language models.
[0103] Furthermore, the deduplication of question and answer data and preference data includes:
[0104] S321, construct a tree index based on a linked list structure; the fields in the linked list structure include: role, content, unique identifier lineID, parent node frontID, root node rootID, end flag end and score score;
[0105] S322: Based on the tree index, perform data deduplication through a tree graph dynamic programming algorithm; the data deduplication includes:
[0106] For the first element in the text data, determine whether the first element is duplicate data; if so, do not process it into the database; otherwise, process it into the database;
[0107] For the i-th element in the text data, i≥2;
[0108] If the i-1th element is not a duplicate data, it is put into storage; if the i-1th element is a duplicate data, it is determined whether the i-th element in the text data is a duplicate data; if so, it is not put into storage; otherwise, it is put into storage.
[0109] Furthermore, in the question and answer data deduplication,
[0110] Determining whether the first element is duplicate data includes: if the first element and the element DA in the element library have the same role, the same content, and the end flag end is false, then determining that the first element is duplicate data;
[0111] Determine whether the i-th element in the text data is duplicate data, including: if the i-th element has the same role, content, and end mark end as false in the element DB in the element library, then further determine whether the role, content, and end mark end of the parent node of the element DB are the same as those of the i-1-th element, and if so, determine that the i-th element is duplicate data;
[0112] The root node rootID of the i-th element is the unique identifier lineID of the first element, and the parent node frontID of the i-th element is the unique identifier lineID of the i-1-th element.
[0113] Furthermore, in the deduplication of the preference data,
[0114] Determining whether the first element is duplicate data includes: if the first element and the element DC in the element library have data with the same role and content, then determining that the first element is duplicate data;
[0115] Determine whether the i-th element in the text data is duplicate data, including: if the first element has the same role, content, parent node frontID and score as the element DD in the element library, then determine that the i-th element is duplicate data;
[0116] The root node rootID and the parent node frontID of the i-th element are the unique identifier lineID of the first element.
[0117] The deduplication of question and answer data and preference data here realizes efficient deduplication processing of each element in the text data by constructing a tree index based on a linked list structure and a tree graph dynamic programming algorithm. It not only improves the efficiency of deduplication, but also ensures the integrity and consistency of the data. For question and answer data, by checking the role, content and end mark of the element, the storage of duplicate data is effectively avoided; for preference data, factors such as role, content, parent node and score are further considered to ensure the accuracy of deduplication. It not only reduces the waste of storage space, but also improves the speed and quality of data processing, providing a higher quality and more diverse training data set for the AI language large model.
[0118] This embodiment provides a method for deduplicating a data set applied to a large AI language model, which achieves efficient deduplication of pre-trained data sets, question-answer data sets, and preference data sets through steps such as preprocessing text data, determining data types, keyword unification, and standardized field mapping. Specifically, it includes: intelligent document parsing, unified format conversion, and data cleaning to ensure data quality and consistency; accurate determination of data types through key feature extraction and classification; keyword unification and standardized field mapping for different data types, and deduplication processing based on embedded models and tree graph dynamic programming algorithms.
[0119] It significantly improves the accuracy and efficiency of data deduplication, reduces redundant data, optimizes resource utilization, and improves the training efficiency and performance of AI language large models. It significantly improves the accuracy and generalization ability of AI language large models, and makes the generation and reasoning capabilities more reliable, thus providing a better user experience in practical applications.
[0120] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0121] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data set deduplication method applied to a large AI language model, characterized in that: The following steps are involved: S1. Obtain text data from a data set used to train an AI language large model, and perform preprocessing to obtain preprocessed text data; S2, performing text feature judgment on the preprocessed text data to determine the data type of the text data; The data types include: pre-training dataset, question-answering dataset and preference dataset; S3. Perform keyword unification and standardized field mapping on text data of a determined data type, and perform corresponding data deduplication; the data deduplication includes: pre-training data deduplication, question and answer data deduplication, and preference data deduplication.
2. According to the method for deduplication of a data set applied to an AI language large model according to claim 1, it is characterized in that: In S1, the preprocessing includes: S11, performing intelligent document parsing on the text data to identify structural elements and semantic information in the text data; the structural elements include titles, paragraphs, lists and tables, and the semantic information includes entities, relationships and events; S12, performing unified format conversion on the text data after the intelligent document is parsed; the unified format conversion supports converting PDF, Word, Excel, CSV into a unified JSON or XML format; S13. Perform data cleaning and standardization on the text data after the uniform format conversion to obtain pre-processed text data.
3. According to the method for deduplication of a data set applied to an AI language large model according to claim 1, it is characterized in that: In S2, the text feature judgment is performed on the preprocessed text data, including: S21, extracting key features from the preprocessed text data; the key features include keywords, phrases, text length, structural elements, metadata and semantic information; S22. Based on the key features, classify the text data using predefined rules or pre-trained models.
4. According to the method for deduplication of a data set applied to an AI language large model according to claim 1, it is characterized in that: In S3, keyword unification and standardized field mapping are performed on the text data of the determined data type, including: For the pre-training dataset, the keywords are uniformly mapped to text; For the question-answering dataset, the keywords are uniformly mapped to roles and contents. For the preference dataset, keywords are uniformly mapped to roles, contents, and scores.
5. According to the method for deduplication of a data set applied to a large AI language model according to claim 1, it is characterized in that: In S3, deduplication of pre-training data includes: S311, based on the embedding model capable of long context encoding, combined with the block algorithm, the text data is divided into N blocks fixed-length text blocks, and obtain the text block set B selected ; S312: the text block set B selected Sampling is performed; the sampling includes: when N blocks =1, select B selected ={1}; when N blocks ≥2, select B selected ={1, N blocks }; S313, converting the sampled text block into a text embedding vector with rich semantic information; S314, combining with the vector library, performing cosine similarity calculation on the text embedding vector, judging whether the current text data is duplicate data based on the similarity threshold T, and removing the duplicate data.
6. The method for deduplicating a data set applied to a large AI language model according to claim 5, characterized in that: S311, the number of text blocks N blocks It is expressed as: Among them, T max Indicates the maximum context length that the embedding model can handle, in tokens; R token Indicates the conversion ratio of Chinese characters to tokens; L text Indicates the maximum length of Chinese text that the block algorithm can process, in characters; L input Indicates the total length of the input text data; L window Indicates the length of the sliding window.
7. The method for deduplicating a data set applied to a large AI language model according to claim 5, characterized in that: In S314, judging whether the current text data is duplicate data based on the similarity threshold T includes: When N blocks =1, there is a text embedding vector V' in the vector library, so that cos(V, V')>T, then the current text data is determined to be duplicate data; When N blocks =2, there are text embedding vectors V1' and V2' in the vector library, such that cos(V1, V1')>T, cos(V2, V2')>T, and the pointIDs of the text embedding vectors V1' and V2' are the same, then the current text data is determined to be duplicate data; When N blocks >2, there are text embedding vectors V1′ and V2′ in the vector library, such that cos(V1, V1′)>T, cos(V2, V2′)>T, and the pointIDs of the text embedding vectors V1′ and V2′ are the same, then the current text data is determined to be duplicate data; If there are text embedding vectors V1′ and V2′ in the vector library, such that cos(V1, V1′)>T, cos(V2, V2′)>T, and the pointIDs of text embedding vectors V1′ and V2′ are different, then select B. selected ={2}, and converted into a text embedding vector V3. If there is a text embedding vector V3′ in the vector library, such that cos(V3, V3′)>T, and the pointID of the text embedding vector V3′ is the same as that of the text embedding vector V1′ or the text embedding vector V2′, then the current text data is determined to be duplicate data.
8. The method for deduplicating a data set applied to a large AI language model according to claim 1, characterized in that: In S3, deduplication of question and answer data and deduplication of preference data includes: S321, construct a tree index based on a linked list structure; the fields in the linked list structure include: role, content, unique identifier lineID, parent node frontID, root node rootID, end flag end and score score; S322: Based on the tree index, perform data deduplication through a tree graph dynamic programming algorithm; the data deduplication includes: For the first element in the text data, determine whether the first element is duplicate data; if so, do not process it into the database; otherwise, process it into the database; For the i-th element in the text data, i≥2; If the i-1th element is not a duplicate data, it is put into storage; if the i-1th element is a duplicate data, it is determined whether the i-th element in the text data is a duplicate data; if so, it is not put into storage; otherwise, it is put into storage.
9. The method for deduplicating a data set applied to a large AI language model according to claim 8, characterized in that: In the question and answer data deduplication, Determining whether the first element is duplicate data includes: if the first element and the element DA in the element library have the same role, the same content, and the end flag end is false, then determining that the first element is duplicate data; Determine whether the i-th element in the text data is duplicate data, including: if the i-th element has the same role, content, and end mark end as false in the element DB in the element library, then further determine whether the role, content, and end mark end of the parent node of the element DB are the same as those of the i-1-th element, and if so, determine that the i-th element is duplicate data; The root node rootID of the i-th element is the unique identifier lineID of the first element, and the parent node frontID of the i-th element is the unique identifier lineID of the i-1-th element.
10. The method for deduplicating a data set applied to a large AI language model according to claim 8, characterized in that: In the preference data deduplication, Determining whether the first element is duplicate data includes: if the first element and the element DC in the element library have data with the same role and content, then determining that the first element is duplicate data; Determine whether the i-th element in the text data is duplicate data, including: if the first element has the same role, content, parent node frontID and score as the element DD in the element library, then determine that the i-th element is duplicate data; The root node rootID and the parent node frontID of the i-th element are the unique identifier lineID of the first element.