Large model training data automatic screening method

Abstract

The invention relates to a large model training data automatic screening method, which comprises the steps of collecting a large-scale text data set, and performing likelihood estimation on each text paragraph in the large-scale text data set through a trained Transform model to obtain a generation probability of each text paragraph; based on the generation probability of each text paragraph, calculating to obtain a semantic mutual information entropy value between the text paragraphs; constructing a mutual information entropy graph by taking each text paragraph as a node and taking a semantic mutual information entropy value between the text paragraphs as a weight of an edge; based on the mutual information entropy graph, node clustering is carried out by adopting a union lookup algorithm to obtain a plurality of clusters; in each cluster, a node is selected and reserved in a random sampling mode, and the remaining nodes are deleted; and integrating the text paragraphs corresponding to all the reserved nodes to obtain a simplified text data set. According to the method, negative effects of repeated data on model training are avoided, and full-automatic and intelligent large model training data automatic screening is realized.

Description

Technical Field

[0001] The present application relates to the technical field of machine learning, and in particular to a method for automatically screening large model training data. Background Art

[0002] With the rapid development of natural language processing technology, pre-training of large models has gradually become an important means to promote the advancement of artificial intelligence. These models rely on massive amounts of text data. As the scale of data increases dramatically, the training efficiency and quality of the models face severe challenges. In order to ensure that large models have a broad understanding of language, pre-training usually uses self-supervised learning methods to learn language structure by predicting missing words or sentences; however, the huge amount of data contains a large amount of redundant and low-quality data. How to optimize the selection and screening of training data has become a key issue in improving model performance.

[0003] At present, during the training of large models, the collected text data often contains a large amount of repeated, similar or irrelevant content. These data redundancy and duplication phenomena are particularly serious in large-scale text data sets, especially in Internet data resources, where similar content generated by different platforms or users frequently appears, which weakens the diversity and information density of the training set, and ultimately affects the overall training effect and performance of the model. Therefore, the existing technology lacks an efficient mechanism to automatically screen and filter these redundant data. Summary of the invention

[0004] Based on this, it is necessary to provide a method for automatically screening large model training data, which includes: S1: Collect a large-scale text dataset, use the trained Transformer model to perform likelihood estimation on each text paragraph in the large-scale text dataset, and obtain the generation probability of each text paragraph; calculate the semantic mutual information entropy value between text paragraphs based on the generation probability of each text paragraph; S2: Take each text paragraph as a node and the semantic mutual information entropy value between text paragraphs as the edge weight to construct a mutual information entropy graph; based on the mutual information entropy graph, use the union-find algorithm to cluster the nodes and obtain multiple clusters; S3: In each cluster, a node is selected and retained by random sampling, and the remaining nodes are deleted; the text paragraphs corresponding to all retained nodes are integrated to obtain a streamlined text dataset.

[0005] Preferably, after the large-scale text data set is collected, preliminary data cleaning is performed on the text data in the large-scale text data set through an automated script.

[0006] Preferably, the training process of the Transformer model includes: Obtain a pre-trained word sequence, input the word sequence into a Transformer model, and output a probability distribution of each word in the word sequence; Calculate the cross entropy loss function based on the probability distribution of each word and its corresponding true label; Select the set Adam optimizer, minimize the cross entropy loss function, and update the parameters of the Transformer model.

[0007] Preferably, the likelihood estimation formula for the generation probability of a text paragraph is: ; ; in, Indicates i The generation probability of a text paragraph; Indicates i paragraphs of text; m Indicates i The number of words in a text paragraph; Represents a paragraph of text The t The probability distribution of words; Represents a paragraph of text The tth word in ; Represents a paragraph of text The first word in ; Represents a paragraph of text The t -1 word; represents the softmax activation function; Represents a trained Transformer model.

[0008] Preferably, the calculation formula of the semantic mutual information entropy value between text paragraphs is: ; in, Indicates i paragraphs and j Semantic mutual information entropy between text paragraphs; Indicates i paragraphs of text; Indicates j paragraphs of text; Represents a paragraph of text and text paragraphs The joint probability of the trained Transformer model is calculated for the text paragraph With text paragraph The overall generation probability after concatenation is obtained; Indicates i The generation probability of a text paragraph; Indicates j The generation probability of a text paragraph.

[0009] Preferably, the semantic mutual information entropy value between text paragraphs is negatively correlated with the correlation between text paragraphs.

[0010] Preferably, the clustering process includes: Step 1: randomly select a node in the mutual information entropy graph, query the nodes that meet the clustering target with the randomly selected node, and merge the randomly selected node and the nodes that meet the clustering target with it into a cluster; the clustering target is constructed based on the semantic mutual information entropy value between text paragraphs; Step 2: Repeat step 1 until all nodes are traversed and multiple clusters are obtained.

[0011] Preferably, the clustering target is expressed as: ; ; in, It means any; Represents a paragraph of text Corresponding nodes and text paragraphs The weight edges between corresponding nodes; E Represents the set of weighted edges in the mutual information entropy graph; Represents weight edge The weight of Indicates a merge operation; Indicates i paragraphs and j Semantic mutual information entropy between text paragraphs; Indicates i paragraphs of text; Indicates j paragraphs of text; Indicates the preset threshold.

[0012] Preferably, random sampling is expressed as: ; in, Indicates k A randomly selected node in a cluster; represents a random sampling operation; Indicates k Clusters.

[0013] Preferably, the expression of the cross entropy loss function is: ; ; Where L represents the cross entropy loss function; n Represents the number of words in a word sequence; Indicates the first word in the sequence t The true label of each word; Indicates the first word in the sequence t The probability distribution of words; Indicates the first word in the sequence t words; Indicates the first word in a word sequence; Indicates the first word in the sequence t -1 word; represents the softmax activation function; Represents the Transformer model.

[0014] Beneficial effects: This method uses deep learning technology to calculate the semantic mutual information entropy of a large text model based on training, and automatically screens data based on mutual information entropy graph clustering, thereby optimizing data diversity and information value, avoiding the negative impact of duplicate data on model training, improving model training efficiency and quality, and realizing fully automatic and intelligent automatic screening of large model training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 This is a flowchart of the automatic screening method of large model training data according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0018] like Figure 1 As shown, this embodiment provides a method for automatically screening large model training data, the method comprising: S1: Collecting a large-scale text data set; in this embodiment, a large-scale text data set is collected from the Internet, including various types of text resources such as news articles, blogs, and social media posts, so as to ensure the diversity and extensiveness of the text data; after collecting the large-scale text data set, the text data in the large-scale text data set is preliminarily cleaned by an automated script to remove format errors, non-target languages, or content that damages data quality, so as to ensure the quality of the training data; Use the trained Transformer model to estimate the likelihood of each text paragraph in the large-scale text dataset and obtain the generation probability of each text paragraph; In this embodiment, a self-supervised learning method is used to pre-train a Transformer model with a parameter scale of more than 1 billion. In the pre-training result, a Language Modeling task is used, and the Transformer model learns the rules of language by predicting missing words in sentences.

[0019] The training process of the Transformer model includes: Get pre-trained word sequences , the word sequence is input into the Transformer model, and the Transformer model outputs the probability distribution of each word in the word sequence, with the word For example, predicting words The probability distribution of , the probability distribution calculation formula is: ; in, n Represents the number of words in a word sequence; Indicates the first word in the sequence t The probability distribution of words; Indicates the first word in the sequence t words; Indicates the first word in a word sequence; Indicates the first word in the sequence t -1 word; represents the softmax activation function; Represents a Transformer model that processes an input sequence of words and outputs a probability distribution over the next word.

[0020] Based on the probability distribution of each word and its corresponding true label, the cross entropy loss function is calculated; the expression of the cross entropy loss function is: ; Where L represents the cross entropy loss function; Indicates the first word in the sequence t The true label of each word.

[0021] The set Adam optimizer is selected and the cross entropy loss function is minimized. The Transformer model is continuously optimized in the iterative process, and the parameters of the Transformer model are updated to more accurately predict the words in the text paragraph.

[0022] In this embodiment, the parameter settings of the Adam optimizer include: the first parameter β 1=0.9, the second parameter β 2=0.999, learning rate η =1e-4, and gradually reduce the learning rate according to the effect during training.

[0023] Collect various types of texts and perform data cleaning to ensure data quality; use self-supervised learning for pre-training, predict missing words in sentences through language modeling tasks, use cross-entropy loss function to optimize the model, and use Adam optimizer to adjust parameters and learning rate. After large-scale pre-training, the model can understand complex language structures, laying the foundation for subsequent data screening.

[0024] Furthermore, after completing the pre-training of the large model, the trained Transformer model is used to perform likelihood estimation on the text paragraphs in the large-scale text dataset and calculate the semantic mutual information entropy between these texts. The core of this step is to calculate the generation probability of each text paragraph separately through the model, and then analyze the degree of information sharing between texts. The likelihood estimation formula for the generation probability of a text paragraph is: ; ; in, Indicates i The generation probability of a text paragraph; Indicates i paragraphs of text; m Indicates i The number of words in a text paragraph; Represents a paragraph of text The t The probability distribution of words; Represents a paragraph of text The tth word in ; Represents a paragraph of text The first word in ; Represents a paragraph of text The t -1 word; represents the softmax activation function; represents a trained Transformer model; the product in the generated probability expresses the probability of the entire text paragraph appearing in the eyes of the model, reflecting the depth of the model's understanding of the text paragraph.

[0025] Furthermore, in order to calculate the text paragraph and text paragraphs The semantic mutual information entropy between them needs to estimate their joint probability and marginal probability; the joint probability is obtained by considering the probability of both being generated simultaneously in a specific context, and it is necessary to calculate the trained Transformer model for the text paragraph. With text paragraph The overall generation probability after concatenation is obtained; The semantic mutual information entropy value between text paragraphs is calculated based on the generation probability of each text paragraph. The calculation formula of the semantic mutual information entropy value between text paragraphs is: ; in, Indicates i paragraphs and j Semantic mutual information entropy between text paragraphs; Indicates i paragraphs of text; Indicates j paragraphs of text; Represents a paragraph of text and text paragraphs The joint probability of Indicates i The generation probability of a text paragraph; Indicates j The probability of generating a text paragraph; this formula quantifies the text paragraph and text paragraphs The amount of shared information is used to describe the semantic relationship between them. A low value of the semantic mutual information entropy indicates that the two text paragraphs have a strong semantic correlation or redundancy, while a high value indicates that the two are relatively independent.

[0026] The generation probability of each text paragraph is calculated through the model to reflect the depth of its semantic understanding; the semantic mutual information entropy between texts is estimated through the joint probability and the marginal probability. In this embodiment, the semantic mutual information entropy value between text paragraphs is negatively correlated with the correlation between the text paragraphs, that is, a low semantic mutual information entropy value indicates a strong semantic correlation between the two text paragraphs, and a high semantic mutual information entropy value indicates that the two text paragraphs are relatively independent; this process reveals the semantic association and redundant information in the data, and provides a quantitative basis for subsequent data screening and compression.

[0027] S2: Take each text paragraph as a node and the semantic mutual information entropy value between text paragraphs as the edge weight to construct a mutual information entropy graph; a higher semantic mutual information entropy value indicates a weaker semantic connection, and a lower semantic mutual information entropy value indicates a closer semantic connection. The mutual information entropy graph is used to show the degree of semantic association between text paragraphs; In this embodiment, the mutual information entropy graph is defined as: G =( V , E ), where V represents a node set, including nodes corresponding to each text paragraph; E represents a set of weighted edges, including each weighted edge; Corresponding nodes and text paragraphs The weight of the edge between the corresponding nodes is determined by the semantic mutual information entropy between the two, and the formula is: ; in, Represents weight edge The weight of Indicates i paragraphs and j The semantic mutual information entropy between two text paragraphs. A high value of the semantic mutual information entropy actually indicates that the semantic connection between the two text paragraphs is weak, indicating that they are relatively independent; while a low value indicates that the two text paragraphs have a strong semantic correlation or redundancy; therefore, in the mutual information entropy graph, a lower weight edge indicates a closer semantic connection between nodes.

[0028] Based on the mutual information entropy graph, the last step is to use a graph-based algorithm to cluster the text paragraphs and perform data screening. The purpose of this step is to optimize the dataset by identifying closely connected groups of nodes, ensuring that the retained text paragraphs are representative and diverse, while removing redundant information.

[0029] First, based on the mutual information entropy graph, the union-find algorithm is used to cluster nodes and obtain multiple clusters. Union-find is a tree-like data structure used to handle some disjoint merging and querying problems. Union-find can effectively merge nodes that are closely connected to each other (i.e., low semantic mutual information entropy values) in the graph into one cluster; at the beginning of the clustering process, each node in the graph represents a separate set. As the algorithm proceeds, nodes that are closely connected through weighted edges are gradually merged.

[0030] Specifically, the clustering process includes: Step 1: randomly select a node in the mutual information entropy graph, query the nodes that meet the clustering target with the randomly selected node, and merge the randomly selected node and the nodes that meet the clustering target with it into a cluster; the clustering target is constructed based on the semantic mutual information entropy value between text paragraphs; In this embodiment, the clustering target is expressed as: ; ; in, It means any; Represents a paragraph of text Corresponding nodes and text paragraphs The weight edges between corresponding nodes; E Represents the set of weighted edges in the mutual information entropy graph; Indicates a merge operation; Indicates i paragraphs of text; Indicates j paragraphs of text; Indicates the preset threshold; in the process of clustering, the setting of the preset threshold has a decisive influence on the granularity of clustering. Values ​​will result in finer clustering results, while higher Values ​​lower than 0 may result in overly broad clustering; Step 2: Repeat step 1 until all nodes are traversed and multiple clusters are obtained.

[0031] S3: After clustering, the nodes in each cluster represent text paragraphs with high semantic relevance to each other. In order to streamline the data set, in each cluster, one node is selected and retained by random sampling, and the remaining nodes are deleted; the text paragraphs corresponding to all retained nodes are integrated to obtain a streamlined text data set.

[0032] In this embodiment, random sampling is expressed as: ; in, Indicates k A randomly selected node in a cluster; represents a random sampling operation; Indicates k Clusters.

[0033] After clustering, a representative text is randomly selected from each cluster and retained, and the rest are considered redundant and deleted. This screening process produces a more streamlined and high-quality training dataset. This dataset not only reduces the amount of original data, but also improves the overall diversity and information value by removing duplicate or overly similar content, thereby improving the training efficiency and performance of the model.

[0034] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0035] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A method for automatically screening large model training data, characterized in that: include: S1: Collect a large-scale text dataset, use the trained Transformer model to perform likelihood estimation on each text paragraph in the large-scale text dataset, and obtain the generation probability of each text paragraph; calculate the semantic mutual information entropy value between text paragraphs based on the generation probability of each text paragraph; S2: Take each text paragraph as a node and the semantic mutual information entropy value between text paragraphs as the weight of the edge to construct a mutual information entropy graph; based on the mutual information entropy graph, use the union-find algorithm to cluster the nodes and obtain multiple clusters; S3: In each cluster, a node is selected and retained by random sampling, and the remaining nodes are deleted; the text paragraphs corresponding to all retained nodes are integrated to obtain a streamlined text dataset.

2. The automatic screening method for large model training data according to claim 1 is characterized in that: After the large-scale text data set is collected, preliminary data cleaning is performed on the text data in the large-scale text data set through an automated script.

3. The automatic screening method for large model training data according to claim 1 is characterized in that: The training process of the Transformer model includes: Obtain a pre-trained word sequence, input the word sequence into a Transformer model, and output a probability distribution of each word in the word sequence; Calculate the cross entropy loss function based on the probability distribution of each word and its corresponding true label; Select the set Adam optimizer, minimize the cross entropy loss function, and update the parameters of the Transformer model.

4. The automatic screening method for large model training data according to claim 1 is characterized in that: The likelihood estimation formula for the generation probability of a text paragraph is: ； ； in, Indicates i The generation probability of a text paragraph; Indicates i paragraphs of text; m Indicates i The number of words in a text paragraph; Represents a paragraph of text The t The probability distribution of words; Represents a paragraph of text The tth word in ; Represents a paragraph of text The first word in ; Represents a paragraph of text The t -1 word; represents the softmax activation function; Represents a trained Transformer model.

5. The method for automatically screening large model training data according to claim 1, characterized in that: The calculation formula of the semantic mutual information entropy value between text paragraphs is: ； in, Indicates i paragraphs and j Semantic mutual information entropy between text paragraphs; Indicates i paragraphs of text; Indicates j paragraphs of text; Represents a paragraph of text and text paragraphs The joint probability of the trained Transformer model is calculated for the text paragraph With text paragraph The overall generation probability after concatenation is obtained; Indicates i The generation probability of a text paragraph; Indicates j The generation probability of a text paragraph.

6. The method for automatically screening large model training data according to claim 5, characterized in that: The semantic mutual information entropy value between text paragraphs is negatively correlated with the correlation between text paragraphs.

7. The method for automatically screening large model training data according to claim 1, characterized in that: The clustering process includes: Step 1: randomly select a node in the mutual information entropy graph, query the nodes that meet the clustering target with the randomly selected node, and merge the randomly selected node and the nodes that meet the clustering target with it into a cluster; the clustering target is constructed based on the semantic mutual information entropy value between text paragraphs; Step 2: Repeat step 1 until all nodes are traversed and multiple clusters are obtained.

8. The method for automatically screening large model training data according to claim 7, characterized in that: The clustering target is expressed as: ； ； in, It means any; Represents a paragraph of text Corresponding nodes and text paragraphs The weight edges between corresponding nodes; E Represents the set of weighted edges in the mutual information entropy graph; Represents weight edge The weight of Indicates a merge operation; Indicates i paragraphs and j Semantic mutual information entropy between text paragraphs; Indicates i paragraphs of text; Indicates j paragraphs of text; Indicates the preset threshold.

9. The method for automatically screening large model training data according to claim 1, characterized in that: Random sampling is expressed as: ； in, Indicates k A randomly selected node in a cluster; represents a random sampling operation; Indicates k Clusters.

10. The automatic screening method for large model training data according to claim 3 is characterized in that: The expression of the cross entropy loss function is: ； ； Where L represents the cross entropy loss function; n Represents the number of words in a word sequence; Indicates the first word in the sequence t The true label of each word; Indicates the first word in the sequence t The probability distribution of words; Indicates the first word in the sequence t words; Indicates the first word in a word sequence; Indicates the first word in the sequence t -1 word; represents the softmax activation function; Represents the Transformer model.

Images