A method for constructing large language datasets based on contrastive learning
Through a contrastive learning-based method, we identify and process strong and weak features in large language datasets to form filtered training sets and validation sets, which solves the problems of model prediction deviation and insufficient entity relationship decoding in the existing technology, and achieves accurate prediction of large language models and synchronous decoding of entity relationships.
Patent Information
- Application Number
- CN202410773687.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-14
- Publication Date
- 2025-09-09
- Estimated Expiration
- 2044-06-14
AI Technical Summary
In the process of constructing existing large language datasets, the filtering steps mostly involve filtering out repeated semantics, which leads to deviations in model predictions and inability to achieve accurate extraction results. At the same time, the entities and relationships in the text are not fully decoded, and the connection between the subject and object of the entity is weak.
A contrastive learning-based method is adopted. The encoder recognizes the text and the extraction module is used to extract triple information, identify strong and weak features in the semantics, independently annotate through the annotation module, correct the semantic data through the correction module, and perform differential comparison using the comparison module to form a filtered training set and a validation set.
It achieves the accuracy of large language model predictions, improves the accuracy of extraction results, enhances the synchronous decoding of entities and relationships in text, and increases the connectivity between entity subjects and objects.
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computers, and in particular relates to a method for constructing a large language data set based on contrastive learning. Background Art
[0002] "Big language datasets" generally refer to datasets containing large amounts of text data, which are used to train and evaluate natural language processing (NLP) models. They can not only generate natural language text, but also deeply understand the meaning of the text and handle various natural language tasks such as text summarization, question answering, translation, etc. These datasets are often used for various NLP tasks such as text classification, named entity recognition, sentiment analysis, machine translation, etc.
[0003] Large language models and their applications in artificial intelligence have become a hot topic in global scientific research. Their growth in scale is particularly remarkable, with the number of parameters increasing from over a billion initially to over a trillion today. This increase in parameters enables the models to more precisely capture the subtleties of human language and gain a deeper understanding of its complexity. Over the past year, large language models have seen significant improvements in incorporating new knowledge, decomposing complex tasks, and aligning images and text. As the technology matures, its application will continue to expand, providing more intelligent and personalized services and further improving people's lives and production.
[0004] Building a large language dataset is a complex and time-consuming process, which usually involves the following steps: first, you need to determine the type and domain of the dataset to be built; once the data type and domain are determined, you need to start collecting data; the collected raw data often contains noise, errors and inconsistencies, so before building the dataset, you need to preprocess it, including removing duplicate data, standardizing text formats, processing missing values, filtering irrelevant text, etc.; add labels to the dataset; and publish and share the formed dataset.
[0005] However, in the current large language dataset construction process, the filtering step is mostly to filter out repeated semantics. Especially for Chinese, the local meaning of the sentence, the features artificially added during data enhancement, etc. These minor changes may directly lead to deviations in the final prediction of the model, and it is impossible to achieve more accurate extraction results. At the same time, the simultaneous decoding of entities and relationships in the text will lead to the problem of weak connection between the subject and object of the entity. Summary of the Invention
[0006] To address the above shortcomings, the present invention provides a method for constructing a large language dataset based on contrastive learning, which is characterized by comprising the following steps:
[0007] S1, recognize the text through the encoder and extract triple information through the extraction module;
[0008] S1-1, after identification by the language recognition module, the pre-processed semantic data X is obtained by combining the existing large language dataset;
[0009] S1-2, after being identified by the language recognition module, the semantic recognition module is used to perform semantic feature recognition. After being identified by the language recognition module, the semantic recognition module is used to perform semantic feature recognition, wherein the semantic recognition module is used to identify strong features and weak features in the semantics;
[0010] S1-3, independently annotating the corresponding language of the semantic features identified in step S1-2 through the annotation module;
[0011] S2. Correct the subject and object corresponding to the independent annotation content through the correction module to obtain processed semantic data Y;
[0012] S3, performing differential comparison between the processed semantic data Y obtained in step S2 and the pre-processed semantic data X obtained in step S1-1 through a comparison module to obtain a difference value λ and an equivalent value γ;
[0013] S4, collecting the difference λ and forming a filtered training set;
[0014] S5. Collect the equivalent γ and form a validation set;
[0015] S6. Publish and share the data set formed by steps S3 and S4.
[0016] Furthermore, the extraction module includes an entity recognizer for identifying entities in text, a relationship classifier for classifying each entity, a feature extractor for extracting features in the relationship classification process, and a rule engine for verifying grammatical rules.
[0017] Furthermore, the semantic recognition module is used to identify strong features and weak features in semantics:
[0018] Weak features include: semantics with low similarity obtained by the semantic recognition module, keywords with weak relevance identified by the entity recognizer, and features with low matching degree with entities and relations in the knowledge base;
[0019] Strong features include: semantics with high similarity obtained through the semantic recognition module, keywords with strong correlation identified by the entity recognizer, and features with a high degree of matching with entities and relationships in the knowledge base.
[0020] Furthermore, the filtered training set is formed by a part-of-speech filtering matrix generated by a part-of-speech tagger, and the loss function of the algorithm in the part-of-speech filtering matrix includes entity loss, relation loss and grammatical loss.
[0021] Furthermore, the formula of the loss function is:
[0022] ;
[0023] ;
[0024] ;
[0025] In the above formula, L rel is the relationship loss, L ent is the entity loss, L fl is the grammatical loss, n r is the number of relation types, n e is the length of the sentence, i and j represent the rows and columns of the matrix, K i,j Is the true value of the matrix, L rel , L ent and L fl The final loss function is obtained after addition.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] 1. By setting strong and weak features, we can solve the problem that the sentence embeddings generated by the current language model are isotropic and have high semantic similarity between different texts. We can achieve data enhancement by combining repeated markers with MLM to construct weak semantic samples and reduce redundant information interference.
[0028] 2. By setting up the annotation module for independent annotation, when performing differential comparison later, the difference and equal values can be formed, and thus classified into filtered training sets and validation sets respectively;
[0029] Filtering the training set can help ensure the quality and reliability of the training data, exclude low-quality data, improve the training effect of the model, and remove biased data, adjust the data distribution to make it more balanced, which helps improve the generalization ability of the model, improve the model efficiency, prevent overfitting, and improve the generalization ability of the model;
[0030] The validation set can be used to tune model hyperparameters, select the optimal model architecture, implement early stopping strategies, and ultimately evaluate model performance. By effectively using the validation set, you can improve model training efficiency and performance, and ensure that the model has good generalization capabilities in practical applications.
[0031] 3. This method can achieve accurate prediction of large language models, improve the accuracy of extraction results, synchronously decode entities and relationships in text, and increase the connectivity between entities and objects. DETAILED DESCRIPTION
[0032] To facilitate understanding of the present invention, the apparatus of the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate embodiments of the apparatus. However, the apparatus can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.
[0033] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "disposed" should be understood in a broad sense. For example, they may refer to fixed connection or disposition, detachable connection or disposition, or integral connection or disposition. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.
[0034] Example
[0035] This embodiment provides a method for constructing a large language dataset based on contrastive learning, wherein the method for constructing a large language dataset based on contrastive learning includes an encoder for text recognition, an extraction module for extracting triple information, a language recognition module for identifying the language of the text, a semantic recognition module for identifying semantic features, an annotation module for annotation, a correction module for correcting the annotated text, a comparison module for semantic differentiation comparison, and a filtered training set and a validation set for collecting differentiation results;
[0036] The specific steps are as follows:
[0037] S1. Recognize the text through the LLaMA encoder and extract triple information through the extraction module, where the extraction module specifically includes:
[0038] Entity Recognizer: This is used to identify entities in text. Named Entity Recognition (NER) technology is used to identify entities from text, including names of people, places, organizations, and other entities with specific meanings. Before entity recognition, the original text is broken down into smaller units using word segmentation tools such as NLTK and spaCy.
[0039] Relationship classifier: classifies each entity, that is, for the identified entity pairs, a relationship classifier is used to determine the type of relationship between them;
[0040] Feature extractor: extracts features during the relationship classification process. When classifying relationships, various features can be used to represent entity pairs and their context information, such as part-of-speech tagging, syntactic analysis, word vector representation, etc., to obtain the encoder's feature vector h iAfter that, it is input into the relation extraction module. A sentence may contain multiple relations, so relation extraction can be considered a multi-classification task. The self-attention mechanism is used to enhance the degree of association between entities and enhance the information sharing between relations and entities. Then, average pooling is used to generalize to the entire sentence. The specific formula is:
[0041] ;
[0042] ;
[0043] ;
[0044] Where W q 、W k 、W v 、W r and b r is a trainable weight matrix, h avg Refers to the information vector of the entire sentence, P rel It refers to the probability that the correct answer is a certain relationship, h i is the feature vector obtained by the encoder, and E refers to the overall simulated entity information vector;
[0045] Rule engine: used to verify grammatical rules to verify whether the extracted triples conform to specific grammatical rules or constraints;
[0046] S1-1. After the language recognition module identifies the corresponding language, the preprocessed semantic data X is obtained by combining it with existing large language datasets, such as Wikipedia, CommonCrawl, GutenbergProject, Reddit, etc.
[0047] S1-2. After being identified by the language recognition module, the semantic recognition module is used to identify semantic features. The semantic recognition module is used to identify strong and weak features in semantics:
[0048] Weak features include: semantics with low similarity obtained by the semantic recognition module. Common semantic similarity measurement methods include word vector-based methods (such as cosine similarity), semantic graph-based methods (such as WordNet or knowledge graph), and deep learning-based methods (such as Siamese network), etc., and the entity recognizer identifies keywords with weak correlation and features with low matching degree with entities and relationships in the knowledge base. This embodiment adopts the cosine similarity method and uses the model to predict the cosine similarity of each text pair to evaluate semantic similarity. The text sequences are arranged in descending order according to cosine similarity and label, forming sets X and Y. The Spearman correlation coefficient of sets X and Y is used to measure the correlation between the two sets of values to evaluate the model's recognition ability for semantically similar texts. The Spearman correlation coefficient is solved according to the sorting position of the original data. The formula is as follows:
[0049] ;
[0050] Where: x i 、y i are the positions of the i-th text pair in sets X and Y respectively, and n is the number of text pairs in the dataset;
[0051] Strong features include: semantics with high similarity obtained by the semantic recognition module, keywords with strong relevance identified by the entity recognizer, and features with a high degree of matching with entities and relations in the knowledge base;
[0052] Its role is to solve the problem that sentence embeddings generated by current language models are isotropic and have high semantic similarity between different texts. It achieves data enhancement by combining repeated markers with MLM to construct weak semantic samples and reduce redundant information interference.
[0053] S1-3. Use the annotation module to independently annotate the corresponding language of the semantic features identified in step S1-2 (the annotation can be entity annotation, relationship annotation, sentiment annotation, etc., depending on the task and purpose of the dataset);
[0054] S2. Correct the subject and object corresponding to the independent annotation content through the correction module to obtain processed semantic data Y;
[0055] S3. Perform differential comparison between the processed semantic data Y obtained in step S2 and the pre-processed semantic data X obtained in step S1-1 through a comparison module to obtain a difference value λ and an equivalent value γ. For example, when performing differential comparison between the entity semantics after annotation and the entity semantics before annotation, the difference value obtained is the difference value λ, and the identical semantics are the equivalent value γ.
[0056] S3. Collect the difference λ and form a filtered training set. The filtered training set is formed by the part-of-speech filter matrix generated by the part-of-speech tagger. The loss function of the algorithm in the part-of-speech filter matrix includes entity loss, relationship loss and grammatical loss. The formula of the loss function is:
[0057] ;
[0058] ;
[0059] ;
[0060] In the above formula, L rel is the relationship loss, L ent is the entity loss, L fl is the grammatical loss, n r is the number of relation types, n e is the length of the sentence, i and j represent the rows and columns of the matrix, K i,j Is the true value of the matrix, L rel , L ent and L fl The final loss function is obtained after addition;
[0061] S4. Collect the equivalent γ and form a validation set. As the frequency of use increases, the data in the validation set will become more and more complete and used for subsequent model training.
[0062] S5: Publishing and sharing the data set formed by steps S3 and S4. Its purpose is to:
[0063] The filtered training set is used to filter the collected identical difference λ in the subsequent use process, which can help ensure the quality and reliability of the training data. When the original training data set contains errors, noise or inaccurate information, it will have a negative impact on the learning of the model. By filtering the training set, these low-quality data can be excluded, the training effect of the model can be improved, and the biased data can also be removed (there may be some biases in the training data, such as more text in a specific field and less text in other fields, or the number of text samples in some categories far exceeds that of other categories. Such biases may cause the model to be biased towards frequently appearing categories or fields when predicting). By filtering the training set, the data distribution can be adjusted to make it more balanced, which helps improve the generalization ability of the model and improves the model efficiency. Filtering the training set can also reduce the computing cost and time consumption during the training process. Large language models usually require a lot of computing resources and time to train. By filtering out some redundant or low-quality data, the size of the training set can be reduced, thereby improving training efficiency. Finally, it can also prevent overfitting. If there is a lot of noise or irrelevant information in the training data, the model may over-learn this data and ignore the truly useful features. By filtering the training set, the risk of overfitting can be reduced and the generalization ability of the model can be improved.
[0064] The validation set can be used to tune model hyperparameters, select the optimal model architecture, implement early stopping strategies, and ultimately evaluate model performance. By effectively using the validation set, you can improve model training efficiency and performance and ensure that the model has good generalization capabilities in real-world applications.
[0065] It should be noted that the structure described in the present invention can be implemented in a variety of different forms and is not limited to the described embodiments. Any equivalent transformations made by ordinary technicians in this field using the contents of the present invention description and drawings, or directly or indirectly applied to other related technical fields, such as the loading and unloading of other items, are included in the scope of protection of the present invention.
Claims
1. A method for constructing a large language dataset based on contrastive learning, characterized in that: The steps include: S1, recognize the text through the encoder and extract triple information through the extraction module; S1-1, after identification by the language recognition module, the pre-processed semantic data X is obtained by combining the existing large language dataset; S1-2, after being identified by the language recognition module, the semantic recognition module is used to perform semantic feature recognition. After being identified by the language recognition module, the semantic recognition module is used to perform semantic feature recognition, wherein the semantic recognition module is used to identify strong features and weak features in the semantics; S1-3, independently annotating the corresponding language of the semantic features identified in step S1-2 through the annotation module; S2. Correct the subject and object corresponding to the independent annotation content through the correction module to obtain processed semantic data Y; S3, performing differential comparison between the processed semantic data Y obtained in step S2 and the pre-processed semantic data X obtained in step S1-1 through a comparison module to obtain a difference value λ and an equivalent value γ; S4, collecting the difference λ and forming a filtered training set; S5. Collect the equivalent γ and form a validation set; S6. Publish and share the data set formed by steps S3 and S4.
2. The method for constructing a large language dataset based on contrastive learning according to claim 1, characterized in that: The extraction module includes an entity recognizer for recognizing entities in text, a relationship classifier for classifying each entity, a feature extractor for extracting features in the relationship classification process, and a rule engine for verifying grammatical rules.
3. The method for constructing a large language dataset based on contrastive learning according to claim 2, wherein: The semantic recognition module is used to identify strong features and weak features in semantics: Weak features include: semantics with low similarity obtained by the semantic recognition module, keywords with weak relevance identified by the entity recognizer, and features with low matching degree with entities and relations in the knowledge base; Strong features include: semantics with high similarity obtained through the semantic recognition module, keywords with strong correlation identified by the entity recognizer, and features with a high degree of matching with entities and relationships in the knowledge base.
4. The method for constructing a large language dataset based on contrastive learning according to claim 1, wherein: The filtered training set is formed by a part-of-speech filtering matrix generated by a part-of-speech tagger, and the loss function of the algorithm in the part-of-speech filtering matrix includes entity loss, relationship loss and grammatical loss.
5. The method for constructing a large language dataset based on contrastive learning according to claim 4, characterized in that: The formula of the loss function is: ； ； ； In the above formula, L rel is the relationship loss, L ent is the entity loss, L fl is the grammatical loss, n r is the number of relation types, n e is the length of the sentence, i and j represent the rows and columns of the matrix, K i,j Is the true value of the matrix, L rel , L ent and L fl The final loss function is obtained after addition.