A data enhancement-based trusted text semantic detection method

Abstract

The application relates to a data enhancement-based trusted text semantic detection method, which comprises the following steps: evaluating a seed data set by using a text semantic evaluation model, extracting all hard negative samples and simple samples to construct a to-be-enhanced sample set, converting the seed data set into an instruction fine-tuning data set by using a pre-set instruction fine-tuning template, training a large language model by using the instruction fine-tuning data set, inputting the to-be-enhanced sample set into the trained large language model to generate meta-semantic sentences similar to or opposite to the semantics of each to-be-enhanced sample, inputting each meta-semantic template into a sentence generator to generate multiple enhanced sentences, constructing an enhanced sample set according to the meta-semantic sentences and the enhanced sentences, and obtaining the true labels of the enhanced samples by using a multi-label checking method, training the text semantic evaluation model by using the enhanced sample set, and performing semantic detection on trusted texts by using the trained text semantic evaluation model, so that the stability of the text semantic evaluation model is improved.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and text information processing, and in particular to a data-enhanced framework for trusted text semantic detection and evaluation. Background Technology

[0002] Text semantic evaluation is a fundamental task in Natural Language Processing (NLP). Its goal is to measure the semantic similarity between two pieces of text, which is crucial in many NLP applications, such as information retrieval, question answering systems, machine translation, text summarization, and semantic search. For example, in information retrieval applications, a user inputs a query, and the system needs to find the document most semantically similar to the query from a large dataset. In question answering systems, the system needs to find the known question most semantically similar to the user's question and then return the corresponding answer. Therefore, text semantic evaluation is a key component of many NLP tasks and is essential for improving their performance.

[0003] Due to a lack of supervised training data, previous text semantic evaluation methods relied on encoding sentences in a high-dimensional space and using vector-based semantic similarity calculations. Later, natural language inference datasets were proposed, where each sentence pair was manually annotated and categorized as entailment, neutrality, or contradiction based on the existence of logical relationships between them. With the advent of natural language inference datasets, text semantic evaluation, through contrastive learning to train on tasks with similar learning objectives, significantly surpassed previous unsupervised methods. However, both unsupervised and natural language inference-based methods are affected by the lack of direct supervision signals—that is, a well-annotated similarity score containing fine-grained semantics for a pair of sentences. The scarcity of such datasets stems from the significant resources required for data annotation. Further analysis of existing methods reveals that current models cannot accurately identify different syntactic heuristics expressing the same semantic meaning. One major reason is the insufficient generalization ability of existing datasets and the lack of diverse positive and hard negative samples, making models susceptible to adversarial examples in real-world scenarios and leading to semantic evaluation errors in downstream tasks. Summary of the Invention

[0004] To address the problems existing in the background art, the present invention provides a data-enhanced trusted text semantic detection method, which includes:

[0005] S1: Obtain a supervised seed dataset for training sentence embeddings, evaluate the seed dataset using a text semantic evaluation model, and extract all hard negative samples and simple samples.

[0006] S2: Construct the sample set to be enhanced by sampling from the hard negative sample set and the simple sample set according to a preset ratio;

[0007] S3: Convert the seed dataset into an instruction fine-tuning dataset using a pre-defined instruction fine-tuning template;

[0008] S4: Train the large language model using the instruction fine-tuning dataset to align with human evaluation dimensions; input the sample set to be enhanced into the trained large language model to generate meta-semantic sentences that are similar to or opposite to the semantics of each sample to be enhanced, and then use a syntactic parser to parse them to obtain a meta-semantic template set;

[0009] S5: Input each metasemantic template into the sentence generator to generate multiple enhanced sentences, construct an enhanced sample set based on the metasemantic sentences and the enhanced sentences, and obtain the real labels of the enhanced samples using multi-label verification.

[0010] S6: Train the text semantic evaluation model using an enhanced sample set, and then use the trained text semantic evaluation model to perform semantic detection on trustworthy text.

[0011] Preferably, the text semantic evaluation model includes, but is not limited to, the Simcse-bert language model.

[0012] Preferably, the instruction fine-tuning template includes, but is not limited to, standard, few-shot, or cot type templates.

[0013] Preferably, the large language model includes, but is not limited to, the open-source large language model llama2-7B or the closed-source large language model GPT-3.5.

[0014] Preferably, the parser includes, but is not limited to, the NLTK toolkit.

[0015] Preferably, the sentence generator includes, but is not limited to, a trained big oracle model or an existing data augmentation algorithm.

[0016] Preferably, obtaining the true label of the enhanced sample using multi-label verification includes:

[0017] S1: Combine the augmented sentence and the metasemantic sentence to form an augmented sample, input it into the trained Big Prophet model, and obtain its similarity value.

[0018] S2: Combine the enhanced sentence and the metasemantic sentence to form an enhanced sample, input it into the text semantic evaluation model, and obtain its similarity value.

[0019] S3: Order If K is greater than the set threshold, the label value of augmented sample i is manually evaluated and used as the true label; if K is less than or equal to the set threshold, then... As the true label value of augmented sample i.

[0020] The present invention has at least the following beneficial effects

[0021] This invention generates synonyms and antonyms of different styles by aligning a large language model, solving the problem of scarce positive and hard negative samples in traditional contrastive learning methods used for sentence embedding training. It generates a large number of meta-semantic templates through the aligned large language model, thereby generating a large number of augmented samples. These samples are OOD (Out of Distribution) data, addressing the issue of insufficient generalization of training data. The generated augmented samples are labeled using the aligned large language model, then evaluated and filtered using an existing text semantic evaluation model, and finally manually checked for discrepancies in the labeled samples. This significantly reduces the labeling cost and manpower / resources required for text semantic similarity datasets. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0023] Figure 2 This is a schematic diagram of the method flow in an embodiment of the present invention. Detailed Implementation

[0024] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0025] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0026] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0027] Please see Figure 1 and Figure 2 This invention provides a data-enhanced trusted text semantic detection method, comprising:

[0028] S1: Obtain a supervised seed dataset for training sentence embeddings, evaluate the seed dataset using a text semantic evaluation model, and extract all hard negative samples and simple samples.

[0029] In this embodiment, seed datasets for training sentence embeddings are first collected. Different datasets are collected for different sentence embedding training tasks. For example, the dataset format for the NLI task is: {sentence1, sentence2, label(entailment\contradiction\neural)}, while the dataset format for the STS task is: {sentence1, sentence2, score}, etc. For these seed datasets, existing text semantic evaluation models, such as the Simcse-BERT language model, are used to evaluate these seed data. Then, by comparing them with the original labels, the seed data is extracted. All hard negative samples (hard negative samples, as the name suggests, are relatively hard and difficult to distinguish; "hard negative samples" usually refer to those negative samples that are very similar to positive samples and are difficult to distinguish from positive samples in the latent space), for example, in this embodiment, a typical hard negative sample in the STS task is: {I am happy but you are sad, you are happy but I am sad}. The semantics of these two sentences are somewhat different, that is, for the same predicate, their subjects are opposite. After encoding and calculating the similarity through the existing text semantic evaluation model, the similarity value obtained is 0.98 (the similarity value range is 0-1). This shows that the high overlap of words has misled the text semantic evaluation model. Simple samples refer to samples that are easy to distinguish.

[0030] S2: Construct the sample set to be enhanced by sampling from the hard negative sample set and the simple sample set according to a preset ratio;

[0031] In this embodiment, samples are taken from both the hard negative sample set and the simple sample set, with a preset sampling ratio of 0.8 for hard negative samples and 0.2 for ordinary samples.

[0032] S3: Convert the seed dataset into an instruction fine-tuning dataset using a pre-defined instruction fine-tuning template;

[0033] In the instruction fine-tuning format conversion, the seed dataset is converted into an instruction fine-tuning dataset using a pre-defined instruction fine-tuning template. Several instruction fine-tuning templates can be used in this step, with standard, few-shot, and cot templates being commonly used. Finally, after converting all seed datasets, the instruction fine-tuning dataset is obtained. The instruction fine-tuning template is a pre-built structure of input messages designed to generate specific types of responses or results. Especially for language models like OpenAI's ChatGPT, instruction fine-tuning templates can help the model generate the expected answers or perform specific tasks. For example, templates for the text semantic similarity (STS) task are shown in Table 1 below.

[0034] Table 1. Example templates for STS (Semantic Similarity Test) task prompts.

[0035]

[0036]

[0037] For example, in this embodiment, for a piece of data in STS—{A brown dog is running in the snow with a white ball in its mouth. Four dogs are playing in the snow with the city skyline behind them.}, according to the standard template, it can be converted to—{Q: Determine the similarity between the following two sentences: “A brown dog is running in the snow with a white ball in its mouth.” and “Four dogs are playing in the snow with the city skyline behind them.” The score should range from 0.0 to 5.0 and can be a decimal. A: The similarity between these two sentences is 1.}

[0038] S4: Train the large language model using the instruction fine-tuning dataset to align with human evaluation dimensions; input the sample set to be enhanced into the trained large language model to generate meta-semantic sentences that are similar to or opposite to the semantics of each sample to be enhanced, and then use a syntactic parser to parse them to obtain a meta-semantic template set;

[0039] In this embodiment, training a large language model to align with human evaluation dimensions refers to enabling the model to learn the similarity order of manually labeled samples. The reason why this invention does not directly use a large language model to evaluate text similarity is that large models are not suitable for downstream tasks, because tasks such as search and recommendation require similarity calculations between massive amounts of documents, and large model inference is too slow. The instruction fine-tuning dataset obtained in the previous step is fed into the large language model for training. Here, the large language model can be any open-source large language model, such as open-source large language models like llama2-7B or closed-source large language models like GPT-3.5. After training, a large language model M that is relatively aligned with human evaluation dimensions can be obtained. Moreover, the large language model has very rich prior knowledge due to training on massive amounts of data, and can generate sentences with extremely high generalization.

[0040] In this embodiment, based on the previously sampled samples to be enhanced, a large language model is used to generate metasemantic sentences that are semantically similar or opposite to the samples. These metasemantic templates are then parsed by a syntactic parser P to obtain metasemantic templates T. Specifically, the samples to be enhanced obtained in the preprocessing module are fed into the large language model M aligned in the previous step, which generates n semantically similar synonyms and semantically opposite antonyms, i.e., metasemantic sentences, with different styles. The generated synonyms may not be semantically identical, but this is not affected by subsequent annotation of the data, and the same applies to antonyms. Then, the syntactic parser P parses these sentences of different styles to obtain n metasemantic templates T. The syntactic parser P uses the NLTK toolkit, which replaces important content in these sentences of different styles with slots to be filled, such as entities and predicate verbs.

[0041] For example, in this embodiment, for the difficult sentence pair (hard negative sample) "{I am happy but you are sad, you are happy but I am sad}", the synonyms and semantic templates generated for the first sentence are shown in Table 2 below:

[0042] Table 2 Example Table of Metasemantic Templates

[0043]

[0044] S5: Input each metasemantic template into the sentence generator to generate multiple enhanced sentences, construct an enhanced sample set based on the metasemantic sentences and the enhanced sentences, and obtain the real labels of the enhanced samples using multi-label verification.

[0045] In this embodiment, a large number of enhanced sentences are generated by a sentence generator G based on a meta-semantic template T. Specifically, the meta-semantic template T generated in the previous step is fed into the sentence generator G. The sentence generator G can be of various types, such as an aligned large language model M or a mature data augmentation algorithm. For each meta-semantic template t∈T, m different enhanced sentences are generated. Finally, the n meta-semantic templates can generate m×n different enhanced sentences, forming an enhanced sentence set W. For example, in this embodiment, for the difficult sentence pair "{I am happy but you are sad, you are happy but I am sad}", the enhanced sentences generated for the first sentence are shown in Table 3 below:

[0046] Table 3 Examples of Enhanced Sentences

[0047]

[0048] Using these enhanced sentence sets W, enhanced samples S are constructed. The true labels of these samples are obtained through multiple validation methods. Specifically, these enhanced sentence sets W and meta-semantic sentences are combined to form sentence pairs, which are then fed into an aligned large language model for evaluation, yielding a similarity value Score1. The sentence pairs are then encoded using a trained text semantic evaluation model, and their similarity value Score2 is calculated. If K is greater than the set threshold, the label value of augmented sample i is manually evaluated and used as the true label; if K is less than or equal to the set threshold, then... The similarity score used to enhance sample i is 0-5, while the similarity score calculated by the text semantic evaluation model is 0-1, making direct comparison impossible.

[0049] In summary, the invention, through the construction of an enhanced sample set with strong generalization and enhancement for difficult samples, can solve the problems of scarce positive samples, false negative samples in negative samples, insufficient generalization of training data, and low stability of text semantic evaluation models when traditional contrastive learning methods are used for sentence embedding training.

[0050] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A data-augmented method for reliable text semantic detection, characterized in that, include: S1: Obtain a supervised seed dataset for training sentence embeddings, evaluate the seed dataset using a text semantic evaluation model, and extract all hard negative samples and simple samples. S2: Construct the sample set to be enhanced by sampling from the hard negative sample set and the simple sample set according to a preset ratio; S3: Convert the seed dataset into an instruction fine-tuning dataset using a pre-defined instruction fine-tuning template; S4: Train a large language model using instruction fine-tuning datasets to align with human evaluation dimensions; The sample set to be enhanced is input into the trained large language model to generate a meta-semantic sentence for each sample that is similar to or opposite to its semantics. Then, the syntactic parser is used to parse it to obtain a meta-semantic template set. S5: Input each metasemantic template into the sentence generator to generate multiple enhanced sentences, construct an enhanced sample set based on the metasemantic sentences and the enhanced sentences, and obtain the real labels of the enhanced samples using multi-label verification. S6: Train the text semantic evaluation model using the enhanced sample set, and perform semantic detection on credible text using the trained text semantic evaluation model; The method of obtaining the true label of the enhanced sample using multi-label verification includes: Step 1: Combine the augmented sentence and the metasemantic sentence to form an augmented sample, input it into the trained large language model, and obtain its similarity value. ; Step 2: Combine the enhanced sentence and the meta-semantic sentence to form an enhanced sample, input it into the text semantic evaluation model, and obtain its similarity value. ; Step 3: Let ,like If the value exceeds the set threshold, the augmented sample will be manually evaluated. The label value is used as the real label, if If it is less than or equal to the set threshold, then As an enhanced sample The actual label value.

2. The data-enhanced trusted text semantic detection method according to claim 1, characterized in that, The text semantic evaluation model includes the Simcse-bert language model.

3. The data-enhanced trusted text semantic detection method according to claim 1, characterized in that, The instruction fine-tuning templates include standard, few-shot, or cot type templates.

4. The data-enhanced trusted text semantic detection method according to claim 1, characterized in that, The large language model includes the open-source llama2-7B large language model or the closed-source GPT-3.5 large language model.

5. The data-enhanced trusted text semantic detection method according to claim 1, characterized in that, The parser includes the NLTK toolkit.

6. The data-enhanced trusted text semantic detection method according to claim 1, characterized in that, The sentence generator includes a trained large language model or an existing data augmentation algorithm.

Images