Industrial semantic data enhancement method for pre-training model EDA random diffusion

Abstract

The invention discloses a pre-training model EDA random diffusion industrial semantic data enhancement method, and relates to an industrial semantic data enhancement method.The method comprises the following steps that S1, similar text sentence pairs are constructed, one sentence is set as a guide sentence, and the other sentence is used as a to-be-enhanced text sentence; s2, an EDA random strategy is applied, slight changes are introduced into text sentences, and the diversity of original texts in forms is increased; s3, under the framework of the diffusion model, performing forward noise adding processing on the text sentences to be enhanced, and gradually increasing noise; s4, the to-be-denoised text sentences are controlled to keep semantic consistency in the denoising process, and text content is gradually restored through a reverse denoising step; and S5, finally generating a new text which is consistent with the original text in semantics but different from the original text in form, wherein the new text is used for enriching training data. According to the method, the data annotation cost is reduced, the model robustness and generalization ability are improved, and the method is suitable for the data scarcity problem of industrial scenes such as automatic question answering.

Description

Technical Field

[0001] The invention relates to a text enhancement method, in particular to an industrial semantic data enhancement method of a pre-training model EDA random diffusion. Background Art

[0002] Pre-trained models are an important concept in the field of deep learning, especially in natural language processing (NLP). These models are usually trained on large-scale datasets to learn general characteristics of language, and can then be used to solve various specific NLP tasks such as text classification, sentiment analysis, question answering systems, and semantic similarity assessment. The main benefit of using pre-trained models is that they can capture rich language representations learned from the data, which can be transferred to specific tasks.

[0003] Data augmentation refers to a technical method used in machine learning and deep learning tasks to increase the diversity and quantity of data by transforming, expanding or generating new data. This can improve the performance and generalization ability of the model. Data augmentation was first applied in the field of images, usually including geometric transformations such as translation, rotation, scaling, and flipping of images, as well as pixel-level transformations such as brightness, contrast, and color. With the development of natural language processing, the concept of data augmentation has also been introduced into the field of text.

[0004] Chinese processing faces unique challenges in NLP, which mainly stem from the language structure and usage habits of Chinese. Chinese does not have clear word boundaries like Western languages. Different combinations of the same Chinese characters form different meanings. Chinese has a large number of synonyms and polysemous words. The same word has different meanings under different semantics, which increases the complexity of semantic understanding. Chinese speakers sometimes prefer to use more implicit expressions, which requires more context to understand the true meaning.

[0005] The Diffusion Model initially made breakthroughs in the field of image generation, especially in generating high-quality images by gradually adding noise and denoising to restore the image. In recent years, the Diffusion Model has been gradually applied in the field of text and has shown its potential. Its basic principle is to introduce noise into text data and then gradually denoise it to generate new text. This method is suitable for enhancing datasets, especially for generating texts with similar semantics but diverse forms, thereby enriching training data. The Diffusion Model can maintain semantic consistency and is suitable for tasks such as text classification, sentiment analysis, and automatic question answering. It solves the problem of scarce text data and improves the generalization ability of the model. Summary of the invention

[0006] The purpose of the present invention is to provide an industrial semantic data enhancement method of random diffusion of a pre-trained model EDA. In response to the problems of scarcity of text data in industrial application question-answering systems and difficulty in manual annotation, the present invention provides a semantic data enhancement method for training samples of industrial production-related question-answering systems based on a pre-trained model, EDA, and random diffusion, so as to enrich training data and improve model performance.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] A method for enhancing semantic data for industrial applications of random diffusion of a pre-trained model EDA, comprising the following steps:

[0009] S1: construct similar text sentence pairs, set one sentence as the guide sentence and the other sentence as the text sentence to be enhanced;

[0010] S2: Apply EDA random strategy to introduce slight changes in text sentences to increase the formal diversity of the original text;

[0011] S3: Under the diffusion model framework, combined with the encoder of the ERNIE pre-trained model, the text sentence to be enhanced is forward-noised and the noise is gradually increased;

[0012] S4: In the discrete domain of the diffusion model, the information of the guide sentence is used, combined with the decoding ability of the ERNIE pre-trained model, to control the semantic consistency of the text sentence to be denoised during the denoising process, and gradually restore the text content through the reverse denoising step;

[0013] S5: Generate new text that is semantically consistent with the original text but different in form to enrich the training data.

[0014] The industrial application semantic data enhancement method of the pre-trained model EDA random diffusion, the original data set uses similar text sentence pairs as the data set, and uses them as guide sentences and text sentences respectively.

[0015] The industrial application semantic data enhancement method of the pre-trained model EDA random diffusion, the EDA random strategy only performs simple enhancement on the text sentences, can increase the diversity of data to a certain extent without introducing additional noise.

[0016] The industrial application semantic data enhancement method of the pre-trained model EDA random diffusion and the denoising and denoising process are combined with the encoding and decoding capabilities of the ERNIE model. ERNIE is integrated into the diffusion model to jointly learn the denoising and denoising process of discrete sequences.

[0017] The industrial application semantic data enhancement method of random diffusion of a pre-trained model EDA, the guide sentence and the text sentence, as the overall input sequence, the model can directly generate a text sequence that meets the expected attributes without the need for an additional classifier to predict the attributes of the text;

[0018] Define sentence a as the introductory sentence S a , define sentence b as text sentence S b , train a model, input S a , can generate sentence S with similar semantics to sentence a b′ :

[0019] p(S b′ |S a )∝p(S b′ )·p(S a |S b′ )

[0020] Among them, p(S b′ ) represents the generated text sentence S b′ The prior probability of generating S under no conditions b The probability, p(S a |S b′ ) means that given the generated text sentence S b′ Under the condition of a The probability of occurrence describes the generated text sentence S b′ With the introductory sentence S a By jointly considering these two parts of probability, we can get the semantic similarity between the given guide sentence S a Generate a text sentence S with similar semantics under the condition b′ The probability p(S b′ |S a ).

[0021] The industrial application semantic data enhancement method of the pre-trained model EDA random diffusion, the existence of the guide part, ensures that the key features of the text part, such as the theme, tone, style, etc., are retained to the greatest extent, so that the denoised text still maintains features similar to the original text.

[0022] The industrial application semantic data enhancement method of random diffusion of a pre-trained model EDA, the overall diffusion process is as follows:

[0023]

[0024] The diffusion single-step process is:

[0025] p(S t-1 |S t ,Sa )∝p(S t-1 |S t )·p(S a |S t-1 ,S t )

[0026] The optimization goal of the tth step in the diffusion process is:

[0027]

[0028] Among them, S t Represents the diffusion state of the input sequence S at time t.

[0029] In the method for enhancing semantic data for industrial applications of random diffusion of a pre-trained model EDA, the role of the guide sentence in the denoising process is to provide a semantic reference to help the generation model better understand the noise and remove it, and to promptly correct the semantically inconsistent parts. Finally, a final target sentence with a different style from the original text sentence and similar semantics to the guide sentence is generated to form a new sequence.

[0030] The advantages and effects of the present invention are:

[0031] 1. The present invention aims to solve the problem of relying on diversified samples in industrial applications, enrich the data set for training, and solve the problems of scarcity of text data and difficulty in manual labeling. It provides a semantic data enhancement method for training samples of industrial production-related question-answering systems based on pre-training models, EDA, and random diffusion to enrich training data.

[0032] 2. The present invention can effectively generate new text that is semantically consistent with the original text but different in form by integrating the random exchange strategy of EDA technology, the deep semantic understanding ability of the pre-trained model, and the innovative text generation mechanism of the diffusion model, thereby enhancing the training data set of the model and improving the performance and generalization ability of the model.

[0033] 3. The present invention can be widely used in intelligent question-answering phrase similarity matching systems such as industrial applications, medical question-answering, and agriculture, and can effectively expand the training set by generating new texts with consistent semantics but diverse forms. This method not only reduces the cost of data annotation, but also improves the robustness and generalization ability of the system when processing diversified expressions, so that it can show high matching accuracy and reliability in a variety of scenarios, thereby improving the practicality and intelligence level of the question-answering system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a detailed structural diagram of the industrial application semantic data enhancement method based on random diffusion of the pre-trained model EDA of the present invention;

[0035] Figure 2It is a flowchart of the industrial application semantic data enhancement method based on random diffusion of pre-trained model EDA of the present invention;

[0036] Figure 3 It is a model fusion process diagram of the industrial application semantic data enhancement method based on random diffusion of the pre-training model EDA of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0038] Example:

[0039] The present invention provides an industrial application semantic data enhancement method of random diffusion of a pre-trained model EDA, comprising the following steps:

[0040] S1: construct similar text sentence pairs, set one sentence as the guide sentence and the other sentence as the text sentence to be enhanced;

[0041] In this embodiment, the text pairs with similar semantics in the original data set are divided into sentence a and sentence a′.

[0042] S2: Apply EDA random strategy to introduce slight changes in text sentences to increase the formal diversity of the original text.

[0043] Since random insertion, deletion or synonym replacement may cause the generated new sentences to be semantically inconsistent with the original sentences, in order to avoid additional noise in the input sequence and affect the training process, this example only uses the EDA random exchange strategy to simply enhance the text sentences, which can increase the diversity of the data to a certain extent without introducing additional noise, thereby avoiding affecting the training process.

[0044] In this embodiment, the leading sentence S a , and the text sentence S generated by random exchange of EDA b Combined together, they form the input sequence represented by S.

[0045] S3: Under the diffusion model framework, combined with the encoder of the ERNIE pre-trained model, the text sentence to be enhanced is forward-noised and the noise is gradually increased.

[0046] In this embodiment, in the forward diffusion direction, S0-S t represents the diffusion state of the input sequence S at time 0-t. Using the pre-trained model as the encoder, only the text sentence S is randomly replaced by EDA. b Each time step can be performed by In the process of forward noise addition, the leading sentence S a With text sentence Sb are passed to the diffusion model together to provide additional semantic information and constraints. The association between the two sentences is regarded as a semantic association to model the feature relationship between the two.

[0047] In the diffusion process, the text sentence is directly diffused, and the random replacement operation of EDA is used to add noise to the text sentence, that is, the mask word [MAKS] is used to randomly replace the original token in the text sentence. In order to solve the problem of text discreteness, the diffusion process is mainly extended to the discrete domain. Direct diffusion is carried out in the discrete domain, and there is no need to encode the text into a continuous space. In the forward diffusion process, the randomly replaced mask words are treated as noise. In the forward diffusion process, starting from time t0, each step gradually uses the mask word to randomly replace the text sentence S b For each original token in , as the time step increases, more and more original tokens are replaced, that is, the noise gradually diffuses. At time step t, the noise is completely diffused, and the text sentence S b All original tokens of are replaced completely randomly.

[0048] S4: In the discrete domain of the diffusion model, the guide sentence information is used, combined with the decoding ability of the ERNIE pre-trained model, to control the semantic consistency of the text sentences to be denoised during the denoising process, and gradually restore the text content through the reverse denoising step.

[0049] In this embodiment, the reverse diffusion process also only affects S b Perform denoising. a As a guide, the pre-trained model decoder fused the diffusion model to S b The calculation process at this time is slightly different from the traditional diffusion model. In the traditional diffusion model, the decoder generates the sentence of t-1 time step based on the sentence generated in t time step, that is, In this case, the decoding denoising process of the decoder depends not only on the generated result of the t time step, but also on the guiding sentence S a The impact of This can be understood as a conditional language model. By fusing the guide sentence in the decoder, it can help the decoder better understand the semantics of the input text and generate an output sentence that is semantically similar to the guide sentence.

[0050] In the process of reverse denoising, the guiding sentence S aThe role of is to provide a semantic reference to help the generative model better understand the noise and remove it, and to promptly correct the semantically inconsistent parts. The guiding attributes of the sequence S remain unchanged, and the text part is denoised. In this process, due to the existence of the guiding part, the key features of the text part, such as the theme, tone, and style, will be retained to the greatest extent, so that the denoised text still retains features similar to the original text.

[0051] The overall diffusion process is:

[0052]

[0053] The diffusion single-step process is:

[0054] p(S t-1 |S t ,S a )∝p(S t-1 |S t )·p(S a |S t-1 ,S t )

[0055] The optimization goal of the tth step in the diffusion process is:

[0056]

[0057] Among them, S t Represents the diffusion state of the input sequence S at time t.

[0058] By combining the guide sentence and the text sentence as the overall input sequence, the model can directly generate a text sequence that meets the expected attributes without the need for an additional classifier to predict the attributes of the text. Due to the similarity between the guide sentence and the text sentence, the generated text tends to be more consistent with the expected semantics, more coherent and reasonable, and less likely to deviate from the original context.

[0059] S5: Generate new text that is semantically consistent with the original text but different in form to enrich the training data.

[0060] In this embodiment, ERNIE adopts a bidirectional language model (BLM), and the diffusion model is a generative model (GM). The contradiction between the two lies mainly in how to balance the relationship between the two. BLM usually considers the information of the entire context in order to better understand the semantics and context. This means that the model needs to capture the bidirectional information of the entire text sequence during the training process, so that it can accurately predict the words at each position. In GM, the model usually needs to generate a text sequence word by word, which conflicts with the need for the bidirectional model to capture the entire context information. However, the input sequence consisting of a guide sentence and a text sentence can avoid this conflict. Specifically, by retaining a sentence as a guide sentence, the model can still obtain the context and semantic information of the entire sentence from the guide sentence, thereby retaining the advantages of the bidirectional language model; due to the existence of the guide sentence, the model can consider the context information word by word during the generation process to ensure that the generated words are consistent with the context, thereby ensuring the needs of the generation model. Finally, the above method is used to obtain a new text that is semantically consistent with the original text but more diverse in form.

[0061] The present invention has been described in detail through the accompanying drawings and embodiments. After reading the content of the present invention, those skilled in the art may make any changes or modifications to the present invention, and these equivalent forms also belong to the scope of protection defined by the appended claims of the present application.

Claims

1. An industrial semantic data enhancement method for random diffusion of a pre-trained model EDA, characterized in that: The method comprises the following steps: S1: construct similar text sentence pairs, set one sentence as the guide sentence and the other sentence as the text sentence to be enhanced; S2: Apply EDA random strategy to introduce slight changes in text sentences to increase the formal diversity of the original text; S3: Under the diffusion model framework, combined with the encoder of the ERNIE pre-trained model, the text sentence to be enhanced is forward-noised and the noise is gradually increased; S4: In the discrete domain of the diffusion model, the information of the guide sentence is used, combined with the decoding ability of the ERNIE pre-trained model, to control the semantic consistency of the text sentence to be denoised during the denoising process, and gradually restore the text content through the reverse denoising step; S5: Generate new text that is semantically consistent with the original text but different in form to enrich the training data.

2. The industrial semantic data enhancement method of random diffusion of a pre-trained model EDA according to claim 1 is characterized in that: The method adopts similar text sentence pairs as a dataset, and uses them as guide sentences and text sentences respectively.

3. The industrial semantic data enhancement method of random diffusion of a pre-trained model EDA according to claim 1 is characterized in that: The EDA random strategy described above only performs simple enhancement on text sentences, increasing the diversity of data without introducing additional noise.

4. The industrial semantic data enhancement method of random diffusion of a pre-trained model EDA according to claim 1 is characterized in that: The method combines the encoding and decoding capabilities of the ERNIE model in the denoising and denoising process, and ERNIE is integrated into the diffusion model to jointly learn the denoising and denoising process of discrete sequences.

5. The industrial semantic data enhancement method of random diffusion of a pre-trained model EDA according to claim 1 or 2, characterized in that: The guide sentence and the text sentence are used as a whole input sequence, and the model directly generates a text sequence that meets the expected attributes without the need for an additional classifier to predict the attributes of the text; Sentence a is the introductory sentence S a , sentence b is the text sentence S b , train a model, input S a , can generate sentence S with similar semantics to sentence a b' : p(S b' |S a )∝p(S b' )·p(S a |S b' ) Among them, p(S b' ) represents the generated text sentence S b' The prior probability of generating S under no conditions b' The probability, p(S a |S b' ) means that given the generated text sentence S b' Under the condition of a The probability of occurrence describes the generated text sentence S b' With the introductory sentence S a The semantic similarity between them; by jointly considering these two parts of probability, we can get a Generate a text sentence S with similar semantics under the condition b ' The probability p(S b' |S a ).

6. The industrial semantic data enhancement method of random diffusion of a pre-trained model EDA according to claim 1 or 2, characterized in that: The existence of the guide part allows the key features of the theme, tone and style of the text part to be retained to the greatest extent, so that the denoised text still maintains features similar to the original text.

7. The industrial semantic data enhancement method of random diffusion of a pre-trained model EDA according to claim 1 is characterized in that: The overall diffusion process is as follows: The diffusion single-step process is: p(S t-1 |S t ,S a )∝p(S t-1 |S t )·p(S a |S t-1 ,S t ) The optimization goal of the tth step in the diffusion process is: ▽x t-1 logp(S t-1 |S t ,S a )=▽S t-1 logp(S t-1 |S t )+▽S t-1 logp(S a |S t-1 ) Among them, S t Represents the diffusion state of the input sequence S at time t.

8. The industrial semantic data enhancement method of random diffusion of a pre-trained model EDA according to claim 1 is characterized in that: The role of the guide sentence in the denoising process is to provide a semantic reference to help the generation model better understand the noise and remove it, and to promptly correct the semantically inconsistent parts; ultimately, a final target sentence is generated that is different from the original text sentence style and has similar semantics to the guide sentence to form a new sequence.

Images