Government affair service text data amplification method based on data enhancement technology, storage medium, equipment and computer program product

Abstract

The invention discloses a government affair service text data amplification method based on a data enhancement technology, a storage medium, equipment and a computer program product, and the method comprises the steps: collecting historical government affair service text data, preprocessing the historical government affair service text data to obtain a corpus data set, and determining keywords of the historical government affair service text data in the corpus data set through a word vector model; the determined keywords are reserved in the corresponding historical government affair service text data, non-keywords are replaced with underlines, and to-be-expanded text data are obtained; and designing a prompt prompt word of the base large model Qwen-7B, taking the to-be-expanded text data as the input of the prompt prompt word, taking the corresponding historical government affair service text data as a label, and amplifying the to-be-expanded text data through the base large model Qwen-7B to obtain amplified government affair service text data. According to the method, the characteristics of the government affair service text data can be better understood, so that low-cost and high-quality government affair service text data amplification is realized.

Description

Technical Field

[0001] The present invention relates to the field of text data augmentation technology, and in particular, to a government service text data augmentation method, storage medium, device and computer program product based on data enhancement technology. Background Art

[0002] Early natural language generation was mainly based on simple rules and templates. Researchers manually designed and wrote rules to generate specific texts. This method can achieve certain results in specific fields and simple scenarios, but due to the limitations of the rules, it is difficult to deal with complex and diverse natural language phenomena and semantic relationships, and its expression ability is limited. With the development of computer technology and the gradual enrichment of corpora, statistical methods have begun to emerge in the field of natural language processing. Statistical methods learn and reason based on a large amount of corpus data, and generate natural and fluent text by calculating statistical information such as the probability distribution between words and language models. They can better cope with the flexibility and diversity of language, but there are also problems such as high model complexity and large demand for large-scale training data. Later, the development of machine learning brought major changes to natural language processing. Neural network models such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and transformers (Transformers) have been widely used in natural language generation tasks and have achieved remarkable results. These models can automatically learn the correlation and dependency between input data, thereby generating high-quality natural language texts, greatly improving the effect and flexibility of text generation, and promoting the further development of natural language text enhancement technology. In recent years, pre-trained language models have become one of the important advances in the field of natural language processing. By performing unsupervised pre-training on massive amounts of text data, the model can learn rich language knowledge and semantic expression capabilities, such as GPT, BERT and other models. The emergence of pre-trained language models not only improves the quality and efficiency of text generation, but also provides a strong foundation for various natural language processing tasks. By fine-tuning or further training on specific tasks, it can quickly realize a variety of applications such as text classification, question-answering systems, and text summarization, further promoting the widespread application of natural language text enhancement technology in different fields.

[0003] In practical applications, the commonly used data augmentation methods are as follows:

[0004] 1. EDA (Easy Data Augmentation) technology increases data diversity by performing random synonym replacement, insertion, exchange and deletion operations on the original text. This method is simple to operate and has a fast amplification speed. However, in the field of government services, for some government texts that require in-depth semantic understanding and contextual information, it may not be able to accurately capture their semantic relationships. In addition, due to the existence of random operations, there is a certain probability that some unreasonable or inconsistent government logic texts will be introduced, which will affect the quality of data amplified in subsequent tasks. Multiple combinations of random operations are likely to lead to problems such as semantic changes and incoherent texts.

[0005] 2. Back-translation technology uses machine translation tools to convert text between different languages ​​to generate new text. Back-translation introduces expressions and semantic information of different languages, enriches the diversity of text, and has unique advantages for multilingual tasks or data enhancement with cross-language characteristics. However, the effect of back-translation depends largely on the performance of the machine translation system and the correspondence between different languages. For some texts in the field of government services, it may not be possible to achieve the ideal amplification effect. There are differences in vocabulary, grammar and culture between different languages. In the process of back-translation, some professional terms and concepts unique to the government field may be lost, affecting the accurate understanding and processing of government texts.

[0006] 3. The BERT (Bidirectional Encoder Representations from Transformers) data augmentation method is mainly based on the powerful understanding and generation capabilities of the BERT model, and transforms and expands the original text data in various ways to increase the diversity and scale of the data, thereby improving the performance and generalization ability of the model. However, the BERT model itself has a complex structure, and training and reasoning require a lot of computing resources to support it. Data augmentation will further increase the amount of computing and time costs. Although data augmentation can increase the amount of data, if the annotation quality of the original data is not high or the annotation information is insufficient, it may affect the training effect of the model. In addition, the annotation data in the government affairs field often requires professional knowledge and manual review, and the cost of obtaining high-quality annotated data is high. If there are problems with the annotated data, even if data augmentation is performed, it is difficult to effectively improve the performance of the model. Summary of the invention

[0007] In response to the problems existing in the prior art, the present invention provides a government service text data amplification method, storage medium, device and computer program product based on data enhancement technology, which can better understand the characteristics of government service text data, thereby achieving low-cost, high-quality government service text data amplification.

[0008] To achieve the above technical objectives, the present invention adopts the following technical solution: a method for amplifying government service text data based on data enhancement technology, specifically comprising the following steps:

[0009] Step S1, collect historical government service text data, filter sensitive words, perform Jieba word segmentation and convert simplified and traditional Chinese fonts on each historical government service text data to obtain a corpus data set;

[0010] Step S2: Using the word vector model to determine keywords for the historical government service text data in the corpus data set;

[0011] Step S3: retain the determined keywords in the corresponding historical government service text data, and replace non-keywords with underscores to obtain the text data to be expanded;

[0012] Step S4, design the prompt word of the base large model Qwen-7B, use the text data to be expanded as the input of the prompt word, and use the corresponding historical government service text data as the label. Amplify the text data to be expanded through the base large model Qwen-7B to obtain the amplified government service text data.

[0013] Furthermore, step S2 includes the following sub-steps:

[0014] Step S2.1, filter each piece of historical government service text data in the corpus data set to obtain a vocabulary containing all non-repeated words, and set the background window size and negative sampling number of the word vector model;

[0015] Step S2.2, traverse all words in the vocabulary, and determine the number of occurrences of each word in the historical government service text data;

[0016] Step S2.3, extract words according to the number of occurrences of each word in the historical government service text data to determine the probability of the word being retained, and determine the background words corresponding to the extracted words according to the background window size, and combine the extracted words and the background words into positive samples; negatively sample the words that do not co-occur with the extracted words according to the number of occurrences of each word in the historical government service text data and the number of negative samples to obtain negative samples;

[0017] Step S2.4: Input the positive samples and negative samples into the word vector model for training, predict the probability of the extracted words as keywords, sort the predicted probabilities from high to low, and select the top 30% of the words as keywords.

[0018] Furthermore, in step S2.3, the calculation process of determining the probability of a word being retained according to the number of occurrences of each word in the historical government service text data is as follows:

[0019]

[0020] Among them, w i represents the i-th word in the vocabulary, z(w i ) indicates w i The number of occurrences in the historical government service text data, p(w i ) indicates w i The probability of being retained.

[0021] Furthermore, the probability of negative sampling of words that do not co-occur with the extracted words in step S2.3 is:

[0022]

[0023] Among them, n represents the total number of words that do not co-occur with the extracted words, j represents the index of n, and w j represents the jth word that does not co-occur with the extracted word, f(w j ) indicates w j The number of occurrences in the historical government service text data, P(w j ) indicates w j The probability of negative sampling.

[0024] Furthermore, the designed prompt words include: role, target and style, the role is set to be a government service hotline staff, the target is set to be expanded with the text data to be expanded, and the style is set to conform to the characteristics of government services.

[0025] Furthermore, the specific process of amplifying the text data to be expanded by the base large model Qwen-7B is as follows:

[0026] i. Set the training parameters of the base large model Qwen-7B, including training weight, batch size, GPU and learning rate;

[0027] ii. The text data to be expanded is used as the input of the prompt word, and the text data to be expanded is amplified by the base large model Qwen-7B to obtain the amplified government service text data;

[0028] iii. Calculate the text similarity loss function of the expanded government service text data and the corresponding labels, repeat steps i-ii until the text similarity loss function converges, and complete the training of the base large model Qwen-7B;

[0029] iv. Amplify the text data to be expanded through the trained base large model Qwen-7B to obtain the amplified government service text data.

[0030] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the government service text data amplification method based on data enhancement technology.

[0031] Furthermore, the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the government service text data amplification method based on data enhancement technology is implemented.

[0032] Furthermore, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the government service text data amplification method based on data enhancement technology.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] (1) The government service text data amplification method based on data enhancement technology of the present invention determines the probability of retaining a word according to the number of occurrences of each word in the historical government service text data, and extracts words. The keyword probability prediction of more unimportant words can be discarded, thereby reducing the calculation burden of the subsequent word vector model and purifying keywords. In addition, the present invention uses positive samples and negative samples to jointly train the word vector model, so that the word vector model can better learn the semantic relationship between words, reduce the amount of calculation, and improve training efficiency;

[0035] (2) The government service text data augmentation method based on data enhancement technology of the present invention better understands the text characteristics of the government field by designing the prompt words of the base large model Qwen-7B, standardizes the augmented government service text data, and thus generates augmented text that conforms to the government service field. At the same time, the corresponding historical government service text data is used as a label to optimize the parameters of the base large model Qwen-7B, so that the augmented text generated by the base large model Qwen-7B can retain the original key phrases and expand the non-key parts in a diversified manner, thereby providing a sufficient and diverse data basis for subsequent tasks and realizing low-cost, high-quality data augmentation.

[0036] (3) The government service text data amplification method based on data enhancement technology of the present invention can automatically generate high-quality government service text data, has a wide range of applications, and is more efficient and low-cost than other amplification methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the method for amplifying government service text data based on data enhancement technology of the present invention;

[0038] Figure 2This is a flow chart of amplifying the text data to be expanded by the large base model Qwen-7B in the present invention. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.

[0040] like Figure 1 The flowchart of the government service text data augmentation method based on data enhancement technology of the present invention is as follows:

[0041] Step S1, collect historical government service text data, perform sensitive word filtering, jieba word segmentation and simplified and traditional font conversion on each historical government service text data to obtain a corpus data set; specifically, sensitive word filtering is to cover the relevant place names, names, ID numbers and mobile phone numbers in the historical government service text data with special characters; jieba word segmentation is to perform jieba word segmentation on the sensitive word filtered historical government service text data, and traverse the word segmentation set according to the stop word list and the sensitive word list to remove stop words and sensitive words; simplified and traditional font conversion is to convert traditional Chinese characters in the historical government service text data into simplified Chinese characters.

[0042] Step S2: Determine keywords from the historical government service text data in the corpus dataset through a word vector model; specifically, the following sub-steps are included:

[0043] Step S2.1, filter each piece of historical government service text data in the corpus data set to obtain a vocabulary containing all non-repeated words, and set the background window size and negative sampling number of the word vector model;

[0044] Step S2.2, traverse all words in the vocabulary, and determine the number of occurrences of each word in the historical government service text data;

[0045] Step S2.3, for commonly used high-frequency but unimportant words, such as the word "if", which has a high probability of appearing in the text, during the word vector model training process, if a large number of non-keywords appear, it will interfere with the word vector model learning keyword sample features. Therefore, it is necessary to reduce the possibility of training the word vector model with high-frequency but unimportant words through sampling and negative sampling. The probability of retaining words is determined according to the number of occurrences of each word in the historical government service text data. The keyword probability prediction of more unimportant words can be discarded, thereby reducing the computational burden of the subsequent word vector model, purifying keywords, and determining the background words corresponding to the extracted words according to the background window size. The extracted words and background words are combined into positive samples; negative sampling is performed on words that do not co-occur with the extracted words according to the number of occurrences of each word in the historical government service text data and the number of negative sampling to obtain negative samples;

[0046] In the present invention, the calculation process of determining the probability of a word being retained according to the number of occurrences of each word in the historical government service text data is as follows:

[0047]

[0048] Among them, w i represents the i-th word in the vocabulary, z(w i ) indicates w i The number of occurrences in the historical government service text data, p(w i ) indicates w i The probability of being retained.

[0049] The probability of negative sampling of words that do not co-occur with the extracted words in the present invention is:

[0050]

[0051] Among them, n represents the total number of words that do not co-occur with the extracted words, j represents the index of n, and w j represents the jth word that does not co-occur with the extracted word, f(w j ) indicates w j The number of occurrences in the historical government service text data, P(w j ) indicates w j The probability of negative sampling.

[0052] Step S2.4: Input the positive samples and negative samples into the word vector model for training, predict the probability of extracting words as keywords, so that the word vector model can better learn the semantic relationship between words, reduce the amount of calculation, improve training efficiency, sort the predicted probabilities from high to low, and select the top 30% of the words as keywords.

[0053] Step S3: retain the determined keywords in the corresponding historical government service text data, and replace non-keywords with underscores to obtain the text data to be expanded.

[0054] Step S4, design the prompt words of the base large model Qwen-7B, including: role, target and style, the role is set as the staff of the government service hotline, the target is set to expand the text data to be expanded, and the style is set to conform to the characteristics of government services. The text data to be expanded is used as the input of the prompt words, so as to better understand the text characteristics of the government field, standardize the expanded government service text data, and thus generate expanded text that conforms to the government service field. The corresponding historical government service text data is used as a label, and the text data to be expanded is amplified by the base large model Qwen-7B to obtain the expanded government service text data. The expanded text generated by the base large model Qwen-7B can retain the original key phrases and expand the non-key parts in a diversified manner, providing a sufficient and diverse data basis for subsequent tasks, realizing low-cost, high-quality data amplification, and can automatically generate high-quality government service text data. It has a wide range of applications and is efficient and low-cost compared to other amplification methods. Specifically, Figure 2 :

[0055] i. Set the training parameters of the base large model Qwen-7B, including training weight, batch size, GPU and learning rate;

[0056] ii. The text data to be expanded is used as the input of the prompt word, and the text data to be expanded is amplified by the base large model Qwen-7B to obtain the amplified government service text data;

[0057] iii. Calculate the text similarity loss function of the expanded government service text data and the corresponding labels, repeat steps i-ii until the text similarity loss function converges, and complete the training of the base large model Qwen-7B;

[0058] iv. Amplify the text data to be expanded through the trained base large model Qwen-7B to obtain the amplified government service text data.

[0059] The government service text data amplification method based on data enhancement technology of the present invention can better understand the characteristics of government service text data, thereby achieving low-cost and high-quality government service text data amplification.

[0060] In a technical solution of the present invention, a computer-readable storage medium is also provided, storing a computer program, wherein the computer program enables a computer to execute the government service text data amplification method based on data enhancement technology.

[0061] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the government service text data amplification method based on data enhancement technology is implemented.

[0062] In a technical solution of the present invention, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the government service text data amplification method based on data enhancement technology.

[0063] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0064] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0065] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for augmenting government service text data based on data enhancement technology, characterized in that: The specific steps include: Step S1, collect historical government service text data, filter sensitive words, perform Jieba word segmentation and convert simplified and traditional Chinese fonts on each historical government service text data to obtain a corpus data set; Step S2: Using the word vector model to determine keywords for the historical government service text data in the corpus data set; Step S3: retain the determined keywords in the corresponding historical government service text data, and replace non-keywords with underscores to obtain the text data to be expanded; Step S4, design the prompt word of the base large model Qwen-7B, use the text data to be expanded as the input of the prompt word, and use the corresponding historical government service text data as the label. Amplify the text data to be expanded through the base large model Qwen-7B to obtain the amplified government service text data.

2. According to a method for amplifying government service text data based on data enhancement technology according to claim 1, it is characterized in that: Step S2 includes the following sub-steps: Step S2.1, filter each piece of historical government service text data in the corpus data set to obtain a vocabulary containing all non-repeated words, and set the background window size and negative sampling number of the word vector model; Step S2.2, traverse all words in the vocabulary, and determine the number of occurrences of each word in the historical government service text data; Step S2.3, extracting words according to the number of occurrences of each word in the historical government service text data to determine the probability of the word being retained, and determining the background words corresponding to the extracted words according to the background window size, and combining the extracted words and the background words into a positive sample; According to the number of occurrences of each word in the historical government service text data and the number of negative samples, negative samples are performed on words that do not co-occur with the extracted words to obtain negative samples; Step S2.4: Input the positive samples and negative samples into the word vector model for training, predict the probability of the extracted words as keywords, sort the predicted probabilities from high to low, and select the top 30% of the words as keywords.

3. According to claim 2, a method for amplifying government service text data based on data enhancement technology is characterized in that: The calculation process of determining the probability of a word being retained according to the number of occurrences of each word in the historical government service text data in step S2.3 is as follows: Among them, w i represents the i-th word in the vocabulary, z(w i ) indicates w i The number of occurrences in the historical government service text data, p(w i ) indicates w i Probability of being retained.

4. According to a method for amplifying government service text data based on data enhancement technology according to claim 2, it is characterized in that: The probability of negative sampling of words that do not co-occur with the extracted words in step S2.3 is: Among them, n represents the total number of words that do not co-occur with the extracted words, j represents the index of n, and w j represents the jth word that does not co-occur with the extracted word, f(w j ) indicates w j The number of occurrences in the historical government service text data, P(w j ) indicates w j The probability of negative sampling.

5. According to a method for amplifying government service text data based on data enhancement technology according to claim 2, it is characterized in that: The designed prompt words include: role, target and style. The role is set as a government service hotline staff, the target is set to expand the text data to be expanded, and the style is set to conform to the characteristics of government services.

6. The method for amplifying government service text data based on data enhancement technology according to claim 5 is characterized in that: The specific process of amplifying the text data to be expanded by the base large model Qwen-7B is as follows: i. Set the training parameters of the base large model Qwen-7B, including training weight, batch size, GPU and learning rate; ii. The text data to be expanded is used as the input of the prompt word, and the text data to be expanded is amplified by the base large model Qwen-7B to obtain the amplified government service text data; iii. Calculate the text similarity loss function of the expanded government service text data and the corresponding labels, repeat steps i-ii until the text similarity loss function converges, and complete the training of the base large model Qwen-7B; iv. Amplify the text data to be expanded through the trained base large model Qwen-7B to obtain the amplified government service text data.

7. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables the computer to execute the government service text data augmentation method based on data enhancement technology as described in any one of claims 1-6.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for amplifying government service text data based on data enhancement technology as described in any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for amplifying government service text data based on data enhancement technology described in any one of claims 1 to 6 is implemented.

Images