A method for removing redundant information of public opinion information

Abstract

The application discloses a kind of redundancy information removal methods of public opinion information, it is related to language processing technical field, the application includes for public opinion information Construction dataset;Data set constructs positive and negative examples, carries out sample enhancement;Construct pre-training model BERT, text is encoded based on pre-training model BERT, whether the similar output text is passed through average pooling layer;Positive example sample and negative example sample are input into pre-training model BERT, and pre-training model BERT is unsupervised training;The text to be identified is input into the pre-training model BERT of training, and the semantic similarity of text sentence vector is identified, if synonymous, this information is discarded, if different in meaning, the information is written into database, and redundancy information is removed;Comprehensively, the method is analyzed to the semantic similarity of the obtained public opinion data, removes the same news data, and is conducive to the efficient execution of the whole public opinion monitoring work.

Description

Technical Field

[0001] This invention relates to the field of language processing technology, specifically to a method for removing redundant information from public opinion information. Background Technology

[0002] Text similarity assessment is a fundamental problem in natural language processing. Its basic task is to determine whether two pieces of text are similar. For example, "how to place the license plate on the front of a car" and "how to install the front license plate" are similar, while "how to install the rear license plate" only has a certain correlation. Unsupervised methods have attracted widespread attention for language similarity assessment because they do not require large-scale labeled data. Furthermore, contrastive learning has achieved excellent results in learning unsupervised sentence vectors, attracting the participation of many scholars engaged in text similarity research.

[0003] Contrastive learning is an unsupervised learning method used to learn general features of a dataset by having the model learn which data points are similar or different without labels. For example, given three sentences: sentence 1, sentence 2, and sentence 3, we want to distinguish their emotional categories. With supervised learning, we would distinguish sentence 1 as having positive emotion, sentence 2 as having negative emotion, and sentence 3 as having positive emotion. However, for unsupervised contrastive learning, we only want to identify sentences 1 and 3 as belonging to the same category, and sentences 2 and 1 as belonging to different categories. That is, contrastive learning does not need to know the true category of each sentence, but only needs to determine which sentences are similar and which are dissimilar. From the perspective of sentence vectors, the goal of contrastive learning is to group all similar sentence vectors into adjacent regions, and dissimilar sentence vectors into non-adjacent regions.

[0004] Chinese patent CN110990525A discloses a method for extracting public opinion information and generating a knowledge base based on natural language processing, including the following steps: 1. Text preprocessing; 2. Named entity recognition, including: recognizing company and organization names and personal names, using a neural network-based method to complete named entity recognition; 3. Relation extraction, using a feature layer + GRU + Attention to extract six types of relationships in the financial field; 4. Entity linking; using the Jaro Winkler distance method, by calculating the distance between linked entities and target entities to determine whether they are the same entity, achieving entity disambiguation.

[0005] However, online public opinion is generally expressed as a short sentence. The above method, which only disambiguates the entities within the sentence without considering the overall meaning of the sentence, suffers from overgeneralization and poor results. Furthermore, entity identification is not 100% accurate. If errors occur during named entity recognition using the above method, they will accumulate in subsequent entity linking disambiguation, greatly affecting the final accuracy. In summary, the above method suffers from poor accuracy. Summary of the Invention

[0006] To address the problem of poor accuracy in text similarity recognition in existing technologies, this invention provides a method for removing redundant information from public opinion data, comprising the following steps:

[0007] Collect public opinion information, construct a dataset based on the public opinion information, and put the dataset into the public opinion database;

[0008] Create positive and negative examples in the dataset to perform sample augmentation;

[0009] Build a pre-trained model BERT, encode text based on the pre-trained model BERT, set dropout to prevent overfitting, perform average pooling on the last hidden layer of BERT, and output whether the texts are similar.

[0010] Positive and negative samples are input into the pre-trained model BERT, and unsupervised training is performed on the pre-trained model BERT. The loss function of the pre-trained model BERT is set. As backpropagation and gradient descent are performed during the unsupervised training of the pre-trained model BERT, an appropriate learning rate is set so that the model converges to a local minimum at an appropriate time. Under the action of the optimizer of the pre-trained model BERT, the loss value of the training loss function becomes smaller and smaller until the model converges.

[0011] The text to be identified is input into the pre-trained BERT model. The semantic similarity is identified through the text sentence vectors to determine whether it is synonymous with the information in the public opinion database. If it is synonymous, the information is discarded; if it is not synonymous, the information is written into the database to remove redundant information.

[0012] Furthermore, the collection of public opinion information and the construction of a dataset based on this information specifically include:

[0013] We use web crawlers to collect public opinion information, and then perform data structure and format judgment, data cleaning, removal of special characters and empty information, and type conversion on the obtained public opinion information to compile the dataset needed for training.

[0014] Furthermore, the process of constructing positive and negative examples for the dataset and performing sample augmentation specifically includes:

[0015] In a batch of samples in the dataset, each sample i is copied once. A common transformation is performed on the copy of i to obtain one positive example. A queue model φ is then constructed. q This results in queue_size*batch_size negative instances.

[0016] Furthermore, the process of performing a uniform transformation on the copy of i to obtain a positive example specifically includes:

[0017] For the specific scenario of public opinion monitoring, a synonym model for public opinion monitoring is trained using the word2vec model based on the crawled public opinion information; a replacement rate r1 is specified, and r1*n words in the original sentence of length n are randomly replaced with synonyms to transform the sentence without changing the original meaning;

[0018] For an original sentence of length n, a random repetition rate r2 is adopted. Then, n*r2 characters are randomly selected from the original sentence and copied to achieve the purpose of making the original sentence's positive example and the original sentence length unequal, thereby improving the model's performance in recognizing the semantic similarity of sentences of different lengths.

[0019] Furthermore, the unsupervised training of the pre-trained model BERT specifically includes:

[0020] Constructing a queue model φ q Using the queue model φ q Generate negative examples and maintain a queue stored on the CPU to hold the encoding of negative examples;

[0021] During unsupervised training of the pre-trained BERT model, the queue model maintains a negative example queue of length queue_size * batch_size, without performing forward or backward propagation, and directly participates in the loss calculation; φ m This represents the forward propagation, back propagation, and loss calculation for each batch of data, and φ is updated using the following formula. q The model, where θ is a constant, aims to ensure consistency between the preceding and following codes in the queue.

[0022] φ q ←θφ q +(1-θ)φ m .

[0023] Furthermore, the pre-trained model BERT is constructed, wherein the loss function of the pre-trained model BERT is:

[0024]

[0025] In the formula, sim(v i ,v' i) is the formula for calculating cosine similarity, where M is the size of a batch, and v' i It represents v i The agreed-upon transformed sample, v' j It is the set of all samples that undergo the same transformation, v' q This represents the negative examples in the queue. τ controls the distribution of the softmax function. The smaller the value, the larger the distance between positive and negative examples. The larger the value, the smoother the distribution.

[0026] In this function, the numerator represents the distance between positive pairs, and the denominator represents the sum of the distances between the original sample, the transformed sample, and different original samples in the batch. The smaller the loss of the loss function, the closer the positive pairs are, and the farther the negative pairs are.

[0027] Furthermore, the pre-trained model BERT is constructed, wherein the metrics used in the pre-trained model BERT to identify sentence semantic similarity are Alignment and Uniformity;

[0028] Here, Alignment represents the degree of alignment between semantically identical sample pairs:

[0029]

[0030] f(x) is an encoder that converts samples into embedded vectors. The smaller the index value, the higher the semantic similarity.

[0031] The Uniformity metric represents the uniformity of sentence encoding, where Pdata represents the data distribution, and the metric indicates that the vectors are evenly distributed in the semantic space.

[0032]

[0033] Compared with existing technologies, the method for removing redundant information from public opinion information provided by this invention has the following advantages:

[0034] Traditional deep learning-based text similarity algorithms require extensive data annotation, consuming significant human and material resources and introducing subjective errors from annotators. This invention employs an unsupervised method, eliminating the need for labeled data and avoiding subjective errors, thus improving recognition accuracy. Compared to existing technologies that construct positive samples by copying existing samples, this invention proposes a consensus transformation method to construct positive examples, significantly enhancing recognition performance. Furthermore, this invention proposes a new paradigm for improving the efficiency of public opinion monitoring, using a model with excellent recognition performance to filter and remove redundant public opinion information, thereby improving the overall efficiency of public opinion monitoring. Attached Figure Description

[0035] Figure 1A flowchart of the method for removing redundant information from public opinion information provided by the present invention. Detailed Implementation

[0036] The following is in conjunction with the appendix Figure 1 The following describes specific embodiments of the present invention in further detail. These embodiments are merely for illustrating the technical solutions of the present invention more clearly and should not be construed as limiting the scope of protection of the present invention.

[0037] Example 1: This invention proposes a method for removing duplicate information from public opinion information; when constructing positive samples, word2vec is used to randomly replace the words in the positive examples with synonyms. Furthermore, since deleting words and adding new words will interfere with the original meaning of the sentence, existing words in the positive examples are randomly added to enhance the model's ability to handle the interference of sentences of different lengths on the judgment of semantic similarity.

[0038] The present invention proposes a method for removing redundant information from public opinion information, comprising the following steps:

[0039] We crawled and collected public opinion information from platforms such as Weibo, Zhihu, Baidu Tieba, and Toutiao, built a dataset based on the public opinion information, and put the dataset into a public opinion database.

[0040] In a batch of samples (batch_size) in the dataset, each sample i is copied once. A common transformation is performed on the copy of i to obtain one positive example. To improve performance and increase the number of negative examples, a queue model φ is constructed. q This results in queue_size * batch_size negative instances;

[0041] Construct a pre-trained model BERT, which includes an optimizer. Encode text based on the pre-trained model BERT, set dropout to prevent overfitting, perform average pooling on the last hidden layer of BERT, and finally output whether the texts are similar.

[0042] While theoretically increasing the number of negative examples can enhance the model's judgment performance, the common strategy of using other samples in the batch as negative examples requires increasing the batch size. This leads to a need for graphics cards with more GPU memory, and the performance deteriorates as the batch size increases. This invention employs a queue model φ. q The method is used to generate negative examples, maintaining a queue stored on the CPU to hold the encodings of negative examples. During training, the queue model maintains a queue of negative examples of length `queue_size * batch_size`, without performing forward or backward propagation, and directly participates in the loss calculation. mThis represents the forward propagation, back propagation, and loss calculation for each batch of data, and φ is updated using the following formula. q The model, θ, is a number close to 1, designed to ensure consistency between the preceding and following codes in the queue. (φ) q and φ q These are all BERT models, and they are copies.

[0043] φ q ←θφ q +(1-θ)φ m ;

[0044] Positive and negative samples are input into the pre-trained BERT model for unsupervised training. The loss function of the pre-trained BERT model is set. As backpropagation and gradient descent are performed during unsupervised training, an appropriate learning rate is set so that the model converges to a local minimum at an appropriate time. Under the action of the optimizer, the loss value of the training loss function becomes smaller and smaller until the loss tends to be stable and very low, the model converges, and achieves good performance in judging semantic similarity.

[0045] The text to be identified is input into the pre-trained BERT model. The semantic similarity is identified through the text sentence vectors to determine whether it is synonymous with the information in the public opinion database. If it is synonymous, the information is discarded; if it is not synonymous, the information is written into the database to remove redundant information.

[0046] In this embodiment, a common transformation is performed on the copy of i to obtain a positive example, specifically including:

[0047] For the specific scenario of public opinion monitoring, a synonym model for public opinion monitoring is trained using the word2vec model based on crawled public opinion information. A replacement rate r1 is specified, and r1*n words (at least one replacement) in the original sentence of length n are randomly replaced with synonyms, transforming the sentence without changing its original meaning.

[0048] For an original sentence of length n, a random repetition rate r2 is adopted. An n*r2 character is randomly selected from the original sentence (the minimum is 2, and the maximum is less than the sentence length. For example, copying "I love you" into "I I love you you" is the longest, and repeating "I" three times is meaningless) and then copying this character. This is to achieve the purpose of the positive example and the original sentence length being unequal, thereby improving the model's performance in recognizing the semantic similarity of sentences of different lengths.

[0049] In this embodiment, public opinion information is collected, and a dataset is constructed based on the public opinion information, specifically including:

[0050] We use web crawlers to collect public opinion information, and then perform data structure and format judgment, data cleaning, removal of special characters and empty information, and type conversion on the obtained public opinion information to compile the dataset needed for training.

[0051] In this embodiment, a pre-trained model BERT is constructed, wherein the loss function of the pre-trained model BERT is:

[0052]

[0053] In the formula, sim(v i ,v' i ) is the formula for calculating cosine similarity, where M is the size of a batch, and v' i It represents v i The agreed-upon transformed sample, v' j It is the set of all samples that undergo the same transformation, v' q This represents the negative examples in the queue. τ controls the distribution of the softmax function. The smaller the value, the larger the distance between positive and negative examples. The larger the value, the smoother the distribution.

[0054] In this model, the numerator of the loss function represents the distance between positive pairs, and the denominator represents the sum of the distances between the original sample, the transformed sample, and different original samples in the entire dataset. The smaller the loss of the loss function, the closer the distance between positive pairs and the farther the distance between negative pairs. With backpropagation and gradient descent, the loss value trained by the optimizer will become smaller and smaller until the loss tends to stabilize and decreases to a very low level, at which point the model converges and achieves good performance in judging semantic similarity.

[0055] In this embodiment, a pre-trained model BERT is constructed, wherein the metrics used in the pre-trained model BERT to identify sentence semantic similarity are Alignment and Uniformity;

[0056] Here, Alignment represents the degree of alignment between semantically identical sample pairs:

[0057]

[0058] f(x) is an encoder that converts samples into embedded vectors. The smaller the index value, the higher the semantic similarity.

[0059] The Uniformity metric represents the uniformity of sentence encoding, where Pdata represents the data distribution, and the metric indicates that the vectors are evenly distributed in the semantic space.

[0060]

[0061] In summary, compared with existing technologies, the method for removing redundant information from public opinion data provided by this invention has the following beneficial effects:

[0062] 1. Traditional deep learning-based text similarity algorithms require a large amount of data annotation, which not only requires a lot of manpower and resources, but also introduces subjective errors from the annotators. This invention uses an unsupervised method, which does not require annotated data, avoids the introduction of subjective errors from the annotators, and helps to improve recognition accuracy.

[0063] 2. Compared with existing technologies that construct positive samples by copying samples, this invention proposes a method of identity transformation to construct positive examples, which greatly improves the recognition effect.

[0064] 3. This invention proposes a new paradigm for improving the efficiency of public opinion monitoring. It uses a model with excellent recognition performance to filter out redundant public opinion information, thereby improving the overall efficiency of public opinion monitoring.

[0065] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0066] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for removing redundant information from public opinion information, characterized in that, Includes the following steps: Collect public opinion information, construct a dataset based on the public opinion information, and put the dataset into the public opinion database; Create positive and negative examples in the dataset to perform sample augmentation; Build a pre-trained model BERT, encode text based on the pre-trained model BERT, set dropout to prevent overfitting, perform average pooling on the last hidden layer of BERT, and output whether the texts are similar. Positive and negative samples are input into the pre-trained model BERT to perform unsupervised training on the pre-trained model BERT. Set the loss function of the pre-trained model BERT. As the pre-trained model BERT is trained in an unsupervised manner, backpropagation and gradient descent are performed. Set an appropriate learning rate so that the model converges to a local minimum at an appropriate time. Under the action of the optimizer of the pre-trained model BERT, the loss value of the training loss function becomes smaller and smaller until the model converges. The text to be identified is input into the pre-trained BERT model. The semantic similarity is identified through the text sentence vectors to determine whether it is synonymous with the information in the public opinion database. If it is synonymous, the information is discarded. If it is not synonymous, the information is written into the database to remove redundant information. The process of constructing positive and negative examples for the dataset and performing sample augmentation specifically includes: In a batch of samples in the dataset, each sample i is copied once. A common transformation is performed on the copy of i to obtain one positive example. A queue model φ is then constructed. q This results in queue_size * batch_size negative instances; The process of performing a uniform transformation on the copy of i to obtain a positive example specifically includes: For the specific scenario of public opinion monitoring, a synonym model for public opinion monitoring is trained using the word2vec model based on the crawled public opinion information; a replacement rate r1 is specified, and r1*n words in the original sentence of length n are randomly replaced with synonyms to transform the sentence without changing the original meaning; For an original sentence of length n, a random repetition rate r2 is adopted. Then, n*r2 characters are randomly selected from the original sentence and copied to achieve the purpose of making the original sentence's positive example and the original sentence length unequal, thereby improving the model's performance in recognizing the semantic similarity of sentences of different lengths. The unsupervised training of the pre-trained model BERT specifically includes: Constructing a queue model φ q Using the queue model φ q Generate negative examples and maintain a queue stored on the CPU to hold the encoding of negative examples; During unsupervised training of the pre-trained BERT model, the queue model maintains a negative example queue of length queue_size * batch_size, without performing forward or backward propagation, and directly participates in the loss calculation; φ m This represents the forward propagation, back propagation, and loss calculation for each batch of data, and φ is updated using the following formula. q The model, where θ is a constant, aims to ensure consistency between the preceding and following codes in the queue. 。 2. The method for removing redundant information from public opinion information as described in claim 1, characterized in that, The collection of public opinion information, and the construction of a dataset based on the public opinion information, specifically includes: We use web crawlers to collect public opinion information, and then perform data structure and format judgment, data cleaning, removal of special characters and empty information, and type conversion on the obtained public opinion information to compile the dataset needed for training.

3. The method for removing redundant information from public opinion information as described in claim 1, characterized in that, The pre-trained model BERT is constructed, wherein the loss function of the pre-trained model BERT is: ； In the formula, Here is the formula for calculating cosine similarity. M It is the size of a batch. It represents Agree on the transformation of samples, It is the set of all samples that have undergone the same transformation. This represents the negative examples in the queue. It controls the distribution of softmax; the smaller the value, the larger the spacing between positive and negative examples, and the larger the value, the smoother the distribution. In this function, the numerator represents the distance between positive pairs, and the denominator represents the sum of the distances between the original sample, the transformed sample, and different original samples in the batch. The smaller the loss of the loss function, the closer the positive pairs are, and the farther the negative pairs are.

4. The method for removing redundant information from public opinion information as described in claim 1, characterized in that, The pre-trained model BERT is constructed, wherein the metrics used in the pre-trained model BERT to identify sentence semantic similarity are Alignment and Uniformity; Here, Alignment represents the degree of alignment between semantically identical sample pairs: ； f(x) is an encoder that converts samples into embedded vectors. The smaller the index value, the higher the semantic similarity. The uniformity metric represents the evenness of sentence coding, where Indicators represent data distribution, where the metrics represent vectors evenly distributed across the semantic space. 。

Images