Method for quickly identifying large number of repeated work orders

Abstract

The invention relates to a natural language data processing technology, in particular to a method for quickly identifying a large number of repeated work orders. The identification method comprises the following steps: preprocessing public opinion data; updating public opinion pool data according to the public opinion pool data volume threshold; word embedding operation: converting the text into a vector; updating the public opinion pool; performing public opinion semantic deduplication; the work order interception and continuous order sending coupling method suitable for the field of public opinion monitoring is constructed for public opinion work order emergency disposal, and the problem that specific public opinion events interfere with normal public opinion work order generation and work order research and judgment during public opinion fermentation generally faced by current public opinion monitoring can be solved. According to the method, a vector database, a public opinion pool, a time window and other modules are added on the basis of data preprocessing and word embedding, and the effects that work orders are distributed continuously and discontinuously for hot events and few work orders are arranged for non-hot events are achieved.

Description

Technical Field

[0001] The present invention relates to natural language data processing technology, and in particular to a method for quickly identifying a large number of repeated work orders. Background Art

[0002] In the Internet age, public opinion events can easily ferment quickly on the Internet and have a negative impact on corporate brands and images. Rapid discovery and accurate identification of public opinion information can help companies adjust their marketing and problem-solving strategies, improve customer service methods, and enhance their service capabilities and quality in the early stages of public opinion fermentation.

[0003] When a public opinion event breaks out, a large number of similar or identical articles are reprinted or forwarded, causing the same public opinion events to appear in large numbers on the Internet. In traditional public opinion monitoring, each piece of public opinion will generate a public opinion work order for manual tracking of the development trend of public opinion. However, when public opinion breaks out, a large number of similar public opinions emerge, resulting in a large number of similar public opinion work orders in the system, which makes it impossible for manual processing to be carried out in a timely manner. At the same time, it will also overwhelm the unfermented public opinion information, affecting the normal development and handling of public opinion work orders. According to the requirements of public opinion business, when public opinion breaks out, the fermentation trend of public opinion events should be continuously tracked, and at the same time, a large number of duplicate work orders generated for the same event at the same time should be reduced as much as possible. Therefore, there is a contradiction between continuous dispatching of public opinion events and intermittent dispatching of public opinion orders.

[0004] Explanation of relevant terms:

[0005] Public opinion work order: A work order generated by public opinion information, which is used for enterprise public opinion tracking and public opinion handling. Usually, a public opinion work order will be generated for each piece of public opinion information found on the Internet.

[0006] Public opinion analysis: After the system generates a work order by discovering a public opinion event through an artificial intelligence model or rules, manual analysis will be conducted to determine whether the public opinion falls within the scope of public opinion that the company is concerned about. Public opinion events that fall within the scope of public opinion will be further tracked and handled.

[0007] In order to solve the problem that a large number of duplicate work orders appear when a public opinion event breaks out, which interferes with normal public opinion research and judgment, the present invention proposes a coupling mechanism of continuous dispatching and intermittent dispatching for public opinion events, so as to solve the problems of tracking hot events and avoiding duplicate dispatching. Summary of the invention

[0008] The present invention aims to solve the long-standing problem of dealing with a large number of repeated work orders during public opinion events. It adopts a dynamic update mechanism with a time window and public opinion pool, uses semantic similarity to identify repeated public opinion data in a short period of time, and uses three layers of restrictions to achieve continuous dispatch and intermittent dispatch coupling. The specific technical solution is as follows:

[0009] A method for quickly identifying a large number of duplicate work orders includes the following processes:

[0010] S100: Preprocessing of public opinion data; specifically, removing semantically irrelevant tags, web page tags, stop words, punctuation marks, and links from the text, and using a vector database to remove duplicate data to reduce the impact of dirty data models;

[0011] S200: Update the public opinion pool data according to the data volume threshold of the public opinion pool, and update the data according to the first-in-first-out principle, so as to quickly "squeeze out" the expired data in the public opinion pool when the public opinion breaks out; the public opinion pool is a public opinion data table, including sequence number, text content, and time; the public opinion pool data is arranged in chronological order;

[0012] S300: word embedding operation. First, Jieba is used to segment the text. After segmentation, Word2Vec is used to complete word embedding and convert the text into a vector.

[0013] S3400: Use the time window setting to dynamically update the data in the public opinion pool, delete expired data, and use the time window threshold to filter out data with a similarity greater than the threshold to update the public opinion pool;

[0014] S500: semantic deduplication of public opinion; specifically including: after S300 obtains the word embedding vectors of the public opinion data in the public opinion pool and the newly added public opinion data, calculates the vector similarity between the newly added public opinion data and the public opinion data in the public opinion pool, uses the similarity threshold to filter out the data with similarity less than the threshold, and completes the update of the public opinion pool data.

[0015] Preferably, in S100, the langdetect tool is used to filter out web page tags, such as 、 <script>、等标签；

[0016] 使用jieba分词工具，添加停用词库，在分词的同时去掉停用词；

[0017] 使用re工具建立正则模式进行去除网页链接和标点符号。

[0018] 优选的，所述S200中，如果舆情池数据量大于阈值，则删除舆情池中最后多条数据，重复检查舆情池数据量直到舆情池数据量满足阈值。

[0019] 优选的，所述S300具体过程如下：

[0020] 对文本使用jieba进行分词，分词后结果使用Word2Vec模型完成词嵌入，将文本转换为向量；Word2vec模型词嵌入操作时使用skip-gram算法，它可以根据给定的词预测上下文单词的概率；对于文本w1,w2,…,wt该模型的目标函数为：

[0021]

[0022] 通过word2vec模型，可以得到当前文本的词嵌入向量，从而为后续步骤的语义级分析提供条件。

[0023] 优选的，所述S500中相似度算法采用余弦相似度，该算法可以很好度量向量之间的方向相似性，并且不依赖于向量长度，计算公式如下：

[0024]

[0025] A、B是两个文本向量，计算得到的相似度值介于[-1,1]；

[0026] 通过上式计算得到当前文本向量与舆情池中所有向量的余弦相似度，查找是否存在相似度大于相似度阈值的数据，如果有则表示当前文本与最近的历史工单重复，属于同一事件，此时不生成工单。

[0027] 本发明针对舆情工单应急处置构建适合舆情监控领域的工单拦截和持续派单耦合方法，可以解决目前舆情监控普遍面临的舆情发酵时特定舆情事件干扰正常舆情工单生成和工单研判问题。该方法在数据预处理、词嵌入的基础上增加了向量数据库、舆情池、时间窗等模块，实现热点事件持续间断性派发工单的，非热点事件少排工单的效果。附图说明

[0028] 图1是本发明的大量重复工单快速识别方法流程示意图。

[0029] 图2是本发明工单拦截和耦合派单耦合机制。

[0030] 图3是本发明实施例中识别方法各阶段目标。

[0031] 图4是本发明实施例中清洗之后的数据样例。

[0032] 图5是本发明实施例中数据样例数据量约束示意图。

[0033] 图6是本发明实施例中Word2vec模型词嵌入操作时使用skip-gram算法模型原理示意图。

[0034] 图7a是本发明实施例中数据样例舆情池数据向量化过程部分示意图，图7a的右部连接图7b的左部。

[0035] 图7b是本发明实施例中数据样例舆情池数据向量化过程部分示意图。

[0036] 图8是本发明实施例中数据样例时间窗约束示意图。

[0037] 图9是本发明实施例中数据样例相似度过滤过程示意图。

[0038] 图10是本发明实施例中数据样例最终舆情池示意图。具体实施方式

[0039] 以下结合附图1-图10对本发明技术方案做详细介绍。

[0040] 如图1所示，大量重复工单快速识别方法，过程如下：

[0041] S100：舆情数据预处理；舆情数据清洗模块负责去掉文本中的语义无关的标签、网页标签、停用词、标点符号、链接，同时利用向量数据库去除重复数据，减少脏数据模型的影响；

[0042] 其中，网页标签去除使用langdetect工具过滤掉网页标签，如、<script>、等标签；停用词去除使用jieba分词工具，添加停用词库，在分词的同时去掉停用词；网页链接和标点符号去除则使用re工具建立正则模式进行去除；清洗之后的数据如图4所示。

[0043] S200：根据舆情池数据量阈值更新舆情池数据；

[0044] 舆情池数据量维度更新：更新舆情池时根据舆情池数据量阈值，对舆情池数据按照由近到远顺序排序，如果舆情池数据量大于阈值，则删除舆情池中最后多条数据，重复检查舆情池数据量直到舆情池数据量满足阈值。如图5，数据约束结果中大于数据量阈值的数据将删除。

[0045] 舆情池将用于S400和S500的间断性派发工单和持续派发工单耦合模型分析。

[0046] S300：词嵌入操作；词嵌入操作，首先对文本使用jieba进行分词，分词后结果使用Word2Vec完成词嵌入，将文本转换为向量。Word2vec模型词嵌入操作时使用skip-gram算法，它可以根据给定的词预测上下文单词的概率。对于文本w1,w2,…,wt该模型的目标函数为：

[0047]

[0048] 如图6所示，该模型可以将目标词(也即中心词)通过隐层预测上下文单词的概率，从而学习到语义特征，经过大量数据训练之后可以学习到词向量的语义表示。

[0049] 通过word2vec模型，可以得到当前文本的词嵌入向量，从而为后续步骤的语义级分析提供条件。

[0050] 以样例数据为例，舆情池数据向量化过程如图7a、图7b所示。

[0051] S400：舆情判断时间窗机制；

[0052] 舆情池时间维度更新：舆情池数据按照时间排序之后，检查是否存在时间大于阈值的数据，如果大于时间阈值则直接移出舆情池。时间窗约束如图8所示，舆情池中过期数据将删除。

[0053] S500：舆情语义去重；S300得到舆情池中的舆情数据和新增舆情数据的词嵌入向量之后，计算新增舆情数据与舆情池中舆情数据的向量相似度，利用相似度阈值过滤掉相似度小于阈值的数据，完成舆情池数据更新。

[0054] 相似度算法采用余弦相似度，该算法可以很好度量向量之间的方向相似性，并且不依赖于向量长度，计算公式为：

[0055]

[0056] A、B是两个文本向量，计算得到的相似度值介于[-1,1]。

[0057] 通过上式计算得到当前文本向量与舆情池中所有向量的余弦相似度，查找是否存在相似度大于相似度阈值的数据，如果有则表示当前文本与最近的历史工单重复，属于同一事件，此时不生成工单。如9，得到相似度计算结果。

[0058] 如图3所示，经过S200、S400和S500实现舆情池数据量、相似度更新，从而实现突发舆情事件爆发时涌现大量数据触发这两个限制条件，从而自动过滤掉相似、重复工单，实现热点事件间断性多派发工单，而舆情事件未爆发时，数据量较少时，并且相似舆情数据较少，因此主要触发时间窗限制，实现非热点事件少派发工单。最终舆情池如图10所示。< / script>

Claims

1. A method for quickly identifying a large number of duplicate work orders, characterized in that: The process includes: S100: Preprocessing of public opinion data; specifically, removing semantically irrelevant tags, web page tags, stop words, punctuation marks, and links from the text, and using a vector database to remove duplicate data to reduce the impact of dirty data models; S200: Update the public opinion pool data according to the data volume threshold of the public opinion pool, and update the data according to the first-in-first-out principle to quickly "squeeze out" the expired data in the public opinion pool when the public opinion breaks out; the public opinion pool is a public opinion data table, including sequence number, text content, and time; the public opinion pool data is arranged in chronological order; S300: word embedding operation. First, Jieba is used to segment the text. After segmentation, Word2Vec is used to complete word embedding and convert the text into a vector. S300: Using the time window setting, dynamically update the data in the public opinion pool, delete the expired data, and use the time window threshold to filter out the data with a similarity greater than the threshold to achieve the update of the public opinion pool; S500: semantic deduplication of public opinion; specifically including: after S300 obtains the word embedding vectors of the public opinion data in the public opinion pool and the newly added public opinion data, calculates the vector similarity between the newly added public opinion data and the public opinion data in the public opinion pool, uses the similarity threshold to filter out the data with similarity less than the threshold, and completes the update of the public opinion pool data.

2. The method for quickly identifying a large number of duplicate work orders according to claim 1, characterized in that: In S100, the web page tags are filtered out using the langdetect tool; Use the Jieba word segmentation tool, add a stop word library, and remove stop words while segmenting; Use the re tool to create a regular pattern to remove web links and punctuation marks.

3. The method for quickly identifying a large number of duplicate work orders according to claim 1 is characterized in that: In S200, if the data volume in the public opinion pool is greater than a threshold, the last plurality of data in the public opinion pool is deleted, and the data volume in the public opinion pool is repeatedly checked until the data volume in the public opinion pool meets the threshold.

4. The method for quickly identifying a large number of duplicate work orders according to claim 1, characterized in that: The specific process of S300 is as follows: Jieba is used to segment the text. The result of segmentation is embedded using the Word2Vec model to convert the text into a vector. The skip-gram algorithm is used in the word embedding operation of the Word2vec model, which can predict the probability of context words based on a given word. For text w1, w2, …, w t The objective function of the model is: Through the word2vec model, the word embedding vector of the current text can be obtained.

5. The method for quickly identifying a large number of duplicate work orders according to claim 1, characterized in that: The similarity algorithm in S500 adopts cosine similarity, which can well measure the directional similarity between vectors and is independent of the vector length. The calculation formula is as follows: A and B are two text vectors. The calculated similarity value is between [-1,1]. The above formula is used to calculate the cosine similarity between the current text vector and all vectors in the public opinion pool, and then check whether there is data with a similarity greater than the similarity threshold. If so, it means that the current text is repeated in the most recent historical work order and belongs to the same event. In this case, no work order is generated.

Images