Event clustering system and method based on large language models

Abstract

The application belongs to the field of computer science and artificial intelligence, and particularly relates to an event clustering system and method based on a large language model, the method comprising: obtaining news text data and preprocessing; inputting the preprocessed news text data into a trained large language model, and outputting event tri-element reasoning results corresponding to the news text data; inputting an occurrence event into a classification model, and outputting a classification category; inputting the occurrence event into an embedding model, and outputting an embedding vector; if the occurrence time and the occurrence place of different news text data under the same classification category are the same, the different news text data are clustered into the same occurrence event; if the occurrence time and the occurrence place are different, the similarity of the embedding vectors corresponding to the different news text data is calculated, and whether the different news text data are clustered into the same occurrence event is determined according to the similarity. The event tri-element reasoning results of the news text data are outputted through the large language model, the accuracy of event clustering is improved, and the cost of manual intervention is greatly reduced.

Description

Technical Field

[0001] This application belongs to the fields of computer science and artificial intelligence, and specifically relates to an event clustering system and method based on a large language model. Background Art

[0002] Event clustering is the process of dividing massive news text data sets into different clusters based on similarity or difference, ensuring that news text data in the same cluster are as similar as possible, and news text data in different clusters are as different as possible. This method can group texts describing the same event into one group, where the same event generally refers to the same time, place, entity, and accompanying results described in multiple news texts.

[0003] Traditional event clustering methods typically rely on manual design based on feature engineering and traditional small models to extract features or information, and then use these features for clustering. This approach is limited by its ability to extract features and understand text, resulting in relatively low accuracy. Furthermore, these methods require significant manual intervention, including the infusion of specialized domain knowledge and manual parameter tuning, which increases complexity and cost. Summary of the Invention

[0004] This invention aims to optimize traditional event clustering methods by introducing a large language model. By using a large language model as the primary technical approach and automatically learning text representations through deep learning, this approach avoids the manual intervention required by traditional feature engineering and improves adaptability to complex contexts and specialized terminology. The pre-training capabilities of the large language model further enhance understanding of text, providing more accurate and comprehensive semantic information for event clustering. This optimization solution aims to reduce reliance on manual intervention, improve accuracy, and better adapt to different fields and contexts, making event clustering methods more intelligent and efficient.

[0005] The present invention provides an event clustering system and method based on a large language model to achieve the above-mentioned purpose, specifically comprising:

[0006] In a first aspect of the present invention, the present invention provides an event clustering method based on a large language model, the method comprising:

[0007] Obtain one or more news text data from different news platforms and preprocess the news text data;

[0008] Input the pre-processed news text data into the trained large language model, and output the three-element reasoning results of the event corresponding to the news text data, including the time, place and event of occurrence;

[0009] Inputting the occurrence event corresponding to the news text data into a classification model, and outputting the classification category corresponding to the news text data;

[0010] Inputting the occurrence event corresponding to the news text data into the embedding model, and outputting the embedding vector corresponding to the news text data;

[0011] The news text data is searched based on its classification category, occurrence time and occurrence location. If different news text data have the same occurrence time and occurrence location under the same classification category, they are clustered into the same occurrence event. If the occurrence time and occurrence location are different, the similarity of the embedding vectors corresponding to the different news text data is calculated, and whether they are clustered into the same occurrence event is determined based on the similarity.

[0012] In a second aspect of the present invention, the present invention further provides an event clustering system based on a large language model, the system comprising:

[0013] News text crawling module, used to obtain one or more news text data from different news platforms;

[0014] A news text preprocessing module, used for preprocessing the news text data;

[0015] The news text reasoning module is used to input the pre-processed news text data into the trained large language model and output the inference results of the three elements of the event corresponding to the news text data, including the time, place and event of occurrence;

[0016] A news text classification module, configured to input the occurrence event corresponding to the news text data into a classification model and output the classification category corresponding to the news text data;

[0017] A news text embedding module, configured to input the occurrence event corresponding to the news text data into an embedding model and output an embedding vector corresponding to the news text data;

[0018] The news text event clustering module searches according to the classification category, occurrence time and location of the news text data. If the occurrence time and location of different news text data under the same classification category are the same, they are clustered as the same event. If the occurrence time and location are different, the similarity of the embedding vectors corresponding to the different news text data is calculated, and whether they are clustered as the same event is determined based on the similarity.

[0019] Compared with the prior art, the advantages of the present invention are:

[0020] The present invention outputs the event three-element reasoning results of news text data through a large language model, thereby improving the accuracy of event clustering and greatly reducing the cost of manual intervention. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a flowchart of an event clustering method of an embodiment of the present application;

[0022] Figure 2 is a flowchart of a large language model training of an embodiment of the present application;

[0023] Figure 3 is a flowchart of a similarity determination of an embodiment of the present application;

[0024] Figure 4 is a structural diagram of an event clustering system of an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0026] Figure 1 is a flowchart of an event clustering method of an embodiment of the present application, as shown in Figure 1 the method comprises:

[0027] 101, obtaining one or more news text data from different news platforms, and preprocessing the news text data;

[0028] In an embodiment of the present application, multiple news text data published by news sources can be obtained through network crawler technology or API interface, etc., for example, from news portals, Baidu net, Sina net, etc. The news text data is the text that needs to be clustered, which can be various forms of text such as news reports. Based on the news text data, the event information of the news text data can be determined in a certain way. Different news text data may describe the same or similar events.

[0029] In an embodiment of the present invention, considering the real-time nature of news text data, and in order to avoid repeated capture of the same news text data from the same news source, this embodiment adopts hash deduplication to ensure that the acquired news text data is unique. According to the actual situation, a suitable storage solution can be selected, such as MySQL, Oracle and other databases, and its hash value can be calculated to ensure that each piece of data has a unique hash value. When new data arrives, it is only necessary to compare the hash value to determine whether the data is repeated. Specifically, the pre-processing of the news text data includes hash deduplication of news text data from different news platforms. If the hash value of the acquired current news text data already exists, the current news text data is deduplicated. If the hash value of the acquired current news text data does not exist, the current news text data is retained.

[0030] In a preferred embodiment of the present invention, regularization and other methods are used to remove abnormal text that may be included in the news collection. Levenshtein distance is used to calculate the similarity between two pieces of data. Based on empirical data, if the similarity is greater than 0.95, it is considered plagiarism of the same news item, and only the earliest or most authoritative source is retained. Duplicate data and general data can be identified as different data to quickly remove duplicate data.

[0031] 102. Input the pre-processed news text data into the trained large language model, and output the inference results of the three elements of the event corresponding to the news text data, including the time, location and event of occurrence;

[0032] In this embodiment, the powerful reasoning ability of the large language model is utilized to infer the three elements of the event of the news text data, such as Figure 2 As shown in Figure 2, the training process of the large language model includes:

[0033] 201. Build a large language model to generate the inference prompt template of the three elements of the event;

[0034] In the embodiment of the present invention, a large model is constructed to generate event three-element training and reasoning prompt templates, and reasonable prompt templates are written. Through continuous testing, the best effect is achieved, reducing the manual review work in the later stage. The prompt can be:

[0035] You can answer the following questions about the following:

[0036] “Fill in specific news content”

[0037] Please summarize the time, location, and events of the above content and return them in JSON format;

[0038] For example:

[0039] {“occurrence time”:2020-10-10 12:00:00,“occurrence location”:A province B city C district,“occurrence event”:occurrence of 2.2 magnitude earthquake, focal depth 5 kilometers}

[0040] 202、According to the inference prompt template, training data is constructed; the training data includes news title, news text and title generation event three elements;

[0041] In this embodiment, in order to generate training data, it can be realized in an automated way, for example, through Chatgpt and other large models to generate data similar to the prompt template in batches. The generated template data is expanded through keyword replacement and other methods. The quality of the generated data is audited by combining programs with manual assistance to ensure the reliability of the data.

[0042] In the preferred embodiment of the present application, the data similar to the prompt template generated by the large model in batches can be sampled and checked. This embodiment uses a random sampling method to take 5% of the data for detection. If the generated data meets the prompt template format, it can be used for training.

[0043] 203、The training data is input into a large language model, and the large language model is fine-tuned by low-rank adaptation. If the inference result output by understanding the news title meets the title generation event three elements, the event three elements are directly output. Otherwise, the large language model is repeatedly called. If the title generation event three elements are still not met after exceeding the preset number of calls, the inference result is output by understanding the news text. If the inference result output by understanding the news text meets the title generation event three elements, the event three elements are directly output. Otherwise, the large language model is repeatedly called. If the event three elements are still not met after exceeding the preset number of calls, the news text is treated as invalid data, and the next training data is trained.

[0044] In the embodiment of the present application, the large language model can be lamma, chatglm and other models. Based on the training data determined in the above manner, the large language model is trained and fine-tuned by a parameter-efficient fine-tuning technique, i.e. low-rank adaptation (LORA) fine-tuning technique.

[0045] In the embodiment of the application, the training data actually includes news titles, news content and event three elements, based on which, first, the news title needs to be understood, the event three elements in the news title are understood, if the title event three elements are contained, they can be returned through json, if not, the maximum number of times of calling the large language model is 3, the preset number of times of calling here is 3, until the event three elements are met. Otherwise, the news body is understood, the event three elements in the news body are understood, and the reasoning process of the news title is similar, if the title event three elements are contained, they can be returned through json, if not, the maximum number of times of calling the large language model is 3, otherwise, the current logic is terminated, and the data is regarded as invalid. The title event three element content generated by the news body needs to be concise and cannot contain any other content, which can be audited by an artificial manner or judged by the large language model.

[0046] 103. inputting the occurrence event corresponding to the news text data into a classification model to output a classification category corresponding to the news text data;

[0047] In the embodiment of the application, in order to reduce the number of similar texts in event clustering and reduce the data operation amount, the embodiment only considers the occurrence event and does not consider the occurrence event and the occurrence place, the occurrence event is input into the classification model to output the corresponding classification category, for example, the classification category is entertainment news, and domestic entertainment news can be further subdivided, the irrelevant news text data can be excluded as much as possible through the classification category. The classification model can be a common text classification model such as bert, Robert, and the application does not make specific limitation.

[0048] 104. inputting the occurrence event corresponding to the news text data into an embedding model to output an embedding vector corresponding to the news text data;

[0049] In the embodiment of the application, in order to facilitate the calculation of the similarity of different news text data, the occurrence event needs to be input into the embedding model, the embedding model refers to an embedding model, which can convert text into a high-dimensional vector model, for example, a model such as text2vec-base-chinese, and the embedding vector of the news text data can be extracted through the embedding model, which is convenient for subsequent processing.

[0050] 105. searching according to the classification category, occurrence time and occurrence place of the news text data, if the occurrence time and occurrence place of different news text data are the same under the same classification category, the occurrence event is clustered as the same, if the occurrence time and occurrence place are different, the similarity of the embedding vectors corresponding to the different news text data is calculated, and whether the occurrence event is clustered as the same is judged according to the similarity.

[0051] In the embodiment of the present application, considering that the obtained news text data has real-time nature, there may be a connection between the current news text data and the historical news text data, the embodiment determines the current news text data and the historical news text data by regular matching through a search method such as regular matching, and can determine whether the current news text data and the historical news text data are of the same classification category. Assuming that the current news text data and the historical news text data are both of the entertainment category, it is still necessary to continue to determine whether the occurrence time and the occurrence location of the current news text data are the same as the occurrence time and the occurrence location of the historical news text data. If the occurrence time and the occurrence location are the same, for example, according to experience, an event occurring at the same time and in the same place can be regarded as the same event, and the specific time can be set according to experience, for example: 12:00:00 can retrieve the same hour range; then it is not necessary to determine whether the occurrence event is the same, which saves the processing time of the occurrence event. If the occurrence time and the occurrence location are different, the similarity of the embedding vectors corresponding to the different news text data is calculated, and whether to cluster the same occurrence event is determined according to the similarity.

[0052] Specifically, if the similarity of the embedding vectors of the current news text data and the recalled news text data exceeds the first threshold value, the occurrence event of the current news text data and the recalled news text data is clustered as the same occurrence event. If the similarity of the embedding vectors of the current news text data and the recalled news text data does not exceed the first threshold value but exceeds the second threshold value, the current news text data and each recalled news text data are input into a preset large language model, and an inference result of whether the current news text data and the recalled news text data are the same occurrence event is output. If the inference results of a preset number of recalled news text data are all the same occurrence event, the same occurrence event is clustered, otherwise, manual auditing is performed. If the similarity of the embedding vectors of the current news text data and the recalled news text data does not exceed the second threshold value, the current news text data is regarded as an independent event. The recalled news text data is the news text data in the recalled data set.

[0053] In the embodiment of the present application, the preset large language model can use the same model as the large language model trained in the embodiment of the present application, or the large language model can be fine-tuned by using text event comparison data to obtain the preset large language model. The text event comparison data can include each news text data and the corresponding label.

[0054] In the embodiment of the present application, the similarity distance can be calculated by using the Euclidean distance, the cosine distance, or a variety of mixed distances. The present application does not limit the similarity distance.

[0055] In the embodiment of the present application, the similarity mode of the current news text data and the embedding vectors of the recall news text data includes calculating the average similarity of the current news text data and the embedding vectors of all recall news text data of the same classification category in the recall data set, and performing reverse ordering on the average similarity of the classification category cluster; the average similarity of the classification category cluster with a high ranking is compared with the size of the first threshold value and the second threshold value.

[0056] For example, as shown in the table, assuming that the first threshold value is 0.95 and the second threshold value is 0.75, the cluster with the maximum similarity is selected according to the empirical value, and it is judged that the similarity greater than 0.95 is directly regarded as the same event. The maximum similarity value less than 0.75 is regarded as a new event and becomes an independent cluster. The whole data processing process ends. If the similarity is between the two, the recall data set 2 is returned (here, the subset data of the maximum cluster with 0.75 < similar < 0.95), and the empirical similarity threshold value needs to be set according to the actual scene and data experiment. Figure 3

[0057] In the embodiment of the present application, if the inference results of more than a preset number of recall news text data are the same event, clustering the same event includes selecting the embedding vectors of part of the recall news text data of the same classification category according to a preset proportion; the inference results of whether the current news text data and the part of the recall news text data are the same event are obtained respectively, if the inference results of more than a preset number of recall news text data are the same event, the classification categories of the current news text data and the part of the recall news text data are consistent, and the current news text data and the part of the recall news text data are clustered as the same event, otherwise, manual auditing is performed. It is assumed that the inference result output by the big data model is True or False, True represents that the recall news text data and the current news text data are the same event, and False represents that the recall news text data and the current news text data are the same event. According to the preset number, it is judged whether it is the same event, if the result ratio is lower than 1:4, it is determined whether it is the same event (for example, in the same cluster, 10 data, nine of which are determined to be the same event, and then the same event is regarded as the same event, and vice versa). If it is higher than 1:4, or even 1:1, it is specially marked, and then manual judgment intervention is performed. In this step, if a model with weak ability is selected, the model can also be fine-tuned by constructing related data to improve the recognition accuracy of the model. Through the mode of the embodiment of the present application, all news text data can be iteratively processed, so that the same event is clustered as the same event cluster, and different events are divided into different event clusters.

[0058] ​In the preferred embodiment of the present application, considering that if the similarity calculation is performed on the embedding vectors of all the recall news text data of the same classification category as the current news text data, a large amount of computing resources will be consumed, the embedding vectors of the current news text data and the recall news text data are projected into a coordinate system in the embodiment, and a part of the recall news text data is excluded by the distance from the origin of the coordinates to each recall news text data, that is, the recall news text data that is too far or too close to the origin of the coordinates is excluded, and only part of the recall news text data is retained, so as to avoid the calculation on all the recall news text data, and the similarity distance is calculated by the embedding vectors of the remaining recall text data and the embedding vector of the current news text data, and then the average of the similarity distance is taken as the similarity between the current news text data and the classification category cluster, and the similarity is used to determine whether the event of the current news text data and the classification is the same event.

[0059] As shown in Figure 4 , Figure 4 is the event clustering system structure diagram of the embodiment of the present application, an event clustering system based on a large language model, comprising:

[0060] A news text grabbing module is configured to obtain one or more news text data from different news platforms.

[0061] A news text preprocessing module is configured to preprocess the news text data.

[0062] A news text reasoning module is configured to input the preprocessed news text data into the trained large language model, and output the event tri-element reasoning result corresponding to the news text data, including the occurrence time, the occurrence place and the occurrence event.

[0063] A news text classification module is configured to input the occurrence event corresponding to the news text data into a classification model, and output the classification category corresponding to the news text data.

[0064] A news text embedding module is configured to input the occurrence event corresponding to the news text data into an embedding model, and output the embedding vector corresponding to the news text data.

[0065] A news text event clustering module is configured to search according to the classification category, the occurrence time and the occurrence place of the news text data, if the occurrence time and the occurrence place of different news text data are the same under the same classification category, the different news text data are clustered as the same occurrence event, and if the occurrence time and the occurrence place are different, the similarity of the embedding vectors corresponding to the different news text data is calculated, and whether the different news text data are clustered as the same occurrence event is determined according to the similarity.

[0066] The application outputs the event three-element reasoning result of news text data through a large language model, improves the accuracy of event clustering, and greatly reduces the cost of manual intervention.

[0067] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium, which can include ROM, RAM, magnetic disk or optical disk, etc.

[0068] Although the embodiments of the present application have been shown and described, it should be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A large language model-based event clustering method, characterized in that, The method comprises: obtaining one or more news text data from different news platforms, and preprocessing the news text data; inputting the preprocessed news text data into a trained large language model, outputting event tri-element inference results corresponding to the news text data, including occurrence time, occurrence place and occurrence event; inputting the occurrence event corresponding to the news text data into a classification model, outputting a classification category corresponding to the news text data; inputting the occurrence event corresponding to the news text data into an embedding model, outputting an embedding vector corresponding to the news text data; According to the classification category, occurrence time and occurrence place of the news text data, if the occurrence time and occurrence place of different news text data are the same under the same classification category, they are clustered into the same occurrence event, if the occurrence time and occurrence place are different, the similarity of the embedding vectors corresponding to different news text data is calculated, and whether to cluster into the same occurrence event is judged according to the similarity; the step of judging whether to cluster into the same occurrence event according to the similarity comprises: if the similarity of the embedding vectors of the current news text data and the recalled news text data exceeds the first threshold value, the occurrence events of the current news text data and the recalled news text data are clustered into the same occurrence event; if the similarity of the embedding vectors of the current news text data and the recalled news text data does not exceed the first threshold value but exceeds the second threshold value, the current news text data and each recalled news text data are input into a preset large language model, and the inference result of whether the current news text data and the recalled news text data are the same occurrence event is output; if the inference results of more than a preset number of recalled news text data are the same occurrence event, they are clustered into the same occurrence event, otherwise, manual auditing is performed; if the similarity of the embedding vectors of the current news text data and the recalled news text data does not exceed the second threshold value, the current news text data is regarded as an independent event; wherein the recalled news text data is the news text data in the recalled data set; If the inference results of more than a preset number of recalled news text data are the same occurrence event, they are clustered into the same event, which comprises selecting the embedding vectors of part of the recalled news text data of the same classification category according to a preset proportion; respectively obtaining the inference results of whether the current news text data and the part of the recalled news text data are the same occurrence event, if the inference results of more than a preset number of recalled news text data are the same occurrence event, the classification categories of the current news text data and the part of the recalled news text data are consistent, and they are clustered into the same occurrence event, otherwise, manual auditing is performed.

2. The event clustering method based on a large language model according to claim 1, characterized in that, The preprocessing of the news text data comprises hash deduplication processing of the news text data from different news platforms, if the hash value of the current news text data obtained already exists, the current news text data is deduplicated, if the hash value of the current news text data obtained does not exist, the current news text data is retained.

3. The event clustering method based on a large language model according to claim 1, characterized in that, The training process of the large language model comprises: building a large language model to generate an event tri-element inference prompt template; According to the inference prompt template, training data is constructed; the training data includes three elements of a news title, a news body, and a title generation event; The training data is input into a large language model, and the large language model is fine-tuned by low-rank adaptation; if the inference result output by understanding the news title meets the three elements of the title generation event, the three elements of the event are directly output, otherwise the large language model is repeatedly called; if the three elements of the event are still not met after exceeding the preset number of calls, the inference result is output by understanding the news body; if the inference result output by understanding the news body meets the three elements of the title generation event, the three elements of the event are directly output, otherwise the large language model is repeatedly called; if the three elements of the event are still not met after exceeding the preset number of calls, the news body is regarded as invalid data, and the next training data is trained.

4. The event clustering method based on a large language model according to claim 1, characterized in that, The similarity between the current news text data and the embedding vectors of the recalled news text data includes calculating the average similarity of the embedding vectors of all recalled news text data in the same classification category of the current news text data and the recalled data set, and sorting the average similarity of the classification category cluster in descending order; the average similarity of the classification category cluster with a high ranking is compared with the size of the first threshold and the second threshold.

5. A large language model-based event clustering system for performing any one of the large language model-based event clustering methods of claims 1-4, characterized in that, The system comprises: A news text grabbing module for obtaining one or more news text data from different news platforms; A news text preprocessing module for preprocessing the news text data; A news text inference module for inputting the preprocessed news text data into the trained large language model, outputting the event three element inference result corresponding to the news text data, including occurrence time, occurrence place and occurrence event; A news text classification module for inputting the occurrence event corresponding to the news text data into a classification model, outputting the classification category corresponding to the news text data; A news text embedding module for inputting the occurrence event corresponding to the news text data into an embedding model, outputting the embedding vector corresponding to the news text data; A news text event clustering module for searching according to the classification category, occurrence time and occurrence place of the news text data; if the occurrence time and occurrence place of different news text data are the same under the same classification category, they are clustered into the same occurrence event; if the occurrence time and occurrence place are different, the similarity of the embedding vectors corresponding to different news text data is calculated, and whether to cluster into the same occurrence event is determined according to the similarity.

Images