Hotspot event discovery method, device and equipment based on large model

Abstract

The invention relates to the field of network public opinions, and discloses a hot event discovery method, device and equipment based on a large model. The method comprises the following steps: acquiring text data, calculating Hash fingerprints and repetition times to identify potential hotspots, carrying out preliminary de-duplication, and carrying out BM25 and SimBERT similarity judgment on the potential hotspots and a historical event library to realize preliminary merging; and for inconsistent data, further calling a multi-stage large language model to carry out deep semantic understanding and event attribute judgment so as to exclude non-event information, supplementing networking information to generate a strong general title, and carrying out secondary BM25 comparison and judgment based on the supplemented title so as to finish accurate event aggregation and new event discovery. Through deep semantic understanding of a large language model and a multi-stage collaborative filtering mechanism, the limitation of a traditional method in processing complex or emerging events is effectively overcome, and the accuracy of event aggregation and the completeness of information supplementation are ensured.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a method, apparatus and device for discovering hot events based on a large model. Background Technology

[0002] In the era of big data, internet information is characterized by high concurrency, high growth, and high heterogeneity. News websites, social media, video platforms, forums, and short video comment sections generate massive amounts of unstructured data daily, with diverse content formats and extremely rapid updates. This information contains a wealth of important clues reflecting social dynamics, emergencies, and public opinion. How to quickly and accurately identify and rationally categorize valuable events from the complex, multi-source, and dynamically changing information flow across the entire internet has become a core issue urgently needing to be addressed in fields such as public opinion monitoring, social governance, emergency response, and media analysis.

[0003] Current methods for discovering and identifying trending events mainly rely on keyword or rule matching, statistical features, and semantic similarity. However, these existing technologies generally suffer from the following technical shortcomings:

[0004] (1) Traditional algorithms cannot meet the real-time processing and updating requirements of network-wide data and are difficult to maintain stable performance in millisecond-level response scenarios;

[0005] (2) Most methods rely on a single algorithm (such as title matching or shallow semantic model), which cannot fully capture semantic diversity and is prone to misjudgment or omission.

[0006] (3) The lack of effective identification and filtering of non-event information (such as entertainment news, common sense, advertising content, etc.) leads to mixed noise in the event set, affecting the accuracy of subsequent analysis;

[0007] (4) Events evolve from multiple sources and have diverse expressions during the process of network propagation. Traditional clustering methods cannot effectively merge different expressions of the same event, resulting in high event repetition rate and serious data redundancy, which affects user experience and downstream analysis applications.

[0008] In summary, existing hot topic discovery technologies have significant shortcomings in terms of real-time performance, accuracy, redundancy control, semantic understanding depth, and event deduplication and merging capabilities. With the accelerating flow of information across society and the growing demand for intelligent public opinion analysis, there is an urgent need for an efficient, accurate, and scalable event identification and merging method. This method should ensure real-time performance while achieving semantic consistency judgment and non-event filtering, thereby effectively supporting the intelligent discovery and continuous updating of large-scale hot topics. Summary of the Invention

[0009] In view of this, this application provides a hot topic event discovery method based on a large model, which solves the technical problem that existing technologies are unable to balance real-time performance, accuracy and redundancy removal in massive heterogeneous data, and cannot efficiently identify and merge hot topics events across society.

[0010] According to a first aspect of this application, a hotspot event discovery method based on a large model is provided, comprising:

[0011] Step S1: Obtain at least two text data entries and calculate and generate a hash fingerprint for each text data entry;

[0012] Step S2: Based on the hash fingerprint of the text data, determine the number of times each piece of text data is repeated. When the number of repetitions is greater than the repetition threshold, it is marked as potential hotspot data.

[0013] Step S3: Calculate the BM25 similarity between each potential hot data point and the title of each event in the historical event database whose publication time is within a preset time range, sort the BM25 similarity in descending order, and select the text data corresponding to the top N events with the highest BM25 similarity as the first comparison text data, where N is a positive integer.

[0014] Step S4: Calculate the semantic similarity between the potential hotspot data and the first comparison text data based on the SimBERT model;

[0015] Step S5: When the semantic similarity is greater than the text similarity threshold, it is determined that the semantics are consistent. The events corresponding to the potential hot data and the first comparison text data are merged, marked as the same event, and stored in the historical event database. When the semantic similarity is less than the text similarity threshold, proceed to step S6 to continue execution.

[0016] Step S6: Concatenate the potential hotspot data with the first comparison text data to form a context, call the first large language model to perform semantic understanding and judgment based on the first preset prompt words, and obtain the semantic judgment result output by the first large language model;

[0017] Step S7: When the semantic determination result output by the first language model indicates that the potential hotspot data and the first comparison text data belong to the same event, the potential hotspot data is merged into the corresponding event and stored in the historical event database; if it is determined that the potential hotspot data and the first comparison text data do not belong to the same event, then proceed to step S8 to continue execution.

[0018] Step S8: Based on the second preset prompt words, call the second major language model to determine the event attributes of potential hot data in order to exclude non-event information. Non-event information includes entertainment news, common sense, experience sharing, game reviews and advertisements.

[0019] Step S9: If the second language model determines that the potential hotspot data is event information, then proceed to step S10 to continue execution; if it determines that the potential hotspot data is not event information, then the process ends.

[0020] Step S10: Based on the third preset prompt word, call the third major language model to supplement the potential hot spot data with network information, so as to generate a highly generalized title and / or a highly generalized summary of the potential hot spot data.

[0021] Step S11: Based on the strongly generalized title and / or strongly generalized summary, calculate the BM25 similarity between each potential hot data point and the title of each event in the historical event database whose publication time is within a preset time range, and sort the BM25 similarity in descending order. Select the text data corresponding to the top N events with the highest BM25 similarity as the second comparison text data, where N is a positive integer.

[0022] Step S12: Based on the fourth preset prompt word, call the fourth language model to determine whether the title and / or summary of the potential hotspot data describes the same event as at least one text data in the second comparison text data; if the output of the fourth language model determines that they are the same event, then merge the events corresponding to the potential hotspot data and the second comparison text data and store them in the historical event database; if they are not the same, then mark the potential hotspot data as a new event and store it in the historical event database.

[0023] In one possible implementation, at least one of the first, second, third, and fourth language models is any one of the following: DeepSeek series models, Tongyi Qianwen model, Wenxin Yiyan model, ChatGPT series models, GPT-4 series models, ERNIE Bot model, GLM model, Claude model, or Tongyi Qianwen model.

[0024] In another possible implementation, step S2 specifically includes:

[0025] The hash fingerprint of each text data is stored in a hash table structure. The Hamming distance between each pair of text data is calculated. When the Hamming distance is less than the preset distance threshold, the corresponding text is determined to be a duplicate text and is included in the duplicate count result.

[0026] When the number of repetitions counted exceeds the repetition threshold, the steps for marking text data as potential hotspot data include:

[0027] In another possible implementation, step S4 specifically includes using the SimBERT model to extract sentence vector representations of potential hotspot data and the first comparison text data respectively, and calculating the cosine similarity between the two on an event-by-event basis.

[0028] In another possible implementation, step S10 specifically includes:

[0029] Information supplementation requests are constructed based on the third preset prompt words, which include event time, location, subject, key behavior, result, scope of impact and related entity elements, so as to guide the third language model to focus on the core content of the event when searching online;

[0030] The third language model is called to perform online information retrieval, expand the keywords in the potential hot data with external information, obtain external text information with high topic relevance to the potential hot data, and merge the potential hot data with the external text information to form the expanded potential hot data.

[0031] Based on the expanded potential hotspot data, the third major language model is invoked to perform semantic compression and information extraction, generating a highly generalized title containing elements of time, subject, behavior and result, and refining it into a highly generalized summary covering key information of the event.

[0032] In another possible implementation, step S6 includes:

[0033] The potential hotspot data is concatenated with the title, time, body and key behaviors of each data point in the first comparison text data according to the preset format to form the context input;

[0034] Based on the first preset prompt word, a semantic comparison request is sent to the first large language model to obtain the judgment result of the first large language model on whether it is the same event.

[0035] In another possible implementation, the first preset prompt includes event elements such as time, location, subject, key behavior, and result, which are used to guide the model to perform deep semantic comparison and generate a judgment result on whether they are the same event.

[0036] According to a second aspect of this application, a hotspot event detection device based on a large model is provided, comprising:

[0037] A historical event database is used to store information on historical hot events;

[0038] The fingerprint acquisition module is used to acquire at least two text data points and calculate and generate a hash fingerprint for each text data point;

[0039] The hotspot filtering module is used to determine the number of times each piece of text data is repeated based on the hash fingerprint of the text data. When the number of repetitions is greater than the repetition threshold, the corresponding text data is marked as potential hotspot data.

[0040] The similarity calculation module is used to calculate the BM25 similarity between each potential hot data and each text data in the historical event database whose publication time is within a preset time range, and sort them in descending order of similarity, and select the top N data with the highest similarity as the first comparison text data, where N is a positive integer;

[0041] The semantic comparison module is used to calculate the semantic similarity between potential hotspot data and the first comparison text data based on the SimBERT model.

[0042] The semantic merging module is used to determine that potential hotspot data and the first comparison text data are semantically consistent when the semantic similarity is greater than the text similarity threshold. It merges the events corresponding to the potential hotspot data and the first comparison text data, marks them as the same event, and stores them in the historical event database.

[0043] The semantic determination module is used to concatenate potential hotspot data with the first comparison text data to form a context when the semantic similarity is less than the text similarity threshold, and call the first large language model to perform semantic understanding determination based on the first preset prompt word, so as to obtain the semantic determination result output by the first large language model.

[0044] The event merging module is used to merge the potential hot data into the corresponding event and store it in the historical event database when the semantic determination result output by the first language model indicates that the potential hot data and the first comparison text data belong to the same event; if the semantic determination result indicates that the two do not belong to the same event, the potential hot data is sent to the event attribute determination module for processing.

[0045] The event attribute determination module is used to call the second language model based on the second preset prompt words to identify the event attributes of potential hot data, so as to exclude non-event information, including entertainment news, common sense, experience sharing, game reviews and advertisements.

[0046] The information supplementation module is used to call the third language model to supplement network information based on the third preset prompt words when the second language model determines that the potential hot data is event information, so as to generate a highly generalized title and / or a highly generalized summary of the potential hot data.

[0047] The secondary comparison module is used to calculate the BM25 similarity between potential hotspot data and each text data in the historical event database whose publication time is within a preset time range, based on the highly generalized title and / or highly generalized summary, and sort them in descending order of similarity, and select the top N data with the highest similarity as the second comparison text data, where N is a positive integer;

[0048] The final determination module is used to call the fourth language model based on the fourth preset prompt word to determine whether the title and / or summary of the potential hot data describes the same event as at least one text data in the second comparison text data. If the determination result output by the fourth language model is the same event, the event corresponding to the potential hot data and the second comparison text data is merged and stored in the historical event database. If the determination results are different, the potential hot data is marked as a new event and stored in the historical event database.

[0049] According to a third aspect of this application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described hotspot event discovery method based on a large model.

[0050] By employing the above technical solution, this application provides a method for discovering hot events based on a large model. This method calculates hash fingerprints on acquired text data to filter out potential hot events with high repetition rates. Then, it performs preliminary BM25 similarity screening and SimBERT semantic similarity comparison between the potential hot events and events in a historical event database, merging those with semantic similarity greater than a threshold. For data with inconsistent semantic similarity, semantic understanding is determined based on a first large language model. For data still not merged, non-event information is excluded based on a second large language model. For data confirmed as event information, a strongly summarized title and / or summary is generated based on a third large language model, and supplemented with online information. Finally, a second round of BM25 similarity screening is performed based on the strongly summarized title and / or summary, and a fourth large language model is invoked for final event merging or new event determination. This invention improves the accuracy, completeness, and timeliness of hot event discovery, avoids interference from non-event information, and enhances the accuracy of event clustering through multi-round semantic understanding and information supplementation using a large model.

[0051] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0052] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0053] Figure 1 This illustration shows a schematic diagram of the processing flow of a hotspot event discovery method based on a large model provided in an embodiment of this application;

[0054] Figure 2 The diagram illustrates a flowchart of a hotspot event discovery method based on a large model provided in an embodiment of this application.

[0055] Figure 3 A schematic diagram of a hotspot event detection device based on a large model provided in an embodiment of this application is shown. Detailed Implementation

[0056] The specific implementation of this application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0057] This invention provides a hotspot event discovery method based on a large model, which can be applied to, for example... Figure 1 In this scenario, the system includes the internet, a server, a historical event database, and an LLM (Large Language Model) module. The server acquires text data from the internet in real time, performs hash fingerprint calculations on the collected multi-source texts to remove duplicate data, and filters potential hotspot data based on the frequency of repetition. Subsequently, the server calls the BM25 algorithm to retrieve similar texts from the historical event database, and uses the SimBERT model to calculate semantic similarity, achieving rapid merging of potential hotspot data with existing events. When the semantic similarity is insufficient to determine event consistency, the server further calls the LLM big language model based on preset prompts for contextual semantic understanding and event determination, excluding non-event information. For text determined to be an event, the big language model performs online information supplementation, generating highly generalized titles and summaries, and again combines with the BM25 algorithm to complete the accurate aggregation and database entry of events. Through the above multi-layered semantic determination and information enhancement collaboration, this invention achieves real-time identification, semantic aggregation, and intelligent updating of hotspot events in internet public opinion, improving the accuracy and processing efficiency of event discovery.

[0058] The present invention will now be described in detail through specific embodiments.

[0059] like Figure 2 As shown, this is a hotspot event discovery method based on a large model provided in an embodiment of the present invention, including:

[0060] Step 201: Obtain text data and calculate the hash fingerprint of the text data;

[0061] This step involves acquiring at least two text data entries and calculating a hash fingerprint for each. The system processes each text entry from the event-related information collected across the entire network, calculating and generating a unique hash fingerprint for each entry. This hash fingerprint can be generated based on text content features using hash algorithms (such as MD5, SHA-256, etc.) and is used to identify the uniqueness of the text at the data level. It should be noted that this step does not limit the number of text entries to be acquired; rather, each acquired text entry is processed individually to ensure that all text information has an independent and traceable fingerprint. This design facilitates rapid duplicate detection and deduplication comparison during subsequent potential hotspot database establishment and event aggregation, thereby improving the accuracy and efficiency of data management.

[0062] Step 202: Screening potential hotspot data;

[0063] This step includes determining the repetition count of each text data based on its hash fingerprint. When the repetition count exceeds a threshold, the text is marked as a potential hotspot. Specifically, this involves storing the hash fingerprint of each text data in a hash table structure, calculating the Hamming distance between each pair of text data, and determining that the corresponding text is a duplicate text and included in the repetition count statistics when the Hamming distance is less than a preset distance threshold.

[0064] This step utilizes Redis for duplicate count statistics. When using Redis for duplicate count statistics, an approximate deduplication scheme based on Simhash can be adopted: the Simhash value of each text data is used as the key, and its corresponding occurrence count is stored as the value in Redis. When new text data arrives, its Simhash value is first calculated, and a query is performed in Redis to see if there is a key that is identical or has a Hamming distance less than a preset threshold. If it exists, the count value of the corresponding key is incremented by one using INCR or HINCRBY operations to accumulate the duplicate count. If it does not exist, the Simhash value is used as the new key and inserted into Redis with the count initialized to 1. This achieves efficient deduplication statistics, while leveraging Redis's in-memory structure to support fast query and update, making it suitable for potential hotspot detection scenarios with massive amounts of text data.

[0065] Step 203: Select the first comparison text data from the historical event database;

[0066] This step involves calculating the BM25 similarity between each potential hot data point and the title of each event in the historical event database whose publication time falls within a preset time range, sorting the BM25 similarity in descending order, and selecting the text data corresponding to the top N events with the highest BM25 similarity as the first comparison text data. N is a positive integer, and the preset time range can be set to, for example, 7 days or 1 month, and can be manually adjusted according to business needs or experience.

[0067] Since the data in the historical event database is stored in association according to event dimensions, with each event containing one or more source texts and their associated information such as titles and body text, multiple implementation methods can be used when calculating the BM25 similarity between each potential hot data point and events in the historical event database whose publication time falls within a preset time range. One approach is to calculate the BM25 similarity between the potential hot data and the unified titles of all events in the historical event database; alternatively, the BM25 similarity can be calculated separately for each set of titles of the potential hot data and each event in the historical event database (i.e., multiple text titles associated and stored under the same event), and the maximum or average similarity score can be used as the overall similarity result for that event. Furthermore, the similarity calculation in this step is not limited to the title field but can be extended to fields that reflect the semantic features of the text, such as body text, abstracts, and keywords, thereby improving the accuracy and robustness of event matching across multiple text feature dimensions.

[0068] In this embodiment, the historical event database can be built and managed using an Elasticsearch database. Elasticsearch has efficient full-text search and similarity matching capabilities, enabling structured indexing and fast querying of historical text events. By storing key information of historical events (such as event text, time, source, tags, and their corresponding feature vectors or Simhash values) in Elasticsearch, fast similar event retrieval and association analysis based on inverted indexes can be achieved. When new text data is added to the database, Elasticsearch's similarity queries (such as More Like This query or vector retrieval) can be used to find similar or identical events in the historical event database, realizing dynamic updates of historical event associations, hotspot evolution analysis, and event clustering. In this embodiment of the invention, identical events are stored together, and text data of the same event have the same event identifier.

[0069] Step 204: Calculate the semantic similarity with the first comparison text data;

[0070] Specifically, this step utilizes the SimBERT model to extract sentence vector representations of potential hotspot data and the first comparison text data. Cosine similarity is calculated on an event-by-event basis (the first comparison text data contains multiple text data corresponding to N events; therefore, this step compares data on an event-by-event basis, comparing it with multiple text data for a specific event). SimBERT is used to determine whether "current data" and "historical events" are semantically identical. Both are input into the SimBERT encoder to obtain sentence vectors (commonly [CLS] or mean-pooling vectors). L2 normalization is performed on the vectors before calculating cosine similarity. To reduce noise, light cleaning (removing URLs / emoticons / repeated spaces), standardizing number and time representations, and then entity extraction (time, location, subject) is performed, with similarity weighted (e.g., semantic similarity × α + entity consistency rate × β). When cosine similarity ≥ a threshold (e.g., 0.85) and key entity consistency rate ≥ a threshold (e.g., 0.6), the data is considered "semantically identical." For example, the current data "Beijing Metro Line 13 morning rush hour signal failure causing widespread delays" is compared with the titles of an event in the first set of comparison text data, namely "Beijing Metro Line 13 signal abnormality causing delays at multiple stations this morning" and "Beijing Metro Line 13 malfunction causing morning rush hour congestion". After encoding, the pairwise similarities are 0.91 and 0.88 respectively, and the consistency rate of key entities (Beijing, Line 13, signal / malfunction, morning rush hour) is 0.75. If the threshold is met, they are merged into the same event; if the similarity is only 0.62 or the key entities are missing, they are not merged and proceed to the subsequent judgment steps.

[0071] Step 205: When the semantic similarity is greater than the text similarity threshold, it is determined that the semantics are consistent. The events corresponding to the potential hot data and the first comparison text data are merged, marked as the same event, and stored in the historical event database. When the semantic similarity is less than the text similarity threshold, proceed to step 206 to continue execution.

[0072] In this system, each piece of text data represents an event. If the similarity between a potential hotspot data point and a piece of text data is greater than a text similarity threshold, then the potential hotspot data point and that piece of text data belong to the same event, and the events are merged. When the semantic similarity is less than the text similarity threshold, prompts are designed based on the semantic understanding capabilities of the large model to further determine whether to merge the events. To avoid misjudgments due to sentence structure differences or implicit semantics, the system designs prompts based on the semantic understanding capabilities of the large model for further judgment. For example, when the system detects two candidate pieces of information as follows:

[0073] The first comparison text data is "On November 5th, an explosion occurred at a chemical plant in XXX province, injuring multiple people." The potential hotspot data is "A safety accident occurred in the production workshop of an enterprise in XXX province; relevant departments have launched an investigation."

[0074] After calculation by the SimBERT model in this embodiment of the invention, the semantic similarity is only 0.55 (below the set threshold of 0.6), but in reality, the two may describe the same event. At this time, the system calls a large model (such as GPT-4 or ERNIE Bot) and inputs the following prompt to perform semantic judgment: "Please judge whether the following two texts describe the same event and explain the basis: The first comparison text data is: "On November 5, an explosion occurred in a chemical plant in XXX province, causing multiple injuries", and the potential hot data is "A safety accident occurred in the production workshop of an enterprise in XXX province, and relevant departments have intervened in the investigation."

[0075] The large model infers based on semantic associations such as time, location, and event type, outputting "Both may point to the same event (both are chemical accidents in XXX province on November 5th), and can be merged." Based on this, the system merges the potential hotspot data with the event corresponding to the original comparison text, marks it as the same event, and stores it in the historical event database. This design allows the system to supplement judgments using the semantic reasoning capabilities of the large model even at similarity threshold boundaries, avoiding the incorrect splitting of the same event due to differences in expression, thereby improving the completeness of event aggregation and intelligent discrimination capabilities.

[0076] Step 206: Call the first major language model to perform semantic understanding and determination;

[0077] Specifically, this step includes concatenating potential hotspot data with the first comparison text data to form a context, calling the first major language model to perform semantic understanding and judgment based on the first preset prompt words, and obtaining the semantic judgment result output by the first major language model. The first preset prompt words include event elements such as time, location, subject, key behavior, and result, used to guide the model to perform deep semantic comparison and generate a judgment result on whether they are the same event. Step 206 specifically includes:

[0078] Step 206-1: Combine the title, time, body, and key actions of each data point in the potential hotspot data and the first comparison text data according to the preset format to form context input;

[0079] Step 206-2: Send a semantic comparison request to the first large language model according to the first preset prompt word, and obtain the judgment result of the first large language model on "whether it is the same event".

[0080] Step 207: When the semantic determination result output by the first language model indicates that the potential hotspot data and the first comparison text data belong to the same event, the potential hotspot data is merged into the corresponding event and stored in the historical event database; if it is determined that the potential hotspot data and the first comparison text data do not belong to the same event, then proceed to step 208 to continue execution.

[0081] Step 208: Call the second major language model to determine event attributes;

[0082] Specifically, this step includes calling the second language model based on the second preset prompt words to determine the event attributes of potential hot data in order to exclude non-event information. Non-event information includes entertainment news, common sense, experience sharing, game reviews and advertisements.

[0083] Step 209: If the second language model determines that the potential hotspot data is event information, then proceed to step 210 to continue execution; if it determines that the potential hotspot data is not event information, then the process ends.

[0084] Step 210: Call the third major language model to generate a highly generalized title and / or a highly generalized summary;

[0085] This step involves using a third preset prompt word to call a third major language model to supplement the potential hotspot data with online information, in order to generate a highly summarized title and / or a highly summarized summary of the potential hotspot data. Specifically, this includes:

[0086] Step 210-1: Construct an information supplementation request based on the third preset prompt words, wherein the third preset prompt words include event time, location, subject, key behavior, result, scope of impact and related entity elements, so as to guide the third language model to focus on the core content of the event when searching online;

[0087] Step 210-2: Call the third language model to perform online information retrieval, expand the keywords in the potential hot data with external information, obtain external text information with high topic relevance to the potential hot data, and merge the potential hot data with the external text information to form the expanded potential hot data.

[0088] Step 210-3: Based on the expanded potential hotspot data, call the third major language model to perform semantic compression and information extraction, generate a highly generalized title containing time, subject, behavior and result elements, and refine it into a highly generalized summary covering key information of the event.

[0089] Step 211: Based on the highly generalized title and / or highly generalized summary, select the second comparison text data from the historical event database;

[0090] Specifically, this step includes calculating the BM25 similarity between each potential hot topic data and the title of each event in the historical event database whose publication time is within a preset time range, based on the strongly generalized title and / or strongly generalized summary, sorting the BM25 similarity in descending order, and selecting the text data corresponding to the top N events with the highest BM25 similarity as the second comparison text data, where N is a positive integer;

[0091] Step 212: Call the fourth language model to determine if they describe the same event;

[0092] Specifically, this step includes calling the fourth language model based on the fourth preset prompt word to determine whether the title and / or summary of the potential hotspot data describes the same event as at least one text data in the second comparison text data; if the output of the fourth language model determines that they are the same event, the event corresponding to the potential hotspot data and the second comparison text data is merged and stored in the historical event database; if they are not the same, the potential hotspot data is marked as a new event and stored in the historical event database.

[0093] In this embodiment, at least one of the first, second, third, and fourth language models is any one of the following: DeepSeek series models, Tongyi Qianwen model, Wenxin Yiyan model, ChatGPT series models, GPT-4 series models, ERNIE Bot model, GLM model, Claude model, or Tongyi Qianwen model.

[0094] Furthermore, as Figure 2 In a specific implementation of the method, this embodiment of the invention provides a hotspot event discovery device based on a large model, such as... Figure 3 As shown, the device includes:

[0095] Historical event database 310 is used to store information on historical hot events;

[0096] The fingerprint acquisition module 311 is used to acquire at least two text data and calculate and generate a hash fingerprint for each text data.

[0097] The hotspot filtering module 312 is used to determine the number of repetitions of each piece of text data based on the hash fingerprint of the text data. When the number of repetitions is greater than the repetition threshold, the corresponding text data is marked as potential hotspot data.

[0098] The similarity calculation module 313 is used to calculate the BM25 similarity between each potential hot data and the title of each event in the historical event database whose publication time is within a preset time range, and sort them in descending order of similarity, and select the text data corresponding to the top N events with the highest similarity as the first comparison text data, where N is a positive integer;

[0099] Semantic comparison module 314 is used to calculate the semantic similarity between potential hotspot data and the first comparison text data based on the SimBERT model;

[0100] The semantic merging module 315 is used to determine that the potential hotspot data and the first comparison text data are semantically consistent when the semantic similarity is greater than the text similarity threshold, merge the events corresponding to the potential hotspot data and the first comparison text data, mark them as the same event, and store them in the historical event database 310.

[0101] The semantic determination module 316 is used to concatenate potential hotspot data and first comparison text data into a context when the semantic similarity is less than the text similarity threshold, and call the first large language model to perform semantic understanding determination based on the first preset prompt word, so as to obtain the semantic determination result output by the first large language model.

[0102] The event merging module 317 is used to merge the potential hotspot data into the corresponding event and store it in the historical event database 310 when the semantic determination result output by the first large language model indicates that the potential hotspot data and the first comparison text data belong to the same event; if the semantic determination result indicates that the two do not belong to the same event, the potential hotspot data is sent to the event attribute determination module 318 for processing.

[0103] The event attribute determination module 318 is used to call the second language model based on the second preset prompt words to identify the event attributes of potential hot data, so as to exclude non-event information, including entertainment news, common sense, experience sharing, game reviews and advertisements.

[0104] The information supplementation module 319 is used to call the third language model to supplement network information based on the third preset prompt word when the second language model determines that the potential hot data is event information, so as to generate a strong summary title and / or a strong summary summary of the potential hot data.

[0105] The secondary comparison module 320 is used to calculate the BM25 similarity between potential hotspot data and each text data in the historical event database 310 whose publication time is within a preset time range, based on the strongly generalized title and / or strongly generalized summary, and sort them in descending order of similarity, and select the top N data with the highest similarity as the second comparison text data, where N is a positive integer;

[0106] The final determination module 321 is used to call the fourth language model based on the fourth preset prompt word to determine whether the title and / or summary of the potential hot data describes the same event as at least one text data in the second comparison text data; if the determination result output by the fourth language model is the same event, the event corresponding to the potential hot data and the second comparison text data is merged and stored in the historical event database 310; if the determination results are different, the potential hot data is marked as a new event and stored in the historical event database 310.

[0107] This invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for discovering hotspot events based on a large model, including:

[0108] Step S1: Obtain at least two text data entries and calculate and generate a hash fingerprint for each text data entry;

[0109] Step S2: Based on the hash fingerprint of the text data, determine the number of times each piece of text data is repeated. When the number of repetitions is greater than the repetition threshold, it is marked as potential hotspot data.

[0110] Step S3: Calculate the BM25 similarity between each potential hot data point and the title of each event in the historical event database whose publication time is within a preset time range, sort the BM25 similarity in descending order, and select the text data corresponding to the top N events with the highest BM25 similarity as the first comparison text data, where N is a positive integer.

[0111] Step S4: Calculate the semantic similarity between the potential hotspot data and the first comparison text data based on the SimBERT model;

[0112] Step S5: When the semantic similarity is greater than the text similarity threshold, it is determined that the semantics are consistent. The events corresponding to the potential hot data and the first comparison text data are merged, marked as the same event, and stored in the historical event database. When the semantic similarity is less than the text similarity threshold, proceed to step S6 to continue execution.

[0113] Step S6: Concatenate the potential hotspot data with the first comparison text data to form a context, call the first large language model to perform semantic understanding and judgment based on the first preset prompt words, and obtain the semantic judgment result output by the first large language model;

[0114] Step S7: When the semantic determination result output by the first language model indicates that the potential hotspot data and the first comparison text data belong to the same event, the potential hotspot data is merged into the corresponding event and stored in the historical event database; if it is determined that the potential hotspot data and the first comparison text data do not belong to the same event, then proceed to step S8 to continue execution.

[0115] Step S8: Based on the second preset prompt words, call the second major language model to determine the event attributes of potential hot data in order to exclude non-event information. Non-event information includes entertainment news, common sense, experience sharing, game reviews and advertisements.

[0116] Step S9: If the second language model determines that the potential hotspot data is event information, then proceed to step S10 to continue execution; if it determines that the potential hotspot data is not event information, then the process ends.

[0117] Step S10: Based on the third preset prompt word, call the third major language model to supplement the potential hot spot data with network information, so as to generate a highly generalized title and / or a highly generalized summary of the potential hot spot data.

[0118] Step S11: Based on the strongly generalized title and / or strongly generalized summary, calculate the BM25 similarity between each potential hot data point and the title of each event in the historical event database whose publication time is within a preset time range, and sort the BM25 similarity in descending order. Select the text data corresponding to the top N events with the highest BM25 similarity as the second comparison text data, where N is a positive integer.

[0119] Step S12: Based on the fourth preset prompt word, call the fourth language model to determine whether the title and / or summary of the potential hotspot data describes the same event as at least one text data in the second comparison text data; if the output of the fourth language model determines that they are the same event, then merge the events corresponding to the potential hotspot data and the second comparison text data and store them in the historical event database; if they are not the same, then mark the potential hotspot data as a new event and store it in the historical event database.

[0120] It should be noted that the above embodiments only illustrate the principles and implementation steps of the present invention using the actual scenario of text data deduplication and public opinion hotspot detection as examples. Specific application scenarios are not limited. The present invention is also applicable to other occasions requiring similarity identification, event aggregation, and hotspot analysis of large-scale text, log data, online public opinion, industrial monitoring data, or scientific research literature. For details regarding the functions or steps achievable by the computer-readable storage medium or computer device, please refer to the descriptions in the foregoing method embodiments; they will not be repeated here.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0122] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0123] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A large model-based hot event discovery method, characterized in that, The method comprises the following steps: Step S1, obtaining at least two pieces of text data, and calculating the hash fingerprints of each piece of text data; Step S2, judging the repetition times of each piece of text data according to the hash fingerprints of the text data, and marking as potential hot data when the repetition times are greater than a repetition times threshold; Step S3, calculating the BM25 similarity of each potential hot data and each event title in the historical event library with a publishing time within a preset time range, and performing BM25 similarity descending sorting to screen out the text data corresponding to the top N events with the highest BM25 similarity as first comparison text data, N being a positive integer; Step S4, calculating the semantic similarity of the potential hot data and the first comparison text data based on a SimBERT model; Step S5, determining semantic consistency when the semantic similarity is greater than a text similarity threshold, merging the potential hot data and the first comparison text data corresponding events, and marking as the same event and storing in the historical event library; when the semantic similarity is less than the text similarity threshold, jumping to step S6 for continuous execution; Step S6, splicing the potential hot data and the first comparison text data as context, calling a first large language model based on a first preset prompt word for semantic understanding determination, and obtaining a semantic determination result output by the first large language model; Step S7, when the semantic determination result output by the first large language model is that the potential hot data and the first comparison text data belong to the same event, merging the potential hot data into the corresponding event and storing in the historical event library; if it is determined that the potential hot data and the first comparison text data do not belong to the same event, then jumping to step S8 for continuous execution; Step S8, calling a second large language model based on a second preset prompt word to determine the event attribute of the potential hot data to exclude non-event information, wherein the non-event information includes entertainment news, life common sense, experience sharing, game review and advertisement; Step S9, if the second large language model determines that the potential hot data is event information, then jumping to step S10 for continuous execution; if it is determined that the potential hot data is non-event information, then the process ends; Step S10, calling a third large language model based on a third preset prompt word to supplement the networking information of the potential hot data to generate a strong summary title and / or a strong summary abstract of the potential hot data; Step S11, calculating the BM25 similarity of each potential hot data and each event title in the historical event library with a publishing time within a preset time range according to the strong summary title and / or the strong summary abstract, and performing BM25 similarity descending sorting to screen out the text data corresponding to the top N events with the highest BM25 similarity as second comparison text data, N being a positive integer; Step S12, calling a fourth large language model based on a fourth preset prompt word, judging whether the title and / or abstract of the potential hot spot data and at least one piece of text data in the second comparison text data describe the same event; if the fourth large language model outputs a determination that they are the same event, the potential hot spot data and the event corresponding to the second comparison text data are merged and stored in the historical event library; if it is determined that they are not the same, the potential hot spot data is marked as a new event and stored in the historical event library.

2. The large model-based hot event discovery method according to claim 1, characterized in that At least one of the first large language model, the second large language model, the third large language model, and the fourth large language model is any one of the following: a DeepSeek series model, a TongYiQianWen model, a WenXinYiYan model, a ChatGPT series model, a GPT-4 series model, an ERNIE Bot model, a GLM model, a Claude model, or a TongYiQianWen model. 3.The large model-based hot event discovery method according to claim 1, characterized in that, The step S2 specifically includes: storing the hash fingerprints of each piece of text data into a hash table structure, calculating the Hamming distance between each pair of text data, and determining that the corresponding text is repeated text and counting the number of repetitions when the Hamming distance is less than a preset distance threshold; when the number of repetitions is greater than a repetition threshold, the text data is marked as potential hot spot data.

4. The large model-based hot event discovery method of claim 1, wherein, The step S4 specifically includes extracting the sentence vector representation of the potential hot spot data and the first comparison text data using a SimBERT model, respectively, and calculating the cosine similarity of the two in units of events.

5. The large model-based hot event discovery method according to claim 4, characterized in that, The step S10 specifically includes: constructing an information supplement request based on a third preset prompt word, wherein the third preset prompt word includes event time, place, subject, key behavior, result, influence range, and related entity elements to guide the third large language model to focus on the core content of the event during network search; calling the third large language model to perform network information retrieval, supplementing external information for the keywords in the potential hot spot data, obtaining external text information with high thematic relevance to the potential hot spot data, and merging the potential hot spot data with the external text information to form expanded potential hot spot data; based on the expanded potential hot spot data, calling the third large language model for semantic compression and information extraction, generating a strong summary title containing time, subject, behavior, and result elements, and refining a strong summary abstract covering key event information.

6. The large model-based hot event discovery method according to claim 1, characterized in that, The step S6 includes: concatenating the title, time, subject, and key behavior of the potential hot spot data and each piece of data in the first comparison text data into a context input according to a preset format; sending a semantic comparison request to the first large language model according to a first preset prompt word to obtain the determination result of the first large language model output "whether it is the same event".

7. The large model-based hot event discovery method according to claim 1, characterized in that, The first preset prompt word includes event elements such as time, place, subject, key behavior, and result, which are used to guide the model to perform deep semantic comparison and generate a judgment result of whether it is the same event. 8.A large model-based hot event discovery apparatus, characterized by comprising: It includes: a historical event database for storing historical hot spot event information; a fingerprint acquisition module, configured to acquire at least two pieces of text data, and calculate a hash fingerprint of each piece of text data; a hotspot screening module, configured to determine a repetition number of each piece of text data according to the hash fingerprint of the text data, and mark the corresponding text data as potential hotspot data when the repetition number is greater than a repetition number threshold; a similarity calculation module, configured to calculate a BM25 similarity of each piece of the potential hotspot data with each piece of text data in the historical event database and published within a preset time range, sort the similarities in descending order, and screen out the top N pieces of data with the highest similarities as first comparison text data, where N is a positive integer; a semantic comparison module, configured to calculate a semantic similarity of the potential hotspot data and the first comparison text data based on a SimBERT model; a semantic merging module, configured to determine that the potential hotspot data and the first comparison text data are semantically consistent when the semantic similarity is greater than a text similarity threshold, merge events corresponding to the potential hotspot data and the first comparison text data, mark the events as the same event, and store the events in the historical event database; a semantic determination module, configured to concatenate the potential hotspot data and the first comparison text data as context when the semantic similarity is less than the text similarity threshold, and call a first large language model based on a first preset prompt word to determine semantics to obtain a semantic determination result output by the first large language model; an event merging module, configured to incorporate the potential hotspot data into a corresponding event and store the potential hotspot data in the historical event database when the semantic determination result output by the first large language model indicates that the potential hotspot data and the first comparison text data belong to the same event; if the semantic determination result indicates that the potential hotspot data and the first comparison text data do not belong to the same event, the potential hotspot data is sent to an event attribute determination module for processing; the event attribute determination module, configured to call a second large language model based on a second preset prompt word to identify event attributes of the potential hotspot data to exclude non-event information, the non-event information including entertainment news, common sense, experience sharing, game reviews, and advertisements; an information supplementing module, configured to call a third large language model based on a third preset prompt word to supplement network information when the second large language model determines that the potential hotspot data is event information, to generate a strong summary title and / or a strong summary abstract of the potential hotspot data; a secondary comparison module, configured to calculate a BM25 similarity of the potential hotspot data with each piece of text data in the historical event database and published within a preset time range according to the strong summary title and / or the strong summary abstract, sort the similarities in descending order, and screen out the top N pieces of data with the highest similarities as second comparison text data, where N is a positive integer; and the event attribute determination module, configured to call a second large language model based on a second preset prompt word to identify event attributes of the potential hotspot data to exclude non-event information, the non-event information including entertainment news, common sense, experience sharing, game reviews, and advertisements. A final determination module is configured to determine whether the title and / or the abstract of the potential hot event data and at least one piece of text data in the second comparison text data describe the same event based on a fourth preset prompt word calling a fourth large language model; if the determination result output by the fourth large language model is the same event, the potential hot event data and the event corresponding to the second comparison text data are merged and stored in the historical event database; if the determination result is not the same, the potential hot event data is marked as a new event and stored in the historical event database.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the hot event discovery based on the large model in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-9. The computer program is executed by the processor to implement the steps of the hot event discovery based on the large model in any one of claims 1 to 7.

Images