A method and system for identifying key events based on a hierarchical processing architecture
By constructing a hierarchical processing architecture and combining semantic extraction and improved clustering methods specific to the government sector, the efficiency and accuracy issues of key event identification in the government sector were resolved, achieving efficient and accurate screening of key events.
Patent Information
- Application Number
- CN202510692617.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-27
- Publication Date
- 2026-02-17
- Estimated Expiration
- 2045-05-27
AI Technical Summary
Existing technologies lack systematic solutions in the government sector, resulting in low identification efficiency and accuracy. They are difficult to screen key events efficiently and accurately, especially when faced with multi-channel and diverse data, leading to duplicate identification and waste of resources.
A hierarchical processing architecture is adopted, including data preprocessing, semantic regularization agglomerative hierarchical clustering, and intra-cluster entity keyword extraction. A three-level processing architecture of 'classification-clustering-discrimination' is constructed. Through semantic extraction specific to the government domain, semantic regularization agglomerative hierarchical clustering, and intra-cluster entity keyword comparison, efficient screening of key events is achieved.
It significantly improves the system's processing efficiency and recognition accuracy, enabling it to quickly and accurately identify key events, meet the real-time response needs in urban governance scenarios, reduce computational complexity, and improve recognition accuracy.
Smart Images
Figure CN120705313B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, and in particular to a key event discrimination method and system based on a hierarchical processing architecture. BACKGROUND
[0002] With the continuous development of the economy and society, the urbanization rate is continuously improving, and personal interests and demands are becoming diversified. The ways and channels for reflecting problems in various fields of society are also continuously enriched. As the main body of social governance, the relevant management departments will receive a large number of problems with different demands every day. For the management department, it is necessary to screen and mark key events from them for priority disposal or attention. For example, events such as water stoppage and heating pipe explosion have a large impact and involve different groups of people in the same area reporting the same event. There are also the same event reported by the same person through different channels. If the problem is not disposed of in time or improperly, it may trigger a group event, seriously affecting social security and stability. Due to the diversity of reporting channels and the large number of reports, the current solution is still manual screening or keyword matching, which is not accurate, time-consuming and labor-intensive, and important event information may be missed due to judgment deviation, resulting in decision-making errors and seriously affecting the efficiency of grassroots governance. Therefore, it is urgent to use the intelligent capabilities of large models to uniformly understand, analyze, screen and mark events, provide early warning according to the impact range and severity of the event, improve work efficiency and screening accuracy, assist decision-making, and eliminate potential social security risks.
[0003] In practical applications, key event recognition technology mainly focuses on the accurate screening of one-event-multiple-reporting scenarios. This technology is a technical problem that urgently needs to be solved in the field of urban governance. Due to the diversity and unpredictability of event types, the lack of unified standards for multi-channel data formats, the possibility of misjudgment of template texts, and the pressure brought by massive data processing, these factors together increase the difficulty of recognition. If the system cannot accurately identify and merge key events, it may lead to problems such as duplication of final statistical results, waste of resources, and decision-making errors.
[0004] Specifically, the technical problems existing in key event screening include but are not limited to:
[0005] 1. The large amount of daily reported data leads to low processing efficiency;
[0006] 2. The event source channels are diverse, and there is a lack of a general solution;
[0007] 3. The event type and content description are complex and variable, affecting the recognition accuracy.
[0008] Therefore, how to efficiently, universally and accurately screen key events from reported events has become a key technical challenge to improve the system processing capability, decision support value and actual application effect.
[0009] There are mainly three solutions for the key event screening scene in the prior art:
[0010] 1. Rule-based manual discrimination method: Through pre-set keyword matching algorithm, geographic location matching rule and named entity recognition technology, etc., key words are screened and discriminated from events. This method usually formulates a series of judgment standards by professionals, such as containing specific keywords in event description, occurring in the same or adjacent area, time interval within a certain range, etc., to identify possible key events.
[0011] 2. Large language model screening method: Using the powerful semantic understanding and generation ability of pre-trained large language models (such as Qwen, etc.), all reported contents can be directly subjected to deep semantic extraction and analysis. Specifically, all event contents reported daily are input into the large language model, and the generalization ability of the model is used to analyze the event contents to find out the key events that are repeatedly reported; the event contents can also be screened and analyzed in batches.
[0012] 3. Deep learning screening method: In recent years, semantic representation technology based on deep learning has gradually become the mainstream method in the field of natural language processing. These technologies can effectively capture the semantic relationship between words and sentences by mapping text to high-dimensional vector space. Specifically, pre-trained embedding models are used to convert the contents of reported events into high-dimensional vector representations with domain-specific features, so as to better capture the deep semantic features of the text. On this basis, the distance between vectors can be calculated to quantify the semantic similarity between texts, and according to the semantic similarity, it is determined whether the two texts match the key event.
[0013] However, the main defects of the prior art include but are not limited to the following aspects:
[0014] 1. Lack of systematic solution: The existing technologies such as deep learning and large language model screening method mainly use single technical means, and fail to effectively integrate classification, clustering and semantic understanding and other methods, resulting in difficulty in achieving a good balance between processing efficiency, recognition accuracy and universality, and being unable to fully cope with complex and diverse data reporting scenarios.
[0015] 2. Low recognition efficiency: The large language model screening method directly uses the model to compare all event data, and the computational complexity increases exponentially. When facing the massive event data collected by relevant departments on a daily basis, the processing efficiency of this method is significantly reduced, the response time is prolonged, and it is difficult to meet the needs of real-time or near real-time processing.
[0016] 3. Low recognition accuracy: the existing technology generally lacks an effective preprocessing mechanism for the text characteristics in the government field. The data sources are diverse and the formats are different, which limits the generalization ability of the model. A large amount of redundant or meaningless information is easily introduced in the semantic extraction process, which affects the calculation effect of the subsequent semantic similarity, and further reduces the accuracy of the key event recognition.
[0017] In summary, in the process of key event recognition in the government field, the existing technology generally has the key technical shortcomings of single system architecture, low recognition efficiency and low recognition accuracy, and it is difficult to realize efficient, accurate and universal key event screening. SUMMARY
[0018] The purpose of the present application is to provide a key event discrimination method and system based on a hierarchical processing architecture, which aims to solve the above problems in the prior art.
[0019] The embodiment of the present application provides a key event discrimination method based on a hierarchical processing architecture, comprising:
[0020] Obtaining reported events from different channels, preprocessing the reported events, obtaining processed event texts, and classifying the event texts;
[0021] Using a semantic regularization agglomerative hierarchical clustering method to cluster the classified texts;
[0022] According to the clustering results, using an intra-cluster entity keyword extraction and comparison mechanism to discriminate the repeated events in the texts, and obtaining key event discrimination results.
[0023] The embodiment of the present application provides a key event discrimination system based on a hierarchical processing architecture, comprising:
[0024] A classification module for obtaining reported events from different channels, preprocessing the reported events, obtaining processed event texts, and classifying the event texts;
[0025] A clustering module for using a semantic regularization agglomerative hierarchical clustering method to cluster the classified texts;
[0026] A discrimination module for using an intra-cluster entity keyword extraction and comparison mechanism to discriminate the repeated events in the texts according to the clustering results, and obtaining key event discrimination results.
[0027] The embodiment of the present application also provides an electronic device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the computer program is executed by the processor to implement the steps of the key event discrimination method based on the hierarchical processing architecture.
[0028] The embodiment of the present application also provides a computer readable storage medium, wherein information transmission is stored on the computer readable storage medium, and the program is executed by a processor to realize the steps of the key event identification method based on the hierarchical processing architecture.
[0029] The embodiment of the present application can have the following beneficial effects: the embodiment of the present application proposes a key event identification method based on a hierarchical processing architecture, constructs a hierarchical processing architecture of "classification-clustering-discrimination", and forms a complete and systematic key event identification solution. The solution combines a special semantic extraction method in the government field, a semantic regularization condensation hierarchical clustering method and an entity keyword extraction technology in a cluster, and through multi-stage collaborative processing of semantic vectorization, clustering optimization and key information verification, significantly improves the processing efficiency and identification accuracy of the system, and effectively meets the actual needs of real-time response and accurate discrimination of large-scale event data in the urban governance scenario. BRIEF DESCRIPTION OF DRAWINGS
[0030] In order to more clearly illustrate the technical solutions in the one or more embodiments of the present application or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative labor under the premise of the drawings.
[0031] Figure 1 is a key event identification method flowchart based on a hierarchical processing architecture of the embodiment of the present application;
[0032] Figure 2 is a key event identification flowchart of the embodiment of the present application;
[0033] Figure 3 is a three-level processing framework overall architecture diagram of the embodiment of the present application;
[0034] Figure 4 is a special semantic extraction flowchart of the government field of the embodiment of the present application;
[0035] Figure 5 is a key event identification system schematic diagram based on a hierarchical processing architecture of the embodiment of the present application. DETAILED DESCRIPTION
[0036] In order for those skilled in the art to better understand the technical solutions in the one or more embodiments of the present specification, the technical solutions in the one or more embodiments of the present specification will be clearly and completely described in the following with reference to the drawings in the one or more embodiments of the present specification. Obviously, the described embodiments are only a part of the embodiments of the present specification, rather than all the embodiments. Based on the one or more embodiments of the present specification, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present document.
[0037] Method embodiments
[0038] According to the embodiments of the present application, a key event discrimination method based on a hierarchical processing architecture is provided, Figure 1 is the flow chart of the key event discrimination method based on the hierarchical processing architecture of the embodiments of the present application, as Figure 1 shown, the key event discrimination method based on the hierarchical processing architecture according to the embodiments of the present application specifically includes:
[0039] Step S101, obtaining reported events from different channels, performing data preprocessing on the reported events, obtaining processed event text, and classifying the event text, specifically including:
[0040] Data cleaning and filtering, text standardization processing, domain-specific processing, and validity verification are performed on the reported events;
[0041] Among them, the data cleaning and filtering includes filtering empty text, removing non-string content and removing interference symbols;
[0042] The text standardization processing includes constructing a corresponding regular expression according to different channels, extracting key content from the reported events through the corresponding regular expression, and performing word segmentation processing on the text using a word segmentation tool;
[0043] The domain-specific processing is to screen the text using a pre-constructed government domain exclusive stopword list, and to remove formatted words, single-word words and pure number fields without actual semantics;
[0044] The validity verification is to check whether the processed text record is empty, and if so, to mark and remove the text record;
[0045] The event text is preliminarily classified according to the event category by a large language model; wherein the event category includes natural disasters, fire safety, traffic safety, public safety, social safety and contradictions and disputes;
[0046] Step S102, using a semantic regularization condensation hierarchical clustering method to cluster the classified text, specifically including:
[0047] The semantic embedding model is used to encode elements in each type of text to generate corresponding semantic embedding vectors, a cosine distance matrix in symmetric form is constructed based on the semantic embedding vectors by using a cosine similarity function to calculate the cosine distance between each pair of texts, the cosine distance matrix is regularized, the regularized distance matrix is clustered by using an average linkage method, and the clustering results are mapped back to the original data structure after clustering is completed, and a corresponding cluster number is assigned to each valid record;
[0048] Step S103, according to the clustering result, the repeated event in the text is distinguished by using the entity keyword extraction and comparison mechanism in the cluster, and the key event discrimination result is obtained, which specifically includes:
[0049] The clustering results are traversed, the isolated clusters containing only single records in the clustering results are removed, and the clusters containing multiple records are judged for repeated content. If there are completely identical text fields, it is directly determined as a repeated event. If there are no completely identical text fields, the top k high-weight entity keywords of each record are extracted to obtain a keyword set of each record. The number of common keywords and the uniqueness ratio in the cluster are calculated based on the keyword set. If the number of common keywords is less than the preset minimum common keyword threshold, and the uniqueness ratio is greater than the preset maximum uniqueness ratio upper limit, the cluster is removed, and the remaining valid cluster is retained. The remaining valid cluster is input into a large language model for discrimination to obtain the final key event discrimination result.
[0050] The specific situation of the key event discrimination method based on the hierarchical processing architecture according to the embodiments of the present application is described below, as shown in Figure 2 The above technical solutions of the embodiments of the present application are described in detail.
[0051] To solve the core problems of the prior art, such as the difficulty of balancing accuracy and computational efficiency, the limitation of system adaptability and generalization ability, etc., the embodiments of the present application propose an innovative technical solution. The solution aims to improve the judgment accuracy while significantly reducing the computational overhead by building a new processing mechanism from multiple levels such as data semantic understanding, feature optimization extraction, and intelligent clustering analysis, thereby realizing more efficient, accurate and transferable technical application effects. The solution specifically includes:
[0052] 1. Construct a three-level hierarchical architecture of "classification-clustering-discrimination" to form a structured key event screening system: The embodiment of the present application proposes a key event identification method based on hierarchical processing architecture, which is used for efficiently identifying key events repeatedly reported through different channels in the field of urban governance. Relying on the natural language rules of "the same event belongs to the same category" and "events of the same category have similar representations in the semantic space", the method constructs a hierarchical processing architecture composed of three stages of classification, clustering and discrimination, forming a systematic and scalable key event screening solution. By integrating the semantic extraction technology specific to the government field, the semantic regularization agglomerative hierarchical clustering method, and the entity keyword extraction and comparison mechanism within the cluster, the embodiment of the present application realizes the whole-process closed-loop processing from raw text preprocessing, semantic vector representation, dynamic clustering analysis to final key event determination, significantly improving the accuracy and efficiency of the screening system.
[0053] 2. Government text semantic cleaning and normalization processing to improve effective semantic representation ability: The embodiment of the present application proposes a text semantic extraction method for government scenarios, aiming to improve the accuracy of event text understanding and the reliability of repetitive determination. The method combines language rules and government data characteristics to construct regular expressions and domain-specific stop word lists, extending the common formatted expressions and redundant words without actual semantics in the reporting text based on general stop words, significantly improving the effective proportion of semantic information after vectorization, and then extracting specific description content through different regular expressions according to the reporting channel. The preprocessing process includes key steps such as empty text filtering, interference symbol removal, word segmentation, government stop word removal, single word and pure number content removal, etc., ensuring that the final text only retains keywords with actual semantics and improving the recognition accuracy.
[0054] 3. Introduce semantic regularization mechanism to improve clustering efficiency and quality: The embodiment of the present application improves the traditional hierarchical clustering algorithm and proposes a semantic regularization agglomerative hierarchical clustering method (SRHC). This method standardizes the semantic similarity matrix to make the data distribution approach to the standard normal distribution, thereby optimizing the similarity calculation and cluster merging strategy in the clustering process. Based on the principle of mathematical statistics, this algorithm significantly improves the computational efficiency while maintaining the stability and discrimination ability of the clustering results.
[0055] 4. Introduce the mechanism of entity keyword extraction and comparison in the cluster to improve the accuracy of judgment: the embodiment of the present application proposes an entity keyword extraction technology in the cluster. After completing the clustering analysis, the text content in each event cluster is deeply semantically analyzed, the high-weight keywords related to the key entities (such as places, names, etc.) are accurately identified and extracted, and the top five keywords in the weight ranking are selected for comparison. According to the uniqueness mechanism and the number of common keywords, the entity keywords are compared and screened for effective elements in the cluster. This mechanism not only ensures that the event cluster meets the judgment condition of "key event" in form, but also realizes strict verification at the content detail level, significantly improving the accuracy of the screening result.
[0056] The embodiment of the present application faces the key event screening scene in the urban governance scene, and proposes a key event identification method based on a hierarchical processing architecture. The method combines the powerful semantic understanding ability of a large language model and an improved semantic embedding driven dynamic hierarchical clustering strategy, constructs a "classification-cluster-discrimination" three-level processing framework, and realizes the efficiency and accuracy of key event screening, as shown in Figure 3 The specific steps are as follows:
[0057] 1. The text semantic extraction method of the government affair scene is as shown in Figure 4 : standardization cleaning process is implemented on the original reported text, and empty text, non-string content and interference symbols are filtered. The text segmentation is completed by using a word segmentation tool, and the reported information of different channels is extracted by using a regular expression. A government affair field exclusive stop word table is further introduced to eliminate formatted words, single words and pure number fields without actual semantic information, so that the final reserved words have actual semantic information. If a record is empty after cleaning, it is marked as an invalid record and is removed.
[0058] 2. The reported content is classified by text based on a large language model. Specifically, the event data reported on the same day is obtained by connecting the database, and the large language model is called to perform intelligent classification labeling through the prompt word and structured output framework. The classification system covers six core categories: natural disasters, fire safety, traffic safety, public safety, social safety and disputes. This process relies on the deep understanding ability of the large model to the semantic boundary of the classification label, realizes the accurate classification of the event content, effectively reduces the subsequent processing range, and establishes a reasonable semantic boundary for the grouping analysis.
[0059] 3、Semantic vectorization and similarity modeling: Call the semantic embedding model for each element in the class classified in the previous step to generate high-dimensional vectors with rich semantic representation ability. Calculate the semantic association degree between each pair of texts based on the cosine similarity function, construct a symmetric form of semantic similarity matrix, and then regularize the semantic similarity matrix to make the data distribution more consistent with the standard normal distribution rule, thereby improving the clustering efficiency and stability. Merge similar clusters layer by layer through the average linkage method, and set the distance threshold or the upper limit of the number of clusters to control the clustering granularity. After clustering is completed, map the results back to the original data structure and assign a corresponding cluster number to each valid record.
[0060] 4、Entity keyword extraction and consistency verification: Traverse all clustering results and eliminate isolated clusters containing only a single record. For clusters containing multiple records, first determine whether there are completely repeated content fields. If so, it is directly determined as a key event. If not, further extract high-weight entity keywords (such as location, name, etc.) in each record to construct a keyword set within the cluster. Calculate the number of common keywords and the uniqueness ratio. Specifically, the number of common keywords represents the number of keywords that appear in all event texts within the cluster, and the formula is:
[0061] CKC = |common_keywords| ≥ MIN_COMMON_KEYWORDS (1);
[0062] Set the minimum threshold MIN_COMMON_KEYWORDS. If it is lower than this value, it is considered that the keyword consistency is insufficient, and the cluster is excluded. The uniqueness ratio measures the proportion of non-common parts in the keyword set within the cluster, avoiding false positives due to low keyword overlap. The formula is:
[0063]
[0064] where n is the number of records within the cluster, keywords i is the keyword set extracted from the ith record. Set the maximum allowed value MAX_UNIQUE_RATIO. If it is higher than this threshold, it is considered that the keyword distribution is too dispersed and does not constitute the same event. This mechanism effectively compensates for the technical shortcomings of traditional methods in the semantic comparison of key entities, significantly improving the screening accuracy.
[0065] 5. Intra-cluster semantic consistency verification based on large language model: Because there is still a small probability of non-key events in the intra-cluster processed by the above steps, a large language model is called to perform semantic-level discrimination on the screened structured effective event cluster to confirm whether the events in the cluster belong to the same event, and finally output the specific description of the key event, that is, the identified key event is marked and warned as an event that needs to be paid attention to by the management personnel for subsequent decision-making reference.
[0066] From the above, the key point of the embodiment of the application is:
[0067] 1. Key event identification method based on hierarchical processing architecture: The embodiment of the application proposes a key event identification method based on a three-level architecture of "classification - clustering - discrimination"; the method combines a special semantic extraction method in the government affairs field, a semantic regularization hierarchical clustering (SRHC) method and a cluster entity keyword extraction technology, and through multi-stage collaborative processing of semantic vectorization, clustering optimization and key information verification, the processing efficiency and identification accuracy of the system are significantly improved.
[0068] 2. Special semantic extraction method in the government affairs field: First, regular expressions are constructed according to different reporting channels to filter out redundant content without actual semantics; then, a special stop word table in the government affairs field is constructed, and the general stop word library is expanded to include formatted words in government reporting; a multi-stage standardization cleaning process is introduced, including empty text filtering, interference symbol removal, word segmentation processing and term standardization; a similar keyword replacement mechanism is designed to unify the standardized terms and abbreviations specific to the government affairs field and eliminate semantic ambiguity; the semantic effectiveness after text vectorization is significantly improved, providing high-quality input for subsequent clustering analysis and improving identification accuracy.
[0069] 3. Semantic regularization hierarchical clustering method: The data distribution is optimized by standardizing the semantic similarity matrix; the clustering efficiency and stability are improved using mathematical statistical rules, and then the cluster merging process is dynamically controlled to ensure that the semantic similarity meets the preset threshold; the rapid identification ability of potential key events is significantly enhanced, and the identification efficiency is improved.
[0070] 4. Cluster entity keyword extraction and consistency verification mechanism: Accurately extract high-weight entity keywords; define the number of common keywords (CKC) and the uniqueness ratio (UR) as quantitative indicators, set the minimum threshold and maximum upper limit; filter clusters with insufficient keyword overlap or too dispersed distribution to ensure semantic consistency; significantly improve the accuracy of key event screening in complex scenarios.
[0071] The detailed steps of the embodiment of the application are as follows:
[0072] S1: Technology selection and architecture design
[0073] S1.1 adopts the embedding model paraphrase-multilingual-MiniLM-L12-v2 to support multi-language semantic representation and efficient text similarity calculation.
[0074] S1.2 uses a semantic regularized hierarchical clustering method (SRHC) for data clustering processing. The algorithm automatically divides the cluster number by dynamically adjusting the distance threshold, and uses the average linkage method to calculate the distance between clusters.
[0075] S1.3 uses textrank4keywords as an entity keyword extraction algorithm, which can effectively extract entity keywords such as names and place names.
[0076] S2: Text semantic extraction method for government affairs scenarios
[0077] S2.1 reads the reported data, filters out empty text or non-string content, cleans up special characters and unnecessary punctuation in the text, and then uses the jieba segmentation tool for text segmentation processing.
[0078] S2.2 first selects different regular expressions according to the channel source to extract the reported content, and then filters out meaningless words, single-character words, and pure number content according to the government domain exclusive stop word list to ensure that the remaining words have actual meaning.
[0079] S3: Initial classification of reported content
[0080] S3.1 uses a large language model to divide the data into multiple categories according to the predefined categories (natural disasters, fire safety, traffic safety, public safety, social safety, and contradictions and disputes), reducing the input data volume of the subsequent clustering algorithm, improving the overall process efficiency and robustness, and performing preliminary classification and annotation on the original data.
[0081] S4: Semantic vectorization and similarity modeling
[0082] S4.1 calls the semantic embedding Sentence-BERT model (paraphrase-multilingual-MiniLM-L12-v2) for each element in the classified category in the previous step to generate corresponding semantic embedding vectors with rich semantic representation capabilities, calculate the cosine distance between elements, and construct a cosine distance matrix.
[0083] S4.2 The matrix is then regularized to make the data distribution more consistent with the standard normal distribution, thereby improving clustering efficiency and stability. Similar clusters are merged layer by layer through average linkage method, and the distance threshold or the upper limit of the number of clusters is set artificially to control the clustering granularity. After clustering is completed, the results are mapped back to the original data structure, and each valid record is assigned a corresponding cluster number.
[0084] S5: Extracting entity keywords in the cluster
[0085] S5.1 For the text content in each cluster, the TextRank4Keyword algorithm is used to extract the top 5 entity keywords with the highest weights, including but not limited to location information and the names of parties involved.
[0086] S5.2 The semantic consistency within the cluster is quantified by calculating the number of common keywords (CKC) and the uniqueness ratio (UR). This mechanism sets a minimum common keyword threshold (MIN_COMMON_KEYWORDS) and a maximum uniqueness ratio upper limit (MAX_UNIQUE_RATIO) to effectively filter clusters with insufficient keyword overlap or excessively dispersed distribution.
[0087] S6: Cluster semantic consistency verification based on large language model
[0088] S6.1 Each cluster is traversed to check if there is exactly the same text content: if there is, it is directly determined to be a "key event" cluster. For clusters without exactly the same content: combined with the entity keyword extraction results, verify whether it meets the requirements of the number of common entity keywords and the uniqueness ratio, and eliminate invalid clusters to retain valid clusters.
[0089] S6.2 The content of the valid cluster is passed to the large language model for final review to confirm that the events within the cluster are different reporting versions of the same event.
[0090] S6.3 According to the review results, the specific description of the key event is finally output, and the identified key event is marked, and reported to the corresponding management personnel, notifying them to prioritize the handling or attention of the event for subsequent governance decision-making reference, ensuring the consistency and accuracy of the data.
[0091] Preferably, the vector embedding model of the embodiment of the application can be replaced by a bge or gte model; the clustering algorithm can be replaced by kmeans, DBSCAN; and the large language model can be replaced by qwen other series models.
[0092] In summary, the embodiment of the application proposes a key event identification method based on a hierarchical processing architecture, which can be used for screening key events in the government field. The method adopts a hierarchical processing architecture of "classification-cluster-discrimination", is based on the natural laws of "the same event has the same category" and "events of the same category have similar characteristics in the semantic space", and converts the processing of full data into the processing of multiple small batches of data through the classification preprocessing of event data, thereby significantly reducing the computational complexity. Theoretical analysis shows that the algorithm complexity is reduced from the original O(n 2 ) to O(m x k 2 ), where n is the total number of events, m is the number of event categories, and k is the average number of single-class events, and the processing efficiency of the system is improved.
[0093] In view of the highly specialized and formatted features of data in the government field, the embodiment of the application proposes a special semantic extraction method for the government field. By constructing a government exclusive stop word table, a regular expression, and a redundant expression filtering rule, and combining a multi-stage standardization cleaning process (including empty text filtering, interference symbol removal, word segmentation processing, and term standardization), the effectiveness of the text semantic information is significantly improved. Experiments show that this method improves the accuracy of text understanding and also improves the recognition rate of specific terms, effectively reduces the misjudgment rate caused by semantic ambiguity, provides a high-quality data basis for subsequent semantic embedding and clustering analysis, and thus greatly enhances the screening accuracy and efficiency of the system for "key events".
[0094] Meanwhile, the embodiment of the application improves the traditional hierarchical clustering algorithm and proposes a semantic regularization agglomerative hierarchical clustering method. By normalizing the semantic similarity matrix, the data distribution conforms to the standard normal distribution, and the clustering process is optimized. The improvement utilizes the mathematical statistical law, so that the system can maintain the clustering quality while greatly improving the processing efficiency. The clustering processing speed is improved, the clustering quality is improved, and the subsequent key event judgment is effectively supported, so that the system can more quickly and accurately identify potential key events.
[0095] The embodiment of the application also proposes a cluster-in-entity keyword extraction and consistency verification mechanism. After clustering, the text content in each cluster is subjected to in-depth semantic analysis, high-weight entity keywords related to key information such as positions and names are accurately extracted, and the semantic consistency in the cluster is quantified by calculating the number of common keywords (CKC) and the uniqueness ratio (UR). By setting the minimum common keyword threshold (MIN_COMMON_KEYWORDS) and the maximum uniqueness ratio upper limit (MAX_UNIQUE_RATIO), the mechanism effectively filters clusters with insufficient keyword overlap or excessively dispersed distribution, makes up for the technical shortcoming of traditional methods in the aspect of semantic comparison of key entities, and significantly improves the accuracy of key event screening in complex scenarios.
[0096] System embodiments
[0097] According to the embodiments of the present application, a key event discrimination system based on a hierarchical processing architecture is provided, Figure 5 is a key event discrimination system based on a hierarchical processing architecture according to the embodiments of the present application, as Figure 5 shown, the key event discrimination system based on a hierarchical processing architecture according to the embodiments of the present application specifically comprises:
[0098] The classification module 50 is configured to obtain reported events from different channels, perform data preprocessing on the reported events, obtain processed event texts, and classify the event texts.
[0099] The clustering module 52 is configured to cluster the classified texts by using a semantic regularization agglomerative hierarchical clustering method, and specifically configured to:
[0100] encode the elements in each class of text by using a semantic embedding model to generate corresponding semantic embedding vectors, calculate the cosine distance between each pair of texts based on the semantic embedding vectors by using a cosine similarity function, construct a symmetric form of the cosine distance matrix, perform regularization processing on the cosine distance matrix, cluster the regularized distance matrix by using an average linkage method, and map the clustering results back to the original data structure after clustering is completed, and assign a corresponding cluster number to each valid record.
[0101] The discrimination module 54 is configured to discriminate repeated events in the texts by using an intra-cluster entity keyword extraction and comparison mechanism according to the clustering results to obtain key event discrimination results, and specifically configured to:
[0102] iterate the clustering results, remove isolated clusters containing only single records in the clustering results, and judge the repeated content of clusters containing multiple records, if there are completely identical text fields, then directly determine that it is a repeated event, if there are no completely identical text fields, then extract the top k high-weight entity keywords of each record to obtain a keyword set of each record, calculate the number of common keywords and the uniqueness ratio in the cluster based on the keyword set, if the number of common keywords is less than a preset minimum common keyword threshold, and the uniqueness ratio is greater than a preset maximum uniqueness ratio upper limit, then the cluster is removed, and the remaining valid clusters are retained, and the remaining valid clusters are input into a large language model for discrimination to obtain the final key event discrimination results.
[0103] The embodiments of the present application are system embodiments corresponding to the above-mentioned method embodiments, and the specific operations of each module can be understood with reference to the description of the method embodiments, which will not be repeated here.
[0104] Compared with the prior art, the embodiment of the application has the following beneficial effects:
[0105] 1. More efficient processing flow: The embodiment of the application adopts a hierarchical processing architecture of "classification-clustering-discrimination", based on the natural laws of "events of the same category have the same category" and "events of the same category have similar characteristics in the semantic space", and through classification preprocessing, the total data is decomposed into multiple small batches of data processing, which significantly reduces the computational complexity. Theoretical analysis shows that the algorithm complexity is reduced from the original O(n 2 ) to O(m x k 2 ), where n is the total number of events, m is the number of event categories, and k is the average number of single-class events. The processing efficiency of the system is effectively improved, and the ability to support real-time response in the urban governance scenario is achieved, solving the efficiency bottleneck problem of traditional methods in large-scale data processing.
[0106] 2. Higher accuracy: The embodiment of the application proposes a special semantic extraction method for the government field, which constructs a government-exclusive stop word list, a regular expression rule, and a redundant expression filtering mechanism, and combines a multi-stage standardization cleaning process (such as empty text filtering, interference symbol removal, word segmentation processing, and term standardization), significantly improving the effectiveness of text semantic information. Experimental results show that this method improves the accuracy of text understanding, and also improves the recognition rate of specific terms, effectively reduces the misjudgment rate caused by semantic ambiguity, provides a high-quality data basis for subsequent semantic embedding and clustering analysis, and greatly enhances the screening accuracy and reliability of the system for key events.
[0107] 3. Faster clustering performance: The embodiment of the application improves the traditional hierarchical clustering algorithm and proposes a semantic regularization agglomerative hierarchical clustering method, which standardizes the semantic similarity matrix to make its distribution conform to the standard normal distribution, thereby optimizing the clustering process. This method uses mathematical statistical laws to significantly improve the processing efficiency while maintaining the quality of clustering. Compared with the prior art, the embodiment of the application can more quickly and accurately identify potential key events.
[0108] 4. More rigorous result determination: The embodiment of the present application proposes a cluster entity keyword extraction and consistency verification mechanism. After clustering is completed, the text content in each cluster is subjected to in-depth semantic analysis, high-weight entity keywords related to position, name and other key information are accurately extracted, and the semantic consistency in the cluster is quantified by calculating the number of common keywords (CKC) and the uniqueness ratio (UR). The mechanism sets a minimum common keyword threshold (MIN_COMMON_KEYWORDS) and a maximum uniqueness ratio upper limit (MAX_UNIQUE_RATIO), effectively filtering clusters with insufficient keyword overlap or excessively dispersed distribution, making up for the technical short board of traditional methods in the aspect of lacking semantic comparison of key entities, and significantly improving the accuracy of key event screening in complex scenarios.
[0109] Device embodiment one
[0110] The embodiment of the present application provides an electronic device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the computer program is executed by the processor to implement the steps as described in the method embodiment.
[0111] Device embodiment two
[0112] The embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores an implementation program of information transmission, and the program is executed by a processor to implement the steps as described in the method embodiment.
[0113] The computer readable storage medium described in the embodiment includes but is not limited to ROM, RAM, magnetic disk or optical disk, etc.
[0114] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for identifying key events based on a hierarchical processing architecture, characterized in that... The method comprises the following steps: obtaining reported events from different channels, data preprocessing the reported events to obtain processed event texts, and classifying the event texts; adopting a semantic regularization condensed hierarchical clustering method to cluster the classified texts, specifically including: adopting a semantic embedding model to encode elements within each class of text to generate corresponding semantic embedding vectors, calculating the cosine distance between each pair of texts based on the semantic embedding vectors through a cosine similarity function, constructing a symmetric form of the cosine distance matrix, and performing regularization processing on the cosine distance matrix, clustering the regularized distance matrix through the average linkage method, and mapping the clustering results back to the original data structure after clustering is completed, and assigning a corresponding cluster number to each valid record; distinguishing repeated events in the texts according to the clustering results using an entity keyword extraction and comparison mechanism to obtain a key event discrimination result.
2. The method of claim 1, wherein, The data preprocessing of the reported events specifically includes: data cleaning and filtering, text standardization processing, domain-specific processing, and validity verification of the reported events; The data cleaning and filtering includes filtering empty texts, removing non-string content, and removing interference symbols; The text standardization processing includes constructing corresponding regular expressions according to different channels, extracting key content from the reported events through the corresponding regular expressions, and performing word segmentation processing on the texts using a word segmentation tool; The domain-specific processing is to screen the texts using a pre-constructed exclusive stopword list in the government affairs field to remove formatted words, single-character words, and pure number fields without actual semantics; The validity verification is to check whether the processed text record is empty, and if so, to mark and remove the text record.
3. The method of claim 1, wherein, The classification of the event texts specifically includes: preliminarily classifying the event texts according to event categories through a large language model; wherein the event categories include natural disasters, fire safety, traffic safety, public safety, social safety, and disputes.
4. The method of claim 1, wherein, The distinguishing of repeated events in the texts according to the clustering results using an entity keyword extraction and comparison mechanism to obtain a key event discrimination result specifically includes: iterating the clustering results, removing isolated clusters containing only single records in the clustering results, and judging the repeated content of clusters containing multiple records, if there are completely identical text fields, then directly judging as repeated events; if there are no completely identical text fields, then extracting the top k high-weight entity keywords of each record to obtain a keyword set of each record, calculating the number of common keywords and the uniqueness ratio in the cluster based on the keyword set, if the number of common keywords is less than a preset minimum common keyword threshold, and the uniqueness ratio is greater than a preset maximum uniqueness ratio upper limit, then removing the cluster, and retaining the remaining valid clusters; inputting the remaining valid clusters into a large language model for discrimination to obtain a final key event discrimination result.
5. A key event discrimination system based on a hierarchical processing architecture, characterized by The method comprises the following steps: The classification module is configured to obtain reported events from different channels, perform data preprocessing on the reported events, obtain processed event texts, and classify the event texts. The clustering module is configured to perform clustering on the classified texts by using a semantic regularization agglomerative hierarchical clustering method, specifically configured to: encode elements in each type of text by using a semantic embedding model to generate corresponding semantic embedding vectors, calculate the cosine distance between each pair of texts based on the semantic embedding vectors by using a cosine similarity function, construct a symmetric form of the cosine distance matrix, perform regularization processing on the cosine distance matrix, perform clustering on the regularized distance matrix by using an average linkage method, map the clustering results back to the original data structure after the clustering is completed, and assign a corresponding cluster number to each valid record. The discrimination module is configured to discriminate repeated events in the texts according to the clustering results by using an entity keyword extraction and comparison mechanism in the cluster to obtain a key event discrimination result.
6. The system of claim 5, wherein, The discrimination module is specifically configured to: perform iteration on the clustering results, remove isolated clusters that only contain a single record in the clustering results, and perform repeated content judgment on clusters containing multiple records, if there are completely identical text fields, then directly determine that the events are repeated, if there are no completely identical text fields, then extract the top-k high-weight entity keywords of each record to obtain a keyword set of each record, calculate the number of common keywords and the uniqueness ratio in the cluster based on the keyword set, if the number of common keywords is less than a preset minimum common keyword threshold, and the uniqueness ratio is greater than a preset maximum uniqueness ratio upper limit, then the cluster is removed, and the remaining valid clusters are retained, and the remaining valid clusters are input into a large language model for discrimination to obtain a final key event discrimination result.
7. An electronic device, comprising: The computer program stored in the memory and executable on the processor implements the steps of the key event discrimination method based on the hierarchical processing architecture according to any one of claims 1-4 when the processor executes the computer program. The computer readable storage medium stores an information transmission implementation program, and the program implements the steps of the key event discrimination method based on the hierarchical processing architecture according to any one of claims 1-4 when the processor executes the program.
8. A computer-readable storage medium, characterized in that,