Automatic news event extraction and combination system based on chapter-level text analysis

Abstract

The invention discloses an automatic news event extraction and combination system based on chapter-level text analysis, which is characterized in that text noise is cleaned and paragraphs are divided through a preprocessing module, and strong event candidates are screened by combining multi-modal conjoint analysis of picture titles and paragraph texts and utilizing semantic similarity calculation and named entity recognition technologies; a multi-task learning framework based on a Transform architecture is adopted to cooperatively extract elements such as time, places, figures, events, causes and passes, and the comprehensiveness and accuracy of event element extraction are improved; and the event similarity is calculated through a Bi-Encoder model, and the events are combined and sorted in combination with timestamps, so that an event chain with clear logic is generated. The system breaks through the limitation of existing sentence-level or paragraph-level analysis, remarkably improves the comprehensiveness, consistency and coherence of news event identification through chapter-level global context, data multi-mode combination and intelligent event chain construction, and has wide application value in the fields of news monitoring, public opinion analysis and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of event extraction and analysis of news texts, and in particular to an automatic news event extraction and merging system based on paragraph-level text analysis. Background Art

[0002] With the rapid development of the Internet and social media, the speed and volume of news information dissemination have exploded. How to quickly and accurately extract key event information from massive news texts has become an important research direction in the field of information processing. Usually, event extraction technology is mainly used to extract time, place, people, cause and process of events from texts, and is widely used in news monitoring, public opinion analysis, information retrieval and other scenarios. This technology can help users quickly grasp the full picture of complex events by converting unstructured text into structured event information.

[0003] In the existing technology, the working principle of event extraction is mainly based on the application of natural language processing technology and deep learning models. Typical event extraction methods usually use sentences or paragraphs as the analysis granularity to extract event elements from the local context. However, under this technical framework, the industry has the following problems: 1. Limited granularity leads to information loss. Traditional methods are mostly limited to sentence-level or paragraph-level analysis, which can only capture event elements within a local range and cannot fully extract event information from the context of the entire article. For example, a news report may describe different parts of an event across paragraphs, such as the time mentioned in the first paragraph and the location mentioned in the third paragraph. Traditional methods find it difficult to integrate these scattered information, resulting in incomplete event element extraction.

[0004] 2. Insufficient multimodal information processing capabilities. That is, in news texts, picture titles often carry important supplementary information. For example, picture captions may contain key information such as time, place or people. However, traditional technologies usually ignore this type of non-text content or process it in isolation, and cannot form effective collaborative analysis with the main text content, resulting in incomplete information mining.

[0005] 3. Difficulty in identifying and merging similar events. In news reports, multiple articles often revolve around the same event or related events. For example, different media describe a news event from different perspectives. Traditional event merging technology usually relies on simple keyword matching or similarity calculation, which is difficult to handle texts that are highly semantically related but have different wording. In addition, for similar events across articles, existing technologies often lack a time sorting mechanism from a global perspective, resulting in the generated event chain being incoherent or lacking logic.

[0006] In order to solve the above problems, the industry has tried to adopt the following improvement methods, but these methods still have new challenges, such as introducing deep learning models to improve extraction accuracy: for example, using the BERT model to extract features from text and improve the accuracy of local event element extraction. However, this method is still limited to a single granularity and cannot effectively integrate chapter-level contextual information. Independent analysis of image titles: Some technical solutions attempt to perform independent entity recognition on image titles, but due to the lack of linkage analysis with the main text content, it is still difficult to fully utilize the synergy of multimodal data. Optimization of event aggregation algorithms: Some studies have introduced clustering algorithms to improve the accuracy of merging similar events, but these methods are not robust enough when faced with diverse news expressions, and have weak processing of temporal logic in event chain construction.

[0007] Therefore, how to comprehensively extract chapter-level event elements, effectively integrate multimodal information and generate a logically coherent event chain has become a technical problem to be solved by the present invention. Summary of the invention

[0008] The technical problem solved by the present invention is to provide a news event automatic extraction and merging system based on paragraph-level text analysis in response to the defects existing in the above-mentioned prior art, so as to solve the problems of insufficient comprehensiveness of event recognition, insufficient multimodal information integration capability, and poor merging and logic of similar events raised in the above-mentioned background technology.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows: A news event automatic extraction and merging system based on chapter-level text analysis, the system comprising: The preprocessing module is used to clean up irrelevant information in the news text, remove special characters and noise, and divide the text into paragraphs based on the text structure and grammatical features; The image title and paragraph text joint analysis module includes: A picture title text extraction unit, used to parse and extract picture title text from news articles; An event candidate screening unit, used for inputting the picture title text and paragraph text into a pre-trained BERT model, generating corresponding semantic vectors, and calculating their cosine similarity; If the similarity is higher than the set threshold, the corresponding paragraph text is marked as a strong event candidate; otherwise, the NER model is used to identify important entities in the picture title text, including but not limited to time, people, and places, and candidate events are further screened based on the entity recognition results; Paragraph text analysis module, including: The keyword extraction unit vectorizes the paragraph text through the TF-IDF model and extracts the core keywords; Entity recognition unit, which uses the NER model to identify important entities in the paragraph text, including but not limited to time, people, and places; An event candidate determination unit is used to compare the extracted keywords with the identified entities. If there is an overlap between the keywords and the entities, the paragraph text is marked as a strong event candidate. Event element extraction module, including: A multi-task learning framework designed based on the Transformer architecture is used to simultaneously extract event elements including but not limited to time, person, place, event, cause or process from strong event candidate texts, where: The underlying feature extraction of the multi-task learning framework is based on the pre-trained BERT model and fine-tuned through transfer learning; The framework adds a specific sequence annotation layer for each event element task, and combines the conditional random field layer with the cross entropy loss function to improve the performance of event element extraction.

[0010] Event merging and sorting module, including: The similar event recognition unit generates embedding vectors for event elements and text based on the Bi-Encoder model, and recognizes similar event pairs by calculating semantic similarity; An event merging unit, used to merge elements of similar events; The event chain generation unit sorts the merged events according to the timestamp information of the events and generates an event chain on the timeline.

[0011] As a further solution of the present invention, the event candidate screening unit of the picture title and paragraph text joint analysis module is implemented by the following steps: Step 1: Use the pre-trained BERT model to embed the image title text and paragraph text into semantic vectors; Step 2: Calculate the cosine similarity between the embedded vectors. If the similarity is higher than the set threshold, mark the paragraph text as a strong event candidate. Step 3: For the image title texts whose similarity is lower than the set threshold, the NER model is used to perform entity recognition, and event candidates are further screened based on the extracted key entities of time, person, place and / or organization name; The event chain generation unit of the event merging and sorting module is implemented by the following steps: S1, use the pre-trained Bi-Encoder model to embed event text and elements and generate corresponding semantic vectors; S2, calculates the similarity between events based on semantic vectors. If the similarity is higher than the set threshold, similar events are merged; S3, sorts the merged events in chronological order according to the timestamp information of the events to generate a logically complete event chain.

[0012] As a further solution of the present invention, the entity recognition unit of the paragraph text analysis module recognizes the time, person, place, and organization name in the paragraph text through the NER model, and fine-tunes the entity type.

[0013] As a further solution of the present invention, the event element extraction module uses a combination of a conditional random field layer and a cross entropy loss function to perform sequence labeling on event elements.

[0014] As a further solution of the present invention, when merging events, the event merging and sorting module calculates the consistency of event elements and preferentially merges events with the same time, person or location.

[0015] As a further solution of the present invention, the picture title text extraction unit uses an HTML parsing tool to efficiently parse the picture titles in the news article.

[0016] As a further solution of the present invention, the event candidate screening unit adapts to the text characteristics of different news sources through a mechanism of dynamically adjusting the cosine similarity threshold.

[0017] As a further solution of the present invention, the multi-task learning framework in the event element extraction module utilizes a shared underlying representation approach.

[0018] As a further solution of the present invention, the threshold adjustment mechanism of the event candidate screening unit in the picture title and paragraph text joint analysis module includes: By analyzing the semantic complexity of the paragraphs of the input text, the variance of the semantic feature vector of each paragraph is calculated; When the variance of the semantic feature vector is lower than the preset value, the screening threshold of the cosine similarity is automatically lowered to enhance the sensitivity of event candidate screening; When the variance of the semantic feature vector is higher than the preset value, the screening threshold of the cosine similarity is automatically increased to improve the accuracy of event candidate screening.

[0019] As a further solution of the present invention, the picture title text extraction unit of the picture title and paragraph text joint analysis module uses the following method to perform HTML parsing: Extract all the content in HTML document using BeautifulSoup The alt attribute of a tag and the adjacent <figcaption>Label content; Regularize the extracted title text, including removing HTML entities, special characters and extra spaces; Based on natural language processing methods, image title texts in different languages ​​or encoding formats are uniformly processed.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. The deep integration of chapter-level event analysis and multimodal joint screening significantly improves the comprehensiveness of event recognition: The multimodal joint screening of chapter-level text analysis and picture titles in the present invention organically combines semantic similarity calculation with entity recognition technology, which not only realizes the capture of complex event elements within the chapter level, but also makes up for the important event clues that may be missed by single text analysis through the synergy of picture titles and text content. Compared with the existing method that simply relies on paragraph or sentence-level analysis, the present invention can capture event information from a wider context, greatly improving the comprehensiveness of event recognition.

[0021] 2. Collaborative extraction of event elements based on multi-task learning to build a highly accurate and rich event structure: The multi-task learning framework of the present invention not only supports the separate extraction of event elements such as time, place, and people, but also uses the mechanism of sharing underlying features to enable each element task to be collaboratively optimized with each other, further improving the extraction accuracy and robustness of the overall model. Compared with the traditional single-task model, collaborative learning significantly reduces the inconsistency problem between different elements, making the extracted event structure more complete and reliable.

[0022] 3. Intelligent event merging and sorting to achieve a dynamic balance between information comprehensiveness and consistency: Through the rich event elements generated by multimodal analysis and multi-task extraction, the present invention further combines semantic similarity calculation and timestamp sorting mechanism to achieve efficient merging and sorting of similar events across articles and multiple perspectives. Through this collaborative process, the event chain constructed by the present invention is not only logically clear but also semantically coherent, thus breaking through the problems of low merging accuracy of similar events and fragmented event chain information in the prior art.

[0023] 4. Dynamic threshold control and coordinated optimization of modular design ensure flexibility and applicability in complex scenarios: The present invention dynamically adjusts the similarity screening threshold between the image title and the paragraph text, and can achieve flexible screening according to data characteristics in diverse scenarios where text information is sparse or redundant. Combined with the modular design architecture, each functional module has the ability to independently expand and optimize while ensuring synergy, so that the present invention can efficiently adapt to a variety of application scenarios such as news monitoring, public opinion analysis and information retrieval.

[0024] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 It is a schematic diagram of the process of identifying event candidates based on picture titles according to the present invention.

[0027] Figure 2 It is a schematic diagram of the process of identifying event candidates based on paragraph text according to the present invention.

[0028] Figure 3 This is a schematic diagram of the process of extracting elements and performing event aggregation on candidate paragraph texts and picture title texts in the present invention. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] See also Figure 1 —3. In an embodiment of the present invention, a system for automatically extracting and merging news events based on chapter-level text analysis comprises: The preprocessing module is used to clean up irrelevant information in the news text, remove special characters and noise, and divide the text into paragraphs based on the text structure and grammatical features; The image title and paragraph text joint analysis module includes: A picture title text extraction unit, used to parse and extract picture title text from news articles; An event candidate screening unit, used for inputting the picture title text and paragraph text into a pre-trained BERT model, generating corresponding semantic vectors, and calculating their cosine similarity; If the similarity is higher than the set threshold, the corresponding paragraph text is marked as a strong event candidate; otherwise, the NER model is used to identify important entities in the picture title text, including but not limited to time, people, and places, and candidate events are further screened based on the entity recognition results; Paragraph text analysis module, including: The keyword extraction unit vectorizes the paragraph text through the TF-IDF model and extracts the core keywords; Entity recognition unit, which uses the NER model to identify important entities in the paragraph text, including but not limited to time, people, and places; An event candidate determination unit is used to compare the extracted keywords with the identified entities. If there is an overlap between the keywords and the entities, the paragraph text is marked as a strong event candidate. Event element extraction module, including: A multi-task learning framework designed based on the Transformer architecture is used to simultaneously extract event elements including but not limited to time, person, place, event, cause or process from strong event candidate texts, where: The underlying feature extraction of the multi-task learning framework is based on the pre-trained BERT model and fine-tuned through transfer learning; The framework adds a specific sequence annotation layer for each event element task, and combines the conditional random field layer with the cross entropy loss function to improve the performance of event element extraction.

[0031] Event merging and sorting module, including: The similar event recognition unit generates embedding vectors for event elements and text based on the Bi-Encoder model, and recognizes similar event pairs by calculating semantic similarity; An event merging unit, used to merge elements of similar events; The event chain generation unit sorts the merged events according to the timestamp information of the events and generates an event chain on the timeline.

[0032] As a further solution of the present invention, the event candidate screening unit of the picture title and paragraph text joint analysis module is implemented by the following steps: Step 1: Use the pre-trained BERT model to embed the image title text and paragraph text into semantic vectors; Step 2: Calculate the cosine similarity between the embedded vectors. If the similarity is higher than the set threshold, mark the paragraph text as a strong event candidate. Step 3: For the image title texts whose similarity is lower than the set threshold, the NER model is used to perform entity recognition, and event candidates are further screened based on the extracted key entities of time, person, place and / or organization name; The event chain generation unit of the event merging and sorting module is implemented by the following steps: S1, use the pre-trained Bi-Encoder model to embed event text and elements and generate corresponding semantic vectors; S2, calculates the similarity between events based on semantic vectors. If the similarity is higher than the set threshold, similar events are merged; S3, sorts the merged events in chronological order according to the timestamp information of the events to generate a logically complete event chain.

[0033] As a further solution of the present invention, the entity recognition unit of the paragraph text analysis module recognizes the time, person, place, and organization name in the paragraph text through the NER model, and fine-tunes the entity type.

[0034] As a further solution of the present invention, the event element extraction module uses a combination of a conditional random field layer and a cross entropy loss function to perform sequence labeling on event elements.

[0035] As a further solution of the present invention, when merging events, the event merging and sorting module calculates the consistency of event elements and preferentially merges events with the same time, person or location.

[0036] As a further solution of the present invention, the picture title text extraction unit uses an HTML parsing tool to efficiently parse the picture titles in the news article.

[0037] As a further solution of the present invention, the event candidate screening unit adapts to the text characteristics of different news sources through a mechanism of dynamically adjusting the cosine similarity threshold.

[0038] As a further solution of the present invention, the multi-task learning framework in the event element extraction module utilizes a shared underlying representation approach.

[0039] As a further solution of the present invention, the threshold adjustment mechanism of the event candidate screening unit in the picture title and paragraph text joint analysis module includes: By analyzing the semantic complexity of the paragraphs of the input text, the variance of the semantic feature vector of each paragraph is calculated; When the variance of the semantic feature vector is lower than the preset value, the screening threshold of the cosine similarity is automatically lowered to enhance the sensitivity of event candidate screening; When the variance of the semantic feature vector is higher than the preset value, the screening threshold of the cosine similarity is automatically increased to improve the accuracy of event candidate screening.

[0040] As a further solution of the present invention, the picture title text extraction unit of the picture title and paragraph text joint analysis module uses the following method to perform HTML parsing: Extract all the content in HTML document using BeautifulSoup The alt attribute of a tag and the adjacent <figcaption>Label content; Regularize the extracted title text, including removing HTML entities, special characters and extra spaces; Based on natural language processing methods, image title texts in different languages ​​or encoding formats are uniformly processed.

[0041] Embodiment 1:

[0042] In order to more intuitively demonstrate the specific application of the system for automatic extraction and merging of news events based on paragraph-level text analysis of the present invention, a detailed description is given below in conjunction with actual application scenarios.

[0043] Suppose a series of related emergencies occurred on the same day in a certain place, including earthquakes, aftershocks, rescue operations, and social reactions. When processing this series of reports, traditional news event extraction technology may not be able to extract complete event elements due to the diversity of reporting sources, different description angles, and information scattered in different paragraphs. Even the same event may be recorded repeatedly, affecting the comprehensiveness and accuracy of the information.

[0044] When the system of the present invention is used for processing, the system first receives report texts from multiple news sources. These texts usually contain text content and supporting pictures and picture titles. For example, a report describes the time, location and initial damage of an earthquake, with a picture showing the epicenter and its title "A 7.0 magnitude earthquake occurred in the center of XX at 8 a.m."

[0045] The system's preprocessing module first cleans up irrelevant information and special characters in the text and divides the text into paragraphs to ensure that the basic data for subsequent processing is clean and structured. Next, the image title and paragraph text joint analysis module begins to work. The system extracts the title text of all images from each news article, uses HTML parsing tools such as BeautifulSoup to parse HTML documents, and extracts The alt attribute of the tag and <figcaption>Tag content and perform regularization to remove HTML entities, special characters, and extra spaces.

[0046] Taking the picture caption "At 8 am, a 7.0 magnitude earthquake occurred in the center of XX city" as an example, the system uses the pre-trained BERT model to embed the title text and the body text of each paragraph into a semantic vector and calculates the cosine similarity. Assuming that a paragraph describes the specific time, location and magnitude of the earthquake in detail, after similarity calculation, it is found that the similarity is higher than the set threshold (such as 0.85), the paragraph text is marked as a strong event candidate. In addition, if some picture captions fail to match any paragraph text with a high enough similarity, the system will use the NER model to identify important entities in the title, such as the time "8 am" and the location "XX city center", and further screen and mark them as strong event candidates.

[0047] Subsequently, the paragraph text analysis module performs keyword extraction and entity recognition on all paragraphs. For example, the TF-IDF model is used to extract keywords such as "earthquake", "rescue", and "aftershock", and entities such as time, person, and place are recognized through the NER model. When there is an overlap between keywords and entities, the system marks the paragraph as a strong event candidate.

[0048] Entering the event element extraction module, the system uses a multi-task learning framework based on the Transformer architecture to simultaneously extract elements such as time, people, places, events, causes, and processes from strong event candidate texts. Feature extraction is performed through the pre-trained BERT model, and the conditional random field layer is combined with the cross entropy loss function to improve the accuracy of event element extraction. For example, from "At 8 am, a 7.0 magnitude earthquake occurred in the center of XX city", the system accurately identifies the time "8 am", the location "XX city center", and the event "7.0 magnitude earthquake occurred".

[0049] In the event merging and sorting module, the system embeds and vectorizes the event elements extracted from different reports, and uses the pre-trained Bi-Encoder model to calculate the semantic similarity between events. For example, the earthquake and its subsequent rescue operations described in multiple reports will be identified as similar events because they share similar time, location, and event elements, and will be merged into a complete chain of events. At the same time, the system sorts the merged events in chronological order according to the timestamp information to generate a logically coherent chain of events, such as the occurrence of an earthquake, aftershocks, rescue operations, social reactions, etc.

[0050] Through the above processing, the system can comprehensively and accurately extract and integrate key event information from multi-source and multi-modal news reports to generate a structured event chain. This not only solves the shortcomings of traditional methods in chapter-level analysis, information integration, and event merging, but also significantly improves the accuracy of event extraction and the comprehensiveness of information through multi-task learning and intelligent merging algorithms.

[0051] Specifically, when processing earthquake events, the system in this embodiment can automatically identify and integrate the time, location, event and other elements from different reports, avoiding information duplication and omission. For example, one report focuses on the time and location of the earthquake, while another details the progress of the rescue operation. The system generates a complete event chain including the earthquake, rescue operations and their progress by associating and merging event elements. This capability shows significant technical advantages in the face of complex and changing news environments, and improves the efficiency and accuracy of news event monitoring and analysis.

[0052] In addition, the present invention can flexibly adjust the similarity screening threshold according to the text characteristics of different news sources through a dynamic threshold control mechanism, ensuring that relevant events can still be efficiently screened out in scenarios where information is sparse or redundant. This mechanism is manifested in practical applications as follows: when certain reports are concise and information is dense, the system can automatically increase the similarity threshold to reduce false matches; while in reports with more dispersed information or diverse expressions, the system lowers the threshold to ensure that important event information is not missed.

[0053] To sum up, through application in specific scenarios, this embodiment fully demonstrates the effectiveness and superiority of the automatic news event extraction and merging system based on paragraph-level text analysis of the present invention in actual operation, which not only solves many difficult problems in the prior art, but also significantly improves the comprehensiveness, accuracy and logical coherence of event information.

[0054] Embodiment 2:

[0055] This embodiment demonstrates the specific application scenario and implementation method of the present invention in public opinion analysis, and solves the limitations of existing public opinion analysis technology in information integration and event logic by combining chapter-level text analysis, multimodal data processing and event chain construction.

[0056] Take a public opinion incident caused by a product problem of a company as an example. Consumers posted a large number of related articles and pictures on different media and social platforms. The content of the reports included the time, location, cause of the incident, company statement and subsequent handling measures. This information is scattered and has diverse semantic expressions. Traditional analysis technology is difficult to integrate these cross-platform and cross-document public opinion data in a short period of time, and thus cannot quickly provide decision support. The specific implementation steps are as follows: 1. Data collection and preprocessing: The system collects relevant articles and social media content through crawler technology, including the text and related image captions. For example, the collected article content may include: Paragraph 1: "On November 5, a company's products were found to have quality problems involving multiple batches." Paragraph 2: "Consumers posted multiple photos of damaged products on social platforms, accompanied by text descriptions." The system cleans this data, removes irrelevant symbols and noise information, and divides the text based on paragraph structure.

[0057] 2. Joint analysis of image titles and paragraph text: The system extracts the titles of all relevant images, such as "Damaged product packaging" and "Accident scene on November 5", and calculates semantic similarity in combination with paragraph text. If the similarity between the image title and a paragraph of text is higher than a threshold (such as 0.85), the system marks the paragraph of text as a strong event candidate; if the similarity is lower, the system uses the NER model to identify the time, place, and key nouns in the image title, and uses this information as a supplementary candidate event. For example, the image title "Accident scene" does not directly match the text, but "accident" and "scene" are extracted through entity recognition and associated with paragraph 2.

[0058] 3. Paragraph text analysis and event candidate screening: The system uses the TF-IDF model to extract paragraph keywords, such as "quality issues", "batch", "accident scene", etc., and combines the NER model to identify entities such as "November 5" and "multiple batches". If the keyword and the entity are consistent or contain a relationship, the paragraph is marked as a strong event candidate.

[0059] 4. Event element extraction and structured representation: Input the selected strong event candidate texts into the Transformer-based multi-task learning framework to collaboratively extract elements such as time, location, event, cause, and process. For example: From "On November 5, a company's products were exposed to quality problems", extract the time "November 5", the place "a company", the event "product quality problem", and the cause "use of substandard materials".

[0060] By sharing underlying features and joint optimization strategies, the model ensures strong consistency and high accuracy among the extracted elements.

[0061] 5. Merge similar events and build event chains: The system uses the Bi-Encoder model to calculate the semantic similarity between event elements and merge events with high similarity. For example: Event 1: Time "November 5", location "a company", event "product quality issue".

[0062] Event 2: Time "November 5", location "accident scene", event "consumer posted pictures".

[0063] Through semantic similarity and timestamp information, the system merges the two events and constructs a public opinion event chain: "Product problems were exposed on November 5 - consumers reported the problem - the company issued a statement."

[0064] The present invention achieves efficient integration of cross-modal information through joint analysis of image titles and paragraph texts, fills the gap between image information and text in the prior art, and significantly improves the comprehensiveness of public opinion analysis. Through paragraph-level analysis and semantic similarity calculation, the system generates a clear public opinion event chain, allowing users to quickly grasp the causes and consequences of events and dynamic developments. At the same time, it adapts to a variety of public opinion data sources and performs well in processing complex public opinion scenarios with diverse semantics and dispersed expressions, providing an efficient solution for corporate crisis management and public opinion response.

[0065] In the present invention, unless otherwise clearly specified and limited, the terms "installation", "setting", "connection", "fixation", "screwing" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0066] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.< / figcaption> < / figcaption> < / figcaption>

Claims

1. A news event automatic extraction and merging system based on chapter-level text analysis, characterized in that: The system comprises: The preprocessing module is used to clean up irrelevant information in the news text, remove special characters and noise, and divide the text into paragraphs based on the text structure and grammatical features; Image title and paragraph text joint analysis module, including: A picture title text extraction unit, used to parse and extract picture title text from news articles; An event candidate screening unit, used for inputting the picture title text and paragraph text into a pre-trained BERT model, generating corresponding semantic vectors, and calculating their cosine similarity; If the similarity is higher than the set threshold, the corresponding paragraph text is marked as a strong event candidate; otherwise, the NER model is used to identify important entities in the picture title text, including but not limited to time, people, and places, and candidate events are further screened based on the entity recognition results; Paragraph text analysis module, including: The keyword extraction unit vectorizes the paragraph text through the TF-IDF model and extracts the core keywords; Entity recognition unit, which uses the NER model to identify important entities in the paragraph text, including but not limited to time, people, and places; An event candidate determination unit is used to compare the extracted keywords with the identified entities. If there is an overlap between the keywords and the entities, the paragraph text is marked as a strong event candidate. Event element extraction module, including: A multi-task learning framework designed based on the Transformer architecture is used to simultaneously extract event elements including but not limited to time, person, place, event, cause or process from strong event candidate texts, where: The underlying feature extraction of the multi-task learning framework is based on the pre-trained BERT model and fine-tuned through transfer learning; The framework adds a specific sequence annotation layer for each event element task, and combines the conditional random field layer with the cross entropy loss function to improve the performance of event element extraction; Event merging and sorting module, including: The similar event recognition unit generates embedding vectors for event elements and text based on the Bi-Encoder model, and recognizes similar event pairs by calculating semantic similarity; An event merging unit, used to merge elements of similar events; The event chain generation unit sorts the merged events according to the timestamp information of the events and generates an event chain on the timeline.

2. According to claim 1, a news event automatic extraction and merging system based on chapter-level text analysis is characterized in that: The event candidate screening unit of the picture title and paragraph text joint analysis module is implemented by the following steps: Step 1: Use the pre-trained BERT model to embed the image title text and paragraph text into semantic vectors; Step 2: Calculate the cosine similarity between the embedded vectors. If the similarity is higher than the set threshold, mark the paragraph text as a strong event candidate. Step 3: For the image title texts whose similarity is lower than the set threshold, the NER model is used to perform entity recognition, and event candidates are further screened based on the extracted key entities of time, person, place and / or institution name; The event chain generation unit of the event merging and sorting module is implemented by the following steps: S1, use the pre-trained Bi-Encoder model to embed event text and elements and generate corresponding semantic vectors; S2, calculates the similarity between events based on semantic vectors. If the similarity is higher than the set threshold, similar events are merged; S3, sorts the merged events in chronological order according to the timestamp information of the events to generate a logically complete event chain.

3. The system for automatically extracting and merging news events based on chapter-level text analysis according to claim 1 is characterized in that: The entity recognition unit of the paragraph text analysis module uses the NER model to identify the time, person, place, and organization names in the paragraph text and fine-tune the entity type.

4. The system for automatically extracting and merging news events based on chapter-level text analysis according to claim 1 is characterized in that: The event element extraction module uses a combination of conditional random field layer and cross entropy loss function to sequence event elements.

5. The system for automatically extracting and merging news events based on chapter-level text analysis according to claim 1 is characterized in that: When merging events, the event merging and sorting module calculates the consistency of event elements and gives priority to merging events with the same time, person or location.

6. The system for automatically extracting and merging news events based on chapter-level text analysis according to claim 1 is characterized in that: The picture title text extraction unit uses HTML parsing tools to efficiently parse the picture titles in news articles.

7. The system for automatically extracting and merging news events based on chapter-level text analysis according to claim 1 is characterized in that: The event candidate screening unit adapts to the text characteristics of different news sources by dynamically adjusting the cosine similarity threshold.

8. The system for automatically extracting and merging news events based on chapter-level text analysis according to claim 1 is characterized in that: The multi-task learning framework in the event feature extraction module utilizes the shared underlying representation approach.

9. The system for automatically extracting and merging news events based on chapter-level text analysis according to claim 1 is characterized in that: The threshold adjustment mechanism of the event candidate screening unit in the joint analysis module of picture title and paragraph text includes: By analyzing the semantic complexity of the paragraphs of the input text, the variance of the semantic feature vector of each paragraph is calculated; When the variance of the semantic feature vector is lower than the preset value, the screening threshold of the cosine similarity is automatically lowered to enhance the sensitivity of event candidate screening; When the variance of the semantic feature vector is higher than the preset value, the screening threshold of the cosine similarity is automatically increased to improve the accuracy of event candidate screening.

10. The system for automatically extracting and merging news events based on chapter-level text analysis according to claim 1 is characterized in that: The picture title text extraction unit of the picture title and paragraph text joint analysis module uses the following method to parse HTML: Extract all the content in HTML document using BeautifulSoup The alt attribute of a tag and the adjacent <figcaption> Label content;< / figcaption> Regularize the extracted title text, including removing HTML entities, special characters and extra spaces; Based on natural language processing methods, image title texts in different languages ​​or encoding formats are uniformly processed.

Images