An event information mining method and system

Abstract

The embodiment of the present application provides an event information mining method and system, comprising: obtaining all corpus data of a to-be-mined event; dividing all corpus data based on granularity levels of geographical space division rules respectively to obtain at least one corpus data set corresponding to each granularity level; for each granularity level, determining each keyword corresponding to the granularity level according to all corpus text contents in each corpus data set corresponding to the granularity level; inputting word frequency of each keyword, production time of the corpus data to which the keyword belongs and production location of the corpus data to which the keyword belongs into an unsupervised model to perform screening processing on each keyword to output each event keyword corresponding to the granularity level; and generating event information of the granularity level according to each event keyword corresponding to the granularity level. The event information can be accurately and concisely mined using a very small amount of corpus information.

Description

Technical Field

[0001] This invention relates to the field of information mining, specifically to a method and system for mining event information. Background Technology

[0002] Existing event mining algorithms typically aggregate massive amounts of text and filter them using models. These methods only utilize the textual information dimension. The approach involves crawling the content of the text, segmenting it into words, generating word vectors, and then using machine learning models to evaluate these word vectors to obtain event keywords. Summary of the Invention

[0003] This invention provides a method and system for mining event information, enabling the accurate, precise, and concise mining of event information using a very small amount of corpus.

[0004] To achieve the above objectives, in one aspect, embodiments of the present invention provide an event information mining method, comprising:

[0005] Obtain all corpus data of the event to be mined, including: production time, production location, and corpus text content;

[0006] Based on the granularity of the geospatial partitioning rules, all corpus data are divided into at least one corpus data set corresponding to each granularity level.

[0007] For each granularity level, the keywords corresponding to that granularity level are determined based on all the text content of the corpus data set corresponding to that granularity level.

[0008] The frequency of each keyword, the production time of the corpus data to which it belongs, and the production location of the corpus data to which it belongs are input into the unsupervised model to perform event keyword filtering on each keyword, and output the event keywords corresponding to the granularity level.

[0009] Event information at the granularity level is generated based on the event keywords corresponding to each granularity level.

[0010] Secondly, a method for training an unsupervised model is provided, including:

[0011] Determine the set number of corpus samples;

[0012] Based on the granularity level of the geospatial partitioning rules, all corpus samples are divided to obtain at least one set of corpus samples corresponding to each granularity level;

[0013] For each corpus sample set, a statistical word segmentation method is used to segment all the text content in the corpus sample set into multiple words; words whose frequency exceeds a set word frequency threshold are used as keywords of the corpus sample set.

[0014] For each granularity level, at least one set of corpus samples is used. The corpus samples containing keywords at the granularity level are input into a point density judgment algorithm network in a finite space, and the event keywords corresponding to the granularity level are output.

[0015] The event keywords of all corpus samples are labeled, and the labeled corpus samples are divided into training set and prediction set at the granular level;

[0016] Each corpus sample in the training set is input into the initial unsupervised model for iterative training. The parameters of the initial unsupervised model are adjusted by the event keywords in the output corpus samples until the accuracy of the prediction set output reaches a preset value, thus obtaining the trained unsupervised model. The initial unsupervised model includes a clustering algorithm model or a deep learning algorithm model.

[0017] The point density judgment algorithm network in the finite space uses points with different areas to represent different corpus samples containing keywords. The area of ​​the point representing the corpus sample is proportional to the number of words obtained by segmenting the corpus sample. The point density judgment algorithm network is specifically used for:

[0018] Each point is laid flat on a plane with the same area as the geographical space where the set number of corpus samples are located; the ratio of the area occupied by all corpus samples containing the same keyword to the area of ​​the plane is used as the spatial coefficient of the keyword at the granularity level, and keywords with spatial coefficients greater than a preset spatial coefficient threshold are selected.

[0019] Keywords with spatial coefficients greater than a preset spatial coefficient threshold are reduced from three-dimensional data to two-dimensional data. The three-dimensional data includes the longitude of the corpus to which the keyword belongs, the latitude of the corpus to which the keyword belongs, and the word frequency of the keyword. The two-dimensional data includes: the spatial coefficient of the keyword at the granularity level, and the ratio between the first word frequency percentage of the keyword within a preset period and the second word frequency percentage within the preset period.

[0020] When the ratio between the first frequency percentage of the keyword and the second frequency percentage of the keyword within a preset period is greater than a preset ratio threshold, the keyword is determined to be an event keyword.

[0021] Thirdly, embodiments of the present invention provide an event information mining system, comprising:

[0022] The corpus acquisition unit is used to acquire all corpus data of the event to be mined. The corpus data includes: production time, production location, and corpus text content.

[0023] The corpus aggregation unit is used to divide all corpus data at the granularity level based on geospatial division rules, and obtain at least one corpus data set corresponding to each granularity level.

[0024] The keyword determination unit is used to determine the keywords corresponding to each granularity level based on the text content of all corpus data sets corresponding to each granularity level.

[0025] The event keyword determination unit is used to input the word frequency of each keyword, the production time of the corpus data to which it belongs, and the production location of the corpus data to which it belongs into the unsupervised model to perform event keyword filtering processing on each keyword, and output the event keywords corresponding to the granularity level.

[0026] The event scenario improvement unit is used to generate event information at the granularity level based on the event keywords corresponding to the granularity level.

[0027] The above technical solution has the following beneficial effects: By dividing all corpus data within a preset geographic space according to the granularity level based on geographic space division rules, at least one corpus data set corresponding to each granularity level is obtained, and the keywords corresponding to each granularity level are determined; for each keyword corresponding to the granularity level, text information, time information, and spatial information are used to more accurately, precisely, and concisely determine the occurrence, specific location, time, and keywords of the event, forming three-dimensional information: keyword frequency, production time of the corpus data to which it belongs, and production location of the corpus data to which it belongs, thereby forming granular-level event information to meet different business needs. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of an event information mining method according to an embodiment of the present invention;

[0030] Figure 2 This is a structural diagram of an event information mining system according to an embodiment of the present invention;

[0031] Figure 3This is a flowchart of a method for three-dimensional event mining of Weibo text based on location, according to an embodiment of the present invention.

[0032] Figure 4 This is a diagram illustrating an algorithm for determining point density in a finite space, according to an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] like Figure 1 As shown, in conjunction with embodiments of the present invention, an event information mining method is provided, comprising:

[0035] S101: Obtain all corpus data of the event to be mined, the corpus data including: production time, production location, and corpus text content;

[0036] S102: Divide all corpus data according to the granularity level of geospatial division rules to obtain at least one corpus data set corresponding to each granularity level;

[0037] S103: For each granularity level, determine the keywords corresponding to the granularity level based on all the text content of the corpus data set corresponding to the granularity level;

[0038] S104: Input the word frequency of each keyword, the production time of the corpus data to which it belongs, and the production location of the corpus data to which it belongs into the unsupervised model to perform event keyword filtering on each keyword, and output the event keywords corresponding to the granularity level;

[0039] S105: Generate event information for the granularity level based on the event keywords corresponding to the granularity level.

[0040] Preferably, step 102 specifically includes:

[0041] S1021: For each granularity level of the geospatial division rule, divide all corpus data into at least one region corresponding to the granularity level;

[0042] S1022: Take the corpus data in the same region corresponding to the granularity level as a corpus data set corresponding to the granularity level, and obtain the corpus data set corresponding to at least one region corresponding to the granularity level respectively.

[0043] Preferably, step 103 specifically includes:

[0044] S1031: For each region corresponding to the granularity level, based on the corpus data set corresponding to the region, a statistical word segmentation method is used to segment all the corpus text content in the corpus data set to obtain multiple words;

[0045] S1032: Aggregate the word frequencies of each word obtained from each region within the granularity level, and take the words whose word frequencies exceed the first word frequency threshold as the keywords corresponding to the granularity level.

[0046] Preferably, step 104 specifically includes:

[0047] The parameters of each keyword corresponding to the granularity level are input into an unsupervised model to perform event keyword filtering based on filtering conditions, and the event keywords corresponding to the granularity level are output.

[0048] The parameters of the keyword include: keyword frequency, production time of the corpus data to which it belongs, and production location of the corpus data to which it belongs; the filtering conditions include that the spatial coefficient of the keyword is greater than a preset spatial coefficient threshold, and that the ratio between the first frequency proportion of the keyword and the second frequency proportion of the keyword within a preset period of a preset length is greater than a preset ratio threshold.

[0049] Preferably, step 105 specifically includes:

[0050] S1051: For each event keyword corresponding to the granularity level, the text content of all corpus containing the event keyword is segmented to obtain secondary keywords containing the event keyword, the word frequency of the secondary keywords is counted, and the secondary keywords whose word frequency exceeds the first word frequency threshold are regarded as high-frequency words.

[0051] S1052: Perform syntactic analysis and part-of-speech tagging on sentences containing high-frequency words, supplement the high-frequency words into complete sentences or complete event content, and use the complete sentences or complete event content as the event information at the granular level; wherein, if the number of complete sentences or complete event content containing the same high-frequency word exceeds a preset number, retain the complete sentences or complete event content with the most complete part of speech and the highest aggregation frequency;

[0052] Preferably, the event information mining method further includes:

[0053] S106: Select the target granularity level of event information to display based on business display requirements.

[0054] In conjunction with embodiments of the present invention, a method for training an unsupervised model is also provided, comprising:

[0055] Determine the set number of corpus samples;

[0056] Based on the granularity level of the geospatial partitioning rules, all corpus samples are divided to obtain at least one set of corpus samples corresponding to each granularity level;

[0057] For each corpus sample set, a statistical word segmentation method is used to segment all the text content in the corpus sample set into multiple words; words whose frequency exceeds a set word frequency threshold are used as keywords of the corpus sample set.

[0058] For each granularity level, at least one set of corpus samples is used. The corpus samples containing keywords at the granularity level are input into a point density judgment algorithm network in a finite space, and the event keywords corresponding to the granularity level are output.

[0059] The event keywords of all corpus samples are labeled, and the labeled corpus samples are divided into training set and prediction set at the granular level;

[0060] Each corpus sample in the training set is input into the initial unsupervised model for iterative training. The parameters of the initial unsupervised model are adjusted by the event keywords in the output corpus samples until the accuracy of the prediction set output reaches a preset value, thus obtaining the trained unsupervised model. The initial unsupervised model includes a clustering algorithm model or a deep learning algorithm model.

[0061] The point density judgment algorithm network in the finite space uses points with different areas to represent different corpus samples containing keywords. The area of ​​the point representing the corpus sample is proportional to the number of words obtained by segmenting the corpus sample. The point density judgment algorithm network is specifically used for:

[0062] Each point is laid flat on a plane with the same area as the geographical space where the set number of corpus samples are located; the ratio of the area occupied by all corpus samples containing the same keyword to the area of ​​the plane is used as the spatial coefficient of the keyword at the granularity level, and keywords with spatial coefficients greater than a preset spatial coefficient threshold are selected.

[0063] Keywords with spatial coefficients greater than a preset spatial coefficient threshold are reduced from three-dimensional data to two-dimensional data. The three-dimensional data includes the longitude of the corpus to which the keyword belongs, the latitude of the corpus to which the keyword belongs, and the word frequency of the keyword. The two-dimensional data includes: the spatial coefficient of the keyword at the granularity level, and the ratio between the first word frequency percentage of the keyword within a preset period and the second word frequency percentage within the preset period.

[0064] When the ratio between the first frequency percentage of the keyword and the second frequency percentage of the keyword within a preset period is greater than a preset ratio threshold, the keyword is determined to be an event keyword.

[0065] like Figure 2 As shown, in conjunction with embodiments of the present invention, an event information mining system is provided, comprising:

[0066] The corpus acquisition unit 21 is used to acquire all corpus data of the event to be mined, and the corpus data includes: production time, production location, and corpus text content.

[0067] Corpus aggregation unit 22 is used to divide all corpus data according to the granularity level of geospatial division rules, and obtain at least one corpus data set corresponding to each granularity level;

[0068] Keyword determination unit 23 is used to determine each keyword corresponding to each granularity level based on the text content of all corpus data sets corresponding to each granularity level.

[0069] The event keyword determination unit 24 is used to input the word frequency of each keyword, the production time of the corpus data to which it belongs, and the production position of the corpus data to which it belongs into the unsupervised model to perform event keyword filtering processing on each keyword, and output each event keyword corresponding to the granularity level;

[0070] Event scenario improvement unit 25 is used to generate event information at the granularity level based on the event keywords corresponding to the granularity level.

[0071] Preferably, the corpus aggregation unit 22 is specifically used for:

[0072] For each granularity level of the geospatial partitioning rules, all corpus data are divided into at least one region corresponding to that granularity level;

[0073] By taking the corpus data within the same region corresponding to the granularity level as a corpus data set corresponding to the granularity level, corpus data sets corresponding to at least one region corresponding to the granularity level are obtained respectively.

[0074] Preferably, the keyword determination unit 23 is specifically used for:

[0075] For each region corresponding to the granularity level, based on the corpus data set corresponding to the region, a statistical word segmentation method is used to segment all the corpus text content in the corpus data set to obtain multiple words;

[0076] The word frequencies of each word obtained from each region within the specified granularity level are aggregated, and words whose word frequencies exceed the first word frequency threshold are taken as the keywords corresponding to the specified granularity level.

[0077] Preferably, the event keyword determination unit 24 is specifically used for:

[0078] The parameters of each keyword corresponding to the granularity level are input into an unsupervised model to perform event keyword filtering based on filtering conditions, and the event keywords corresponding to the granularity level are output.

[0079] The parameters of the keyword include: keyword frequency, production time of the corpus data to which it belongs, and production location of the corpus data to which it belongs; the filtering conditions include that the spatial coefficient of the keyword is greater than a preset spatial coefficient threshold, and that the ratio between the first frequency proportion of the keyword and the second frequency proportion of the keyword within a preset period of a preset length is greater than a preset ratio threshold.

[0080] Preferably, the event scenario improvement unit 25 includes:

[0081] For each event keyword corresponding to the granularity level, the text content of all corpus containing the event keyword is segmented to obtain secondary keywords containing the event keyword. The word frequency of the secondary keywords is counted, and the secondary keywords whose word frequency exceeds the first word frequency threshold are regarded as high-frequency words.

[0082] Sentences containing high-frequency words are subjected to syntactic analysis and part-of-speech tagging. The high-frequency words are supplemented into complete sentences or complete event content, and the complete sentences or complete event content are used as event information at the granular level. Among them, when the number of complete sentences or complete event content containing the same high-frequency word exceeds a preset number, the complete sentences or complete event content with complete part-of-speech tagging and the highest aggregation frequency are retained.

[0083] The event information mining system also includes:

[0084] The display unit is used to select and display event information at the target granularity level according to business display needs.

[0085] The beneficial effects achieved by the embodiments of the present invention are as follows:

[0086] By dividing the corpus data within a preset geographic space according to the granularity level based on geographic space partitioning rules, at least one corpus data set corresponding to each granularity level is obtained, and the keywords corresponding to each granularity level are determined. For each keyword corresponding to the granularity level, textual information, time information, and spatial information are used to more accurately, precisely, and concisely determine the occurrence, specific location, and time of the event, as well as the keyword, forming three dimensions of information: keyword frequency, production time of the corpus data to which it belongs, and production location of the corpus data to which it belongs. This forms granular-level event information to meet different business needs.

[0087] The three-dimensional information of this invention can accurately mine event scenarios with very small corpora. With a corpus of no less than three items, the corresponding event scenario information can be mined using the method of this invention, avoiding the need for massive data support (event data quantity >= 100) in existing event mining technologies.

[0088] The three-dimensional information of this invention can be used to discover similar events that occur simultaneously in different locations, avoiding the shortcomings of existing event mining methods that cannot capture similar events that occur in different spaces at the same time.

[0089] The unsupervised model of this invention does not require pre-training the model with data that are already keywords. It can mine events of uncertain type, unknown, and untrained events, and obtain mining results quickly with less computing power (the processing time of one 16-core 32G machine is less than 3 hours), avoiding the disadvantage of the existing technology that requires far more computing power for processing.

[0090] The unsupervised model of this invention does not require human intervention or continuous manual negative feedback annotation. This avoids the shortcomings of existing technologies where "event mining relies on complex machine learning models, has extremely poor interpretability, cannot explain errors and false hits, cannot be accurately optimized, and requires further manual intervention for negative feedback annotation."

[0091] The mining results of this invention do not require human intervention or continuous manual negative feedback annotation. This avoids the shortcomings of existing technologies where "event mining relies on complex machine learning models, has extremely poor interpretability, cannot explain errors and false hits, cannot be accurately optimized, and requires further manual intervention for negative feedback annotation."

[0092] The technical solutions of the present invention will be described in detail below with reference to specific application examples. For technical details not described in the implementation process, please refer to the relevant descriptions above.

[0093] This invention provides an event mining algorithm based on location (microblog) across three dimensions of spatiotemporal text for event mining:

[0094] 1. Obtain the corpus data of the events to be explored on Weibo, including the corpus production time, corpus production location, and corpus text content.

[0095] 2. The text content of the corpus is segmented and aggregated based on a preset geographic spatial radius (different granularities between 100m and 1000km).

[0096] 3. Using algorithms (image algorithms, density algorithms, machine learning algorithms, deep learning algorithms, etc.), the aggregated corpus is analyzed to determine whether it conforms to the characteristics of suddenness and uniqueness in time and space, and non-event aggregated corpus is filtered out to obtain event keywords.

[0097] 4. Perform scenario completion mining on event keywords. By scanning and aggregating all corpora containing the same event keyword, other aggregated corpora information under the same event keyword are obtained, and event details are generated.

[0098] like Figure 3 As shown, the technical solution of the present invention is described in detail below:

[0099] (1) Granularity classification:

[0100] Based on the granularity of the geospatial partitioning rules, all corpus data are divided into at least one corpus data set corresponding to each granularity level. Specifically, for each granularity level of the geospatial partitioning rules, all corpus data are divided into at least one region corresponding to the granularity level. The corpus data in the same region corresponding to the granularity level are taken as a corpus data set corresponding to the granularity level, and the corpus data sets corresponding to each of the at least one region corresponding to the granularity level are obtained respectively.

[0101] The spatial dimension data is divided into regional blocks (different granularities between 100m and 1000km) for all corpora within the production location of the event to be mined. These blocks are used as parameters for clustering algorithms, density algorithms, machine learning algorithms, or deep learning algorithms to determine whether the event has actually occurred.

[0102] Geohash divides the map corresponding to all corpus production locations into several grids of varying granularity, as shown in Table 1. Geohash definition: Geohash divides the entire map into several squares of equal size. The longer the geohash, the smaller the side length of the square; the shorter the geohash, the larger the side length of the square. In other words, the higher the granularity, the smaller the area.

[0103] Geohash is used to divide the material into segments based on side length (geohash level size), with granularity levels ranging from 100m to 1000km. Each granularity level segments all the corpus and performs aggregation separately; that is, the same corpus is repeatedly segmented at different granularity levels, and the grid size corresponding to each granularity level is different. The longer the geohash, the smaller the actual geographical area covered by the grid. Therefore, based on the geohash length and grid size, we can obtain geohash grids with sides of 100m, 1km, 10km, 100km, and 1000km.

[0104] Table 1

[0105]

[0106] Using grids as the basis for division, the content within the same level of geohash grid is considered a single region. For the same granularity level, several sets of corpora are obtained, each divided based on geohash.

[0107] (2) Detailed word segmentation:

[0108] For each region's corpus at each granularity level, word segmentation is performed on the corresponding text content to determine the keywords corresponding to each granularity level. Commonly used word segmentation methods in the industry include those based on existing word segmentation libraries. These existing libraries are further categorized as: rule-based segmentation methods (forward longest matching algorithm, reverse longest matching algorithm, bidirectional longest matching algorithm), statistical segmentation methods (N-gram model, Hidden Markov Model (HMM), Maximum Entropy Model (ME), Conditional Random Fields (CRF)), semantic-based segmentation methods (extended transition network method, matrix constraint method), and understanding-based segmentation methods (expert system segmentation method, neural network segmentation method, neural network expert system integrated segmentation method). The principles of word segmentation matching algorithms are not detailed here. Commonly used word segmentation libraries include jieba, HanLP (Harbin Institute of Technology), and NLPIR (Chinese Academy of Sciences).

[0109] This invention primarily employs an original statistical-based word segmentation method. For each region, after segmenting the corpus text using statistical-based word segmentation software or algorithms, several words within the same geohash region are obtained, i.e., words of varying lengths. The frequency of each word of varying length within the text block is then statistically analyzed, a process known as aggregation. In reproduction, any existing word segmentation library (jieba, hanlp) can be directly used, with essentially equivalent results.

[0110] (3) Keyword acquisition method: After segmenting words using word segmentation software or algorithms, several words of varying lengths under the same geohash will be obtained. At this time, the frequency of each word of varying lengths appearing in the block is counted, which is called aggregation operation. When the number of times a word appears is greater than the first occurrence frequency (first word frequency), it is taken as the keyword corresponding to the granularity level.

[0111] Keyword filtering: TF-IDF can be used to extract and filter words of varying lengths within a sentence block to obtain new keywords. This step is optional. The TF-IDF corpus feature extraction algorithm can identify keywords with more than a predetermined number of commonalities and keywords with more than a predetermined number of unique characteristics. Collaborative filtering (aiming to remove words with more than a predetermined number of commonalities and retain words with more than a predetermined number of unique characteristics) can yield filtered keyword results. Here, the default parameters of this algorithm are used for filtering.

[0112] (4) Use event mining algorithms to determine whether a keyword is an event keyword:

[0113] The algorithm primarily uses time coefficient, spatial coefficient, and word frequency coefficient as input parameters to determine whether a keyword is an event keyword.

[0114] 4.1 Cold Start Phase:

[0115] It mainly consists of non-machine learning (unsupervised model) components, including image algorithms, density judgment algorithms, etc.

[0116] One algorithm is listed here. After using spatial coefficients and word frequency coefficients as preprocessing, it combines them with simple rules of time coefficients to determine whether a certain keyword is an event keyword.

[0117] Among them, such as Figure 4The algorithm shown is based on point density judgment in a finite space. It uses points to represent the corpus within the corresponding region of a keyword. By tiling spatially overlapping points onto a plane of the same size, the denser the points, the more overlapping they are, and the smaller the area of ​​the dark rectangle on the white background image. By judging the proportion of the dark area to the white area in the final region (this proportion is the spatial coefficient), it is determined whether the keyword meets the spatial suddenness, and thus used to further determine whether the keyword is an event type.

[0118] Keywords with spatial coefficients greater than a preset spatial coefficient threshold are reduced from three-dimensional data to two-dimensional data. The three-dimensional data includes the longitude and latitude of the corpus to which the keyword belongs, and the word frequency of the keyword. The two-dimensional data includes: the spatial coefficient of the keyword within this level, and the ratio of the word frequency percentage of the keyword within a preset time period (e.g., 6 days, 20% of the time) to the word frequency percentage within the same preset time period (e.g., the most recent month) (e.g., 80%).

[0119] 4.2 After cold start:

[0120] The cold start phase data is divided into a training set and a test set. The training set parameters include multiple spatial coordinates and multiple time points, and the training objective is to determine whether a data point is an event keyword. Training can use clustering algorithms or deep learning algorithms for binary judgment. The process is basically the same as the conventional process. Training is conducted through machine learning until the accuracy on the prediction set reaches a preset value, at which point training ends, resulting in a more accurate event keyword judgment model.

[0121] 4.3 Event Keyword Filtering and Processing:

[0122] The parameters of each keyword corresponding to the granularity level are input into an unsupervised model for event keyword filtering based on filtering conditions, and the event keywords corresponding to the granularity level are output. The parameters of the keywords include: keyword frequency, production time of the corpus data to which they belong, and production location of the corpus data to which they belong. The filtering conditions include that the spatial coefficient of the keyword is greater than a preset spatial coefficient threshold, and that the ratio between the first frequency proportion of the keyword and the second frequency proportion of the keyword within a preset period of a preset length is greater than a preset ratio threshold.

[0123] (5) Complete event scene information using event keywords:

[0124] After obtaining the event keywords, the entire corpus containing the event keywords is segmented again. The word frequency of the segmented results is calculated to obtain secondary keywords containing the event keywords. The word frequency of these secondary keywords is also calculated. Secondary keywords whose word frequency reaches a first word frequency threshold (keywords in the corpus containing the event keywords) are considered high-frequency words. Using the TF-IDF algorithm, useless words are filtered out from these high-frequency words. Subsequently, syntactic analysis and part-of-speech tagging are performed to complete the keywords into a complete sentence or a complete event content, making the event keywords a complete event information. For example, the obtained keyword might be "cancel dine-in service," which, after completion, becomes "universities cancel dine-in service," making the overall meaning more complete. Randomly selecting several original sentences containing the event keywords as completion content can also complete the event keywords into a complete event scenario. If there is a large amount of material, the sentence with the highest aggregation frequency is selected while preserving the completeness of the sentence's part-of-speech.

[0125] In this scenario, all results will be retained, and the final decision on whether to use large-area small-level grids (the shorter the grid ID number, the larger the coverage area, refer to geohash) or small-area large-level grids will be made by the business display format.

[0126] The beneficial effects achieved by the embodiments of the present invention are as follows:

[0127] Existing event mining algorithms typically aggregate massive amounts of text directly and filter them using models. These methods only utilize one dimension: textual information. This invention, however, divides the corpus data within a predefined geographic spatial range according to a granularity level based on geographic spatial partitioning rules. This results in at least one corpus data set corresponding to each granularity level, identifying keywords for each level. For each keyword at each granularity level, textual, temporal, and spatial information is used to more accurately, precisely, and concisely determine the event occurrence, specific location, time, and keywords, forming three dimensions of information: keyword frequency, the production time of the corpus data to which it belongs, and the production location of the corpus data to which it belongs. This creates granular-level event information to meet diverse business needs.

[0128] The three-dimensional information of this invention can accurately mine event scenarios with very small corpora. With a corpus of no less than three items, the corresponding event scenario information can be mined using the method of this invention, avoiding the need for massive data support (event data quantity >= 100) in existing event mining technologies.

[0129] The three-dimensional information of this invention can be used to discover similar events that occur simultaneously in different locations, avoiding the shortcomings of existing event mining methods that cannot capture similar events that occur in different spaces at the same time.

[0130] The unsupervised model of this invention does not require pre-training the model with data that are already keywords. It can mine events of uncertain type, unknown, and untrained events, and obtain mining results quickly with less computing power (the processing time of one 16-core 32G machine is less than 3 hours), avoiding the disadvantage of the existing technology that requires far more computing power for processing.

[0131] The unsupervised model of this invention does not require human intervention or continuous manual negative feedback annotation. This avoids the shortcomings of existing technologies where "event mining relies on complex machine learning models, has extremely poor interpretability, cannot explain errors and false hits, cannot be accurately optimized, and requires further manual intervention for negative feedback annotation."

[0132] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0133] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0134] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0135] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0136] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0137] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0138] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0139] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0140] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for mining event information, characterized in that, include: Obtain all corpus data of the event to be mined, including: production time, production location, and corpus text content; For each granularity level of the geospatial partitioning rules, all corpus data are divided into at least one region corresponding to that granularity level; The corpus data within the same region corresponding to the granularity level are taken as a corpus data set corresponding to the granularity level, and the corpus data sets corresponding to at least one region corresponding to the granularity level are obtained respectively. For each region corresponding to the granularity level, based on the corpus data set corresponding to the region, a statistical word segmentation method is used to segment all the corpus text content in the corpus data set to obtain multiple words; The word frequencies of each word obtained from each region within the specified granularity level are aggregated, and words whose word frequencies exceed the first word frequency threshold are taken as keywords corresponding to the specified granularity level. The parameters of each keyword corresponding to the granularity level are input into an unsupervised model to perform event keyword filtering based on filtering conditions, and the event keywords corresponding to the granularity level are output. The parameters of the keyword include: keyword frequency, production time of the corpus data, and production location of the corpus data; the filtering conditions include that the spatial coefficient of the keyword is greater than a preset spatial coefficient threshold, and the ratio between the first frequency proportion of the keyword and the second frequency proportion of the keyword within a preset period of a preset length is greater than a preset ratio threshold; wherein, points with different areas are used to represent different corpus data containing the keyword, the area of ​​the points representing the corpus data is proportional to the number of words obtained by segmenting the corpus data, each point is laid flat on a plane with the same area as the geographical space where the number of corpus data is located, and the ratio of the area occupied by all corpus data containing the same keyword to the area of ​​the plane is used as the spatial coefficient of the keyword at the granularity level; Event information at the granularity level is generated based on the event keywords corresponding to each granularity level.

2. The event information mining method according to claim 1, characterized in that, The step of generating event scene information at the granularity level based on the event keywords corresponding to the granularity level specifically includes: For each event keyword corresponding to the granularity level, the text content of all corpus containing the event keyword is segmented to obtain secondary keywords containing the event keyword. The word frequency of the secondary keywords is counted, and the secondary keywords whose word frequency exceeds the first word frequency threshold are regarded as high-frequency words. Sentences containing high-frequency words are subjected to syntactic analysis and part-of-speech tagging. The high-frequency words are supplemented into complete sentences or complete event content, and the complete sentences or complete event content are used as event information at the granular level. Among them, when the number of complete sentences or complete event content containing the same high-frequency word exceeds a preset number, the complete sentences or complete event content with complete part-of-speech tagging and the highest aggregation frequency are retained.

3. The event information mining method according to claim 1, characterized in that, The method further includes: Select and display event information at the target granularity level according to business presentation needs.

4. A training method for an unsupervised model, characterized in that, include: Determine the set number of corpus samples; Based on the granularity level of the geospatial partitioning rules, all corpus samples are divided to obtain at least one set of corpus samples corresponding to each granularity level; For each corpus sample set, a statistical word segmentation method is used to segment all the text content in the corpus sample set into multiple words; words whose frequency exceeds a set word frequency threshold are used as keywords of the corpus sample set. For each granularity level, at least one set of corpus samples is used. The corpus samples containing keywords at the granularity level are input into a point density judgment algorithm network in a finite space, and the event keywords corresponding to the granularity level are output. The event keywords of all corpus samples are labeled, and the labeled corpus samples are divided into training set and prediction set at the granular level; Each corpus sample in the training set is input into the initial unsupervised model for iterative training. The parameters of the initial unsupervised model are adjusted by the event keywords in the output corpus samples until the accuracy of the prediction set output reaches a preset value, thus obtaining the trained unsupervised model. The initial unsupervised model includes a clustering algorithm model or a deep learning algorithm model. The point density determination algorithm network in the finite space uses points with different areas to represent different corpus samples containing keywords. The area of ​​the point representing the corpus sample is proportional to the number of words obtained by segmenting the corpus sample. The point density determination algorithm network is used for: Each point is laid flat on a plane with the same area as the geographical space where the set number of corpus samples are located; the ratio of the area occupied by all corpus samples containing the same keyword to the area of ​​the plane is used as the spatial coefficient of the keyword at the granularity level, and keywords with spatial coefficients greater than a preset spatial coefficient threshold are selected. Keywords with spatial coefficients greater than a preset spatial coefficient threshold are reduced from three-dimensional data to two-dimensional data. The three-dimensional data includes the longitude of the corpus to which the keyword belongs, the latitude of the corpus to which the keyword belongs, and the word frequency of the keyword. The two-dimensional data includes: the spatial coefficient of the keyword at the granularity level, and the ratio between the first word frequency percentage of the keyword within a preset period and the second word frequency percentage within the preset period. When the ratio between the first frequency percentage of the keyword and the second frequency percentage of the keyword within a preset period is greater than a preset ratio threshold, the keyword is determined to be an event keyword.

5. An event information mining system, characterized in that, include: The corpus acquisition unit is used to acquire all corpus data of the event to be mined. The corpus data includes: production time, production location, and corpus text content. The corpus aggregation unit is used to divide all corpus data at the granularity level based on geospatial division rules, and obtain at least one corpus data set corresponding to each granularity level. The keyword determination unit is used to determine the keywords corresponding to each granularity level based on the text content of all corpus data sets corresponding to each granularity level. The event keyword determination unit is used to input the word frequency of each keyword, the production time of the corpus data to which it belongs, and the production location of the corpus data to which it belongs into the unsupervised model to perform event keyword filtering processing on each keyword, and output the event keywords corresponding to the granularity level. The event scenario refinement unit is used to generate event information at the granularity level based on the event keywords corresponding to the granularity level. The corpus aggregation unit is specifically used for: For each granularity level of the geospatial partitioning rules, all corpus data are divided into at least one region corresponding to that granularity level; The corpus data within the same region corresponding to the granularity level are taken as a corpus data set corresponding to the granularity level, and the corpus data sets corresponding to at least one region corresponding to the granularity level are obtained respectively. The keywords determine the unit, specifically used for: For each region corresponding to the granularity level, based on the corpus data set corresponding to the region, a statistical word segmentation method is used to segment all the corpus text content in the corpus data set to obtain multiple words; The word frequencies of each word obtained from each region within the specified granularity level are aggregated, and words whose word frequencies exceed the first word frequency threshold are taken as keywords corresponding to the specified granularity level. The event keyword determination unit is specifically used for: The parameters of each keyword corresponding to the granularity level are input into an unsupervised model to perform event keyword filtering based on filtering conditions, and the event keywords corresponding to the granularity level are output. The parameters of the keyword include: keyword frequency, production time of the corpus data, and production location of the corpus data; the filtering conditions include that the spatial coefficient of the keyword is greater than a preset spatial coefficient threshold, and the ratio between the first frequency proportion of the keyword and the second frequency proportion of the keyword within a preset period of a preset length is greater than a preset ratio threshold; wherein, points with different areas are used to represent different corpus data containing the keyword, the area of ​​the points representing the corpus data is proportional to the number of words obtained by segmenting the corpus data, each point is laid flat on a plane with the same area as the geographical space where the number of corpus data is located, and the ratio of the area occupied by all corpus data containing the same keyword to the area of ​​the plane is used as the spatial coefficient of the keyword at the granularity level.

6. The event information mining system according to claim 5, characterized in that, The event scenario improvement unit includes: For each event keyword corresponding to the granularity level, the text content of all corpus containing the event keyword is segmented to obtain secondary keywords containing the event keyword. The word frequency of the secondary keywords is counted, and the secondary keywords whose word frequency exceeds the first word frequency threshold are regarded as high-frequency words. Sentences containing high-frequency words are subjected to syntactic analysis and part-of-speech tagging. The high-frequency words are supplemented into complete sentences or complete event content, and the complete sentences or complete event content are used as event information at the granular level. Among them, when the number of complete sentences or complete event content containing the same high-frequency word exceeds a preset number, the complete sentences or complete event content with complete part-of-speech tagging and the highest aggregation frequency are retained.

Images