A method for simplifying event rules in event information collection

Abstract

The present invention discloses a method for simplifying event rules in event information collection, and the steps are as follows: (1) splitting a user-customized event rule ER into a sub-event rule set SER; (2) determining whether the SERs satisfy an intersection relationship, selecting event rules through calculation, and obtaining a simplified SER set LS. The method of the present invention aims at the problem of low efficiency in event information collection, provides guidance for the element composition of event rules and sub-event rules, analyzes the intersection relationship between sub-event rules, and improves the performance of event information collection.

Description

Technical Field

[0001] The present invention relates to an information mining technology, and in particular to an event rule simplification method in event information collection. Background Art

[0002] Since many events in the real world are reflected in different ways on the Internet, it has become very urgent to obtain and continuously track event information on the Internet. However, with the wide variety of media and the massive growth of information, quickly obtaining event information of interest from various media is an urgent problem that the current Internet information processing system needs to solve. Although users can obtain relevant event information to a certain extent using general search engines, such as Baidu, Google, Bing and other search engines, since these search engines face general customers and collect comprehensive information from various media on the Internet, users are affected in terms of real-time and accuracy in obtaining events of interest. How to improve the performance of information collection has become a hot topic in current research.

[0003] At present, mainstream media presents the characteristics of massive information, namely, large amount of information, various types, fast update, and low value density. In order to solve this problem, mainstream Internet media usually provide built-in search functions to find specific information. For mainstream media with built-in search functions, the existing event information collection methods can no longer achieve efficient information collection in the increasing amount of massive information.

[0004] In the conference proceedings published in China in 2020: Proceedings of the 2nd International Conference on Information Technology and Computer Application, the title is: Application and research of heuristic search algorithm in crawler field, the authors are Liu H, Wang K, Liu Z. The paper believes that the breadth-first strategy in topic collection fails to make full use of the effective information of the page, resulting in time-consuming collection. A heuristic search algorithm is proposed, which mainly uses the keywords, suffix names, hyperlinks, topic tags and other information of the page. In order to improve the speed of Web collection, the topic collection of this method is too broad, and it is difficult to propose more effective methods for some special types of collection targets, such as event information collection, character information collection, product information collection, industry policy collection, etc., which is difficult to adapt to the growing amount of network information collection.

[0005] In the 2021 issue of the American journal Expert Systems With Applications, the title is: An intelligent system for focused crawling from big data sources, the authors are: Bifulco I, Cirillo S, Esposito C, Guadagni R, Polese G. When collecting subject information, the article manually provided some subject keywords and URL seeds. However, when collecting subject information, this method followed the methods and techniques of subject collection, without in-depth analysis of the constraints of event elements, making it difficult to simplify event rules and improve information collection efficiency.

[0006] In the 2019 American journal ACM Computing Surveys, titled Evaluating domain ontologies: clarification, classification, and challenges, the authors are McDaniel M, Storey VC. The article uses domain ontologies, including terms, entities, and the relationships between them, when searching for subject information. This method takes into account the constraints of event elements, but lacks consideration of the relationship between multiple topics or events, and is not applicable to mainstream media with built-in search functions.

[0007] In the process of implementing the present invention, the inventors found that the event information collection method and technology for mainstream media with built-in search engines have the following three shortcomings:

[0008] (1) The subject collection is too broad, and it is difficult to come up with more effective methods for some special types of collection targets;

[0009] (2) Most of them use the methods and techniques of topic collection without in-depth analysis of the constraints of event elements;

[0010] (3) The relationship between multiple topics or events is not considered. The present invention deeply analyzes the constraint relationship of event elements, and then provides a reasonable way to collect event descriptions and an efficient collection method. By formulating simplified event collection rules, the efficiency of event information collection is greatly improved, which has great practical value. Summary of the invention

[0011] In view of the problems and shortcomings of the prior art mentioned above, the technical problem to be solved by the present invention is to provide a method for simplifying event rules in event information collection. This method more reasonably utilizes the event rule relationship and is more suitable for mainstream media with built-in search functions to achieve efficient event information collection.

[0012] In order to solve the above problems, the present invention adopts the following technical solution: a method for simplifying event rules in event information collection, and its specific steps are as follows:

[0013] A. Split the user-customized event rule ER into sub-event rule sets SER;

[0014] B. Determine whether the SERs satisfy the intersection relationship, select event rules through calculation, and obtain the simplified SER set LS.

[0015] The method for simplifying event rules in event information collection of the present invention, the preferred specific steps of step (A) are as follows:

[0016] A1. Determine whether the event rule ER needs to be split. ER consists of the "∧" and "∨" relationships between event elements. "∧" is an "and" relationship and "∨" is an "or" relationship. SER only consists of the "∧" relationship between event elements. When ER only has the "∧" relationship, it does not need to be split. When ER contains both "∧" and "∨" relationships, ER follows the "and" distribution law. i ∧(o i1 ∨o i2 )=(a i ∧o i1 )∨(a i ∧o i2 )

[0017] Split into two parts, where a and o represent the action element and object element of the rule respectively; the number i at the bottom of a represents the number of the action element, the number i1 at the bottom of o represents the first action element of the i-th action element, and so on;

[0018] A2. Calculate the number Num of "AND" distributive laws in ER;

[0019] A3. Perform Num times of “AND” distributive operation on ER to obtain ER i , split ER according to the "∨" relationship i , get the sub-event rule set SER; ER i The number i in the lower corner indicates the action element.

[0020] The method for simplifying event rules in event information collection of the present invention, the preferred specific steps of step (B) are as follows:

[0021] B1. Loop and take out each SER from SER i (1≤i≤m); B2, judge SER i With each subsequent SER j (i+1≤j≤m) whether it satisfies the intersection relation, i.e. SER i and SER j The action element a satisfies a i =a j , and the three elements of time t, location l, and object o satisfy t i , l i , o i With t j , l j , o j The same number u≥1;

[0022] B3. By SER i and SER j Get SER i，j =α i ∧A(S i，j ), where S i，j It is SER i SER j The set of the same elements in t, l, and o, A(S i，j ) is for S i，j The "∧" operation of the elements in the i，j Alternative to SER i , delete SER j ;

[0023] B4, loop through B2 and B3 to obtain the reduced SER set LS;

[0024] i,j represents the i,jth sub-event rule set, and m is a natural number.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The event rule simplification method in event information collection of the present invention makes more reasonable use of event rule relationships and is more suitable for mainstream media with built-in search functions to achieve efficient event information collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the event rule simplification method in event information collection of the present invention;

[0028] Figure 2 yes Figure 1 The flowchart of splitting the user-customized event rule ER into the sub-event rule set SER as described in step 101;

[0029] Figure 3 yes Figure 1 The flowchart of determining whether the SERs satisfy the intersection relationship described in step 102, selecting event rules through calculation, and obtaining the simplified SER set LS; DETAILED DESCRIPTION

[0030] The implementation process of the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] The event rule simplification method in event information collection, refer to Figure 1 , the method comprises the following steps:

[0032] Step 101: Split the user-customized event rule ER into sub-event rule sets SER, referring to Figure 2 , the specific steps are as follows:

[0033] Step 201: Determine whether the event rule ER needs to be split. ER consists of the “∧” and “∨” relationships between event elements (“∧” is an “and” relationship, and “∨” is an “or” relationship), while SER only consists of the “∧” relationship between event elements. That is, if ER only has the “∧” relationship, it does not need to be split. If ER contains both “∧” and “∨” relationships, ER follows the “and” distribution law.

[0034] a i ∧(o i1 ∨o i2 )=(a i ∧o i1 )∨(a i ∧o i2 )

[0035] Split. Get the element relationship of the event rule ER and determine the ER that needs to be split;

[0036] Step 202: Calculate the number Num of "AND" distributive laws in ER. This step can be used to determine whether there are any "AND" relationships that need to be split in the event rule;

[0037] Step 203: Perform Num times of “AND” distributive operation on ER to obtain ER i , split ER according to the "∨" relationship i , and obtain the sub-event rule set SER. According to the number of "and" relationships contained in the event rule ER, the event rule is split to obtain the sub-event rule set.

[0038] Step 102: Determine whether the SERs satisfy the intersection relationship, select event rules through operation, and obtain the simplified SER set LS. Figure 3 , the specific steps are as follows:

[0039] Step 301: Loop and extract each SER from LS i (1≤i≤m). Traverse each rule in the initial sub-event rule set;

[0040] Step 302: Determine SER i With each subsequent SER j (i+1≤j≤m) whether it satisfies the intersection relation, i.e. SER i and SER j The action element a satisfies a i =a j , and the three elements of time t, location l, and object o satisfy t i , l i , o i With t j , l j , o j The same number u ≥ 1. The SER obtained by simplifying the definition of event constraint effect must contain at least one a and one element among t, l, and o to determine the relationship between sub-event rules;

[0041] Step 303: SER i and SER j Get SER i，j =a i ∧A(S i，j ), where S i，j It is SER i SER j The set of the same elements in t, l, and o, A(S i，j ) is for S i，j The "∧" operation of the elements in the i，j Replace SER in LS i , delete SER j . Traverse and compare each sub-event rule, and replace the original rule with the calculated simplified event rule;

[0042] Step 304: cyclically execute the simplified SER set of 302 and 303. Traverse each sub-event rule and compare and simplify them one by one, and finally obtain the simplified SER set LS.

[0043] Use three different collection methods to count the amount of event information collected in different unit times and compare the efficiency of event information collection. The four methods are as follows:

[0044] (1) Method 1-Base-1: Each event topic contains multiple SERs. A SER is randomly selected and the event elements it contains are used as keywords. Regardless of their relationship, they are put into each built-in search engine for search. The first 20 URLs of the returned results are taken as seed pages. If a page includes all keywords, it is considered relevant, and all URLs contained in the page are used as extension objects.

[0045] (2) Method 2-Base-2: The selection of seed URL is similar to that of method 1. When calculating the relevance between a page and a topic, if a page contains all the keywords, it is considered relevant. When selecting an extended page, it is required that the anchor tag in the page or the text tag corresponding to the URL link include at least one keyword.

[0046] (2) Method 3-SER-Crawler, using the method introduced in the present invention, only splits the event rule ER into sub-event rules SER, does not consider the simplification of ER, and distributes all SER to the built-in search engine for collection.

[0047] (3) Method 4 - RSER-Crawler, using the method introduced in this paper, ER simplification of event rules is performed, and the simplified SER is collected in the built-in search engine.

[0048] (4) The present invention uses two SaaS platforms: one is industry SaaS, which is an event information collection system for the fire protection industry in a certain province; the other is regional SaaS, which is an event information collection system for multiple industries in a certain city, involving five industries such as environmental protection, fire protection, publicity, education, and tobacco.

[0049] (5) When collecting information, the four methods uniformly limit the time range to January 1, 2021 - July 31, 2021. When SER-Crawler and RSER-Crawler collect event information, no specific information relevance calculation method is used, because each built-in search engine has its own method for calculating the relevance between query items and information. In order to count the amount of event information obtained, the four methods only perform URL deduplication on the information obtained, and do not perform content deduplication.

[0050] (6) For industry SaaS platforms, the amount of event information collected by the four methods at different unit times is shown in Table 1. Table 2 shows the trend of the amount of event information collected by the corresponding industry SaaS platforms at different unit times. For regional SaaS platforms, the amount of event information collected by the four methods at different unit times is shown in Table 2.

[0051] Table 1 The amount of event information collected by industry SaaS platforms at different time units

[0052] method 0.5 hours 1 hour 2 hours 4 hours 8 hours 16 hours Base-1 1834 1939 2033 2033 2033 2033 Base-2 2041 2193 2199 2199 2199 2199 SER-Crawler 2534 3022 3210 3218 3231 3237 RSER-Crawler 3117 3206 3218 3227 3235 3246

[0053] Table 2. The amount of event information collected by regional SaaS platforms at different time units

[0054] method 0.5 hours 1 hour 2 hours 4 hours 8 hours 16 hours Base-1 4332 4839 4845 4845 4845 4845 Base-2 4971 5320 5328 5328 5328 5328 SER-Crawler 6071 7952 8377 8952 9101 9224 RSER-Crawler 7223 8751 9007 9116 9220 9233

[0055] Compared with the four methods, the method RSER-Crawler proposed in this paper obtains the most ideal effect. The amount of event information obtained by methods Base-1 and Base-2 is significantly less than that of methods SER-Crawler and RSER-Crawler.

[0056] The main reason is that both methods Base-1 and Base-2 are expanded around 10 specific built-in targets, and the expanded URL surface is very limited. The reason why method Base-2 is better than Base-1 is that when expanding the URL, anchor tags and text tags corresponding to the URL link are added, which has a certain heuristic guidance and avoids blind expansion. Methods Base-1 and Base-2 both completed the collection task in about 1 hour, and the relevant event information page could no longer be found. Methods Base-1 and Base-2 rely on URL seed expansion collection. This model is more suitable for comprehensive expansion collection on the Internet, and has great limitations on the collection of specific targets.

[0057] For the industry SaaS platform, the SER-Crawler method completed a round of collection tasks in about 2 hours, and the amount of information changed very little in the later period, while the RSER-Crawler method collected a large amount of relevant information in about 0.5 hours and completed a round of collection tasks in about 1 hour. For the regional SaaS platform, the SER-Crawler method completed a round of collection tasks in about 4 hours, and the amount of information changed very little in the later period. The main reason is that the sub-event rules (6832) of the regional SaaS platform are much more than the sub-event rules (1667) of the industry SaaS platform. The RSER-Crawler method collected a large amount of relevant information in about 1 hour and completed a round of collection tasks in about 2 hours. Comprehensively considering the industry SaaS platform and the regional SaaS platform, the collection efficiency of the RSER-Crawler method is much higher than that of the SER-Crawler method. Especially for some requirements for real-time collection of event information, the RSER-Crawler method can better reflect its collection advantages after the simplification of event rules.

[0058] The method described in the present invention is not limited to the embodiments described in the specific implementation manner. Other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the technical innovation scope of the present invention.

Claims

1. A method for simplifying event rules in event information collection, characterized in that: The steps are as follows: A. Split the user-customized event rule ER into sub-event rule sets SER; B. Determine whether the SERs satisfy the intersection relationship, select event rules through calculation, and obtain the simplified SER set LS; The specific steps of step (A) are as follows: A1. Determine whether the event rule ER needs to be split. ER consists of the "∧" and "∨" relationships between event elements. "∧" is an "and" relationship and "∨" is an "or" relationship. SER only consists of the "∧" relationship between event elements. When ER only has the "∧" relationship, it does not need to be split. When ER contains both "∧" and "∨" relationships, ER follows the "and" distribution law. i ∧(o i1 ∨o i2 )=(a i ∧o i1 )∨(a i ∧o i2 ), where a and o represent the action element and object element of the rule respectively; the number i at the bottom of a represents the number of the action element, the number i1 at the bottom of o represents the first action element of the i-th action element, and so on; A2. Calculate the number Num of "and" distributive laws in ER; A3. Perform Num times of "AND" distributive operation on ER to obtain ER i , split ER according to the "∨" relationship i , get the sub-event rule set SER; ER i The number i in the lower corner indicates the action element; The specific steps of step (B) are as follows: B1. Loop and take out each SER from SER i (1≤i≤m); B2, judge SER i With each subsequent SER j (i+1≤j≤m) whether it satisfies the intersection relation, i.e. SER i and SER j The action element a satisfies a i =a j , and the three elements of time t, location l, and object o satisfy t i , l i , o i With t j , l j , o j The same number u≥1; B3. By SER i and SER j Get SER i , j = a i ∧A(S i，j ), where S i，j It is SER i SER j The set of the same elements in t, l, and o, A(S i，j ) is for S i，j The "∧" operation of the elements in the i，j Alternative to SER i , delete SER j ; B4, loop through B2 and B3 to obtain the reduced SER set LS; i, j represent the i-th, j-th sub-event rule set, and m is a natural number.

Images