A main event deduplication method based on graph node selection and optimization

Abstract

The present application discloses a method for deduplication of subject events based on graph node selection and optimization, including: obtaining subject event text data, extracting subject events, and generating subject events to be detected; constructing a connected subgraph of subject events to obtain TOP‑K leading nodes; calculating similarity and attribute correlation between the leading nodes in each connected subgraph of subject events and the subject event to be detected; judging whether it is a suspected duplicate event based on whether the similarity and attribute correlation are within a preset threshold interval; calculating the spatiotemporal distance and importance change degree between the suspected duplicate event and the connected node; and further determining whether the suspected duplicate event and the connected node are clear duplicate events. The present application realizes real-time deduplication detection and real-time deduplication analysis through the structure of the graph, and adds financial subject event-related features in subsequent calculations to achieve the goal of improving the accuracy of deduplication.

Description

Technical Field

[0001] The present application belongs to the field of financial big data technology, and specifically relates to a method for deduplicating main events based on graph node selection and optimization. Background Art

[0002] With the Internet and information technology highly developed today, the Web has become an important source of information for people. Event information that appears in text form and focuses on target entities provides an important way for people to obtain a large amount of valuable information from the Web. These event information describing entities such as "enterprises", "people", and "products" are of great significance in the fields of market intelligence analysis, public opinion analysis, e-commerce, and business intelligence, and also play a vital role in people's work and social production. We call these events that describe entities subject events.

[0003] The Web is a free and open space. Event information describing entities has the characteristics of diverse forms, arbitrary release, and free expression. It is a very common phenomenon to use different expressions for the same subject event. At the same time, with the continuous development of the Internet, the amount of subject event information on the Web has also exploded (Information Explosion), which has brought great difficulties to the efficient deduplication of subject events. Therefore, how to accurately and efficiently distinguish subject events with different expressions and the same meaning is a problem that needs to be solved.

[0004] However, the current main event deduplication method cannot achieve real-time text deduplication and cannot accurately obtain the main event. The characteristics of financial news itself and the characteristics of event development are crucial to the deduplication method. Therefore, we extract relevant features from financial news and calculate the main event duplicate labels together with similarity to improve the accuracy of the deduplication method. The graph structure can reduce a large amount of repeated calculations to achieve the purpose of real-time detection. Summary of the invention

[0005] This application proposes a method for deduplicating main events based on graph node selection and optimization, and a method for performing deduplication operations in news main events to improve the efficiency and accuracy of deduplication.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] A method for deduplicating main events based on graph node selection and optimization includes the following steps:

[0008] Acquire subject event text data entering the domain, extract subject events in the subject event text data through professional vocabulary and vocabulary ratings and labels, and generate subject events to be detected;

[0009] Constructing a main event connected subgraph, obtaining a node clustering coefficient of each node of each connected subgraph, and obtaining TOP-K leader nodes according to the node clustering coefficient;

[0010] Calculating the similarity between the leader node in each of the subject event connected subgraphs and the subject event to be detected, and calculating the attribute relevance based on the professional vocabulary;

[0011] Determine whether the calculation result of the similarity and the calculation result of the attribute correlation belong to the preset threshold range. If so, determine that the subject event to be detected is a suspected repeated event. If not, establish a new connected subgraph and obtain a non-repeated label;

[0012] Calculating the temporal and spatial distance of the development between the suspected repeated event and the connected node and the degree of importance change;

[0013] Preset threshold judgments are respectively performed on the spatiotemporal distance and the importance change degree to determine whether the suspected repeated event and the connected node are clear repeated events.

[0014] Preferably, the subject event includes a characteristic word set related to the financial field, a sentence set of characteristic words, an entity of the subject event, remaining entities of the subject event and time.

[0015] Preferably, the words in the professional word library are divided into positive words, negative words and neutral words. When a word cannot be determined to be a positive word or a negative word, it is classified as a neutral word.

[0016] The professional vocabulary includes three parts. The first part is the label, which indicates the maximum relevant information of the feature word. The second part is the feature word, which indicates the feature word of the special proof sentence. The third part is the level, which indicates the degree of each feature word itself.

[0017] Preferably, the node clustering coefficient of each node is calculated by using the constructed main event connected subgraph to count the degree of each node in the graph and the number of stable triangle structures formed with its neighboring nodes;

[0018] According to the node clustering coefficient, TOP-K leader nodes are selected through a sorting algorithm.

[0019] Preferably, the vector representation of the subject event to be detected and the subject event of the leader node is obtained by TF-IWF calculation, and the cosine similarity is used to represent the similarity between the two.

[0020] According to the label mapping relationship in the professional vocabulary, the intersection and union between the feature word mapping label of the subject event to be detected and the feature word mapping label of the leader node are obtained, and the attribute relevance is obtained based on the intersection and the union.

[0021] Preferably, when the similarity calculation results and the attribute correlation calculation results of the two subject events to be detected are both greater than a preset threshold, the subject event to be detected is determined to be a suspected repeated event;

[0022] When any one of the similarity calculation results and the attribute correlation calculation results of the two subject events to be detected is not greater than a preset threshold, the subject event to be detected is determined to be a non-repetitive event.

[0023] Preferably, the time difference between the suspected repeated event and the release time of the leader node is calculated, and when there is a time difference between the two events, the spatiotemporal distance is calculated.

[0024] Preferably, the importance of event development is extracted, the importance of events is scored according to the levels of feature words, the importance scores of each feature word of the suspected repeated events and the importance scores of each feature word of the leading node are calculated, and the degree of change under the heavy pressure is further calculated.

[0025] The beneficial effects of this application are:

[0026] This application discloses a method for deduplicating main events based on graph node selection and optimization, which uses a combination of a statement-based event representation method and an attribute-based event representation method to represent main events. Real-time deduplication detection and analysis are achieved through the structure of the graph, and relevant features of financial main events are added in subsequent calculations to achieve the goal of improving the accuracy of deduplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solution of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 This is a flow chart of a method for deduplicating main events based on graph node selection and optimization according to the first embodiment of the present application;

[0029] Figure 2 This is a schematic diagram of the data preprocessing process in step 1 of Example 1 of the present application;

[0030] Figure 3A schematic diagram of a process for establishing a main event connected subgraph in step 2 of embodiment 1 of the present application;

[0031] Figure 4 This is a schematic diagram of a main event connected subgraph in Example 1 of the present application;

[0032] Figure 5 This is a schematic diagram of the overall process of the first stage of Example 1 of the present application;

[0033] Figure 6 This is a schematic diagram of the overall process of the second stage of Example 1 of the present application;

[0034] Figure 7 This is a flow chart of a method for deduplicating main events of CATL according to Embodiment 2 of the present application;

[0035] Figure 8 This is a schematic diagram of the node clustering coefficient of the connected subgraph of Example 2 of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0038] Embodiment 1

[0039] The main event deduplication method based on graph node selection and optimization proposed in the embodiment of the present application is a method for performing deduplication operations in news main events, which improves the efficiency and accuracy of deduplication. Figure 1 As shown, the main steps include:

[0040] Step 1: Prepare the subject event data set, crawl the main financial information websites, process the data, extract the subject events and determine the data labels by professional data personnel to obtain the final subject events to be detected; obtain the professional vocabulary in the financial field through professional data analysts, which contains the corresponding labels and levels.

[0041] In this embodiment, the crawled financial data is processed by extracting the company entities in the data, determining their characteristic words and characteristic sentences that can accurately describe the data, and attaching duplicate labels to the data according to the release time or crawling time of the data. The professional vocabulary in the financial field requires professional data analysts in the field to construct internal labels of various levels and label scores in a targeted manner.

[0042] Specifically, we crawl news from major news media websites and filter out finance-related news. The crawled data attributes include news ID, news release time, news crawling time, and news text. Each piece of data can correspond to one or more main events.

[0043] Construct a professional vocabulary LE in the financial field. The vocabulary is divided into positive words, negative words and neutral words. When a word cannot be determined to be positive or negative, it is classified as a neutral word. In the vocabulary LE, each type contains three parts. The first part is the label, which indicates the maximum relevant information of the feature word. The second part is the feature word, which indicates the feature word of the special proof sentence. The third part is the level, which indicates the degree of each feature word itself. These three parts are determined by professional data analysts.

[0044] After the data preparation is completed and the professional financial domain vocabulary LE is constructed, the subject event is extracted. In this embodiment, the subject event is represented by the symbol E, and its expression is shown in formula (1).

[0045] E=(W,S,C,O,T) (1)

[0046] Among them, W represents the feature word set, S represents the sentence set containing the feature words, C represents the entity of the main event, O represents the remaining entities of the event, and T represents all timestamps of the event.

[0047] According to the representation of the main events, all the main events can be extracted from the news text, the entities in the news text can be identified, the sentences where the entities and the characteristic words coexist can be extracted as event sentences, and the event sentences extracted from the news text can be spliced ​​as the main events. Figure 2 Shown

[0048] Step 2: In the graph-based calculation method, the node clustering sparsity of the graph is first calculated, and the TOP-K leader nodes of each connected subgraph under the subject are obtained according to the node clustering sparsity of each node.

[0049] In this embodiment, the deduplication detection method is based on the connected subgraph of the subject event. It is necessary to perform graph processing on the historical data of the subject, and record the ID and repeated data group of the duplicate data. A connected subgraph is constructed according to the duplicate information and grouping (if there is an edge between two data, there is a connection path between them, and two data without a connected path are distributed in different connected subgraphs). Using the constructed connected subgraph, the degree of each node in the graph and the number of stable triangle structures formed with its neighboring nodes are counted to calculate the clustering coefficient of each node, and TOP-K nodes are selected as the leading nodes of the connected subgraph.

[0050] Specifically, the connected subgraph construction process is as follows Figure 3 The main event connected subgraph is represented by G c It is expressed as shown in formula (2).

[0051] G c =(V,A)(E∈V&E(C)=c) (2)

[0052] Among them, V is the main event set in the connected subgraph, A is the edge relationship between the main event sets, G c It represents a connected subgraph under the current subject c.

[0053] There are multiple connected subgraphs under a subject c, as shown in the schematic diagram Figure 4 Therefore, in this embodiment, all connected subgraphs under the subject c are represented as a set D c , whose expression is shown in formula (3).

[0054]

[0055] The node clustering coefficient is used to select the leading node set of the connected subgraph. The node clustering coefficient mainly indicates the closeness between the node and its neighboring nodes. The larger the coefficient value, the closer the connection between the node and its neighboring nodes, and vice versa. c =(V, A), V represents the set of all nodes, A represents the set of all edges, and edge a(u,v)∈A represents the interconnection between node u and node v. The neighbor set N(u) of node u is a set of nodes, which can be expressed by formula (4).

[0056] N(u)={v∈V|(u, v)∈A} (4)

[0057] Based on the neighbor set of the node, the clustering coefficient of the node is represented by NC(u), and the specific representation is shown in formula (5):

[0058]

[0059] In the above formula, Z u,xIt represents the number of triangles actually formed by edge a(u, x) and neighboring nodes, and deg(x) represents the degree of node x. NC(u) represents the influence of node u on neighboring nodes. The higher the NC(u) value, the stronger the leadership of node u on its neighbors.

[0060] The node clustering coefficient of each node in each connected subgraph can be obtained by this step 2. According to the node clustering coefficient, the TOP-K leader nodes are selected by a sorting algorithm. The leader node set subgraph in each node is shown in formula (6).

[0061]

[0062] Among them, c represents the entity, Represents the set of top K leader nodes of the nth connected subgraph under entity c.

[0063] Step 3: The subject event to be detected and the leader node of the subject's graph history data Perform one-to-one calculations to obtain the similarity FS value of the current two main events; use the labels of the financial domain vocabulary LE to calculate the main event to be detected and the leader node of the subject's graph history data The attribute correlation FW value.

[0064] In this embodiment, the main event and the leading node are vectorized by TF-IWF, and the cosine similarity is calculated to indicate the similarity between the two. The feature word library in the financial field constructed by the domain data analyst has three levels of labels and a label importance score. The feature words of the main event and the leading node are mapped to the first-level label, and the number of the same first-level labels and the number of the union of the first-level labels of the two are counted. Then the importance change value of the main event and the leading node is calculated.

[0065] Specifically, the subject event to be detected is represented as It represents the pth subject event under subject c, and the leader node of the graph historical data of the subject comparison calculation is represented as represents the kth leader node of the qth subject event connected subgraph under subject c, which belongs to the set The vector representations of the subject event to be detected and the subject event of the leader node calculated by TF-IWF are shown in Equation 7 and Equation 8 respectively:

[0066]

[0067]

[0068] The cosine similarity is used to calculate the similarity of the two main events in the first stage. The similarity calculation formula is shown in formula (9):

[0069]

[0070] In the above formula, Represents the vectorized subject event to be detected, Represents the leader node for vectorization.

[0071] The feature words of the subject event to be detected are a type of thesaurus. There are spelling differences (such as word order, different expressions, parts of speech, etc.), grammatical differences (such as homonyms) and word meaning differences (heteronyms) in thesaurus. Therefore, these differences still exist in the feature word list. However, these differences cannot be identified in traditional calculations. Therefore, it is necessary to perform equivalence mapping and hierarchical mapping on the feature word list. The differences between feature words are more detailed, making the algorithm's duplicate detection more accurate.

[0072] In this embodiment, the mapping between feature words is clarified in the professional word library LE in the financial field, and the correlation between the attributes of the subject event can be calculated through the feature word library. In financial news, if the feature words of the two subject events being compared belong to the same category, it is considered that the descriptions are of the same subject event. If the feature words of the two subject events being compared do not belong to the same category, it is considered that the descriptions are not of the same category. The feature words of the subject event to be detected are W x , through the mapping rules, it can be mapped to l in the professional library LE in the financial field y Label. Its label mapping function is f, and the specific relationship is shown in formula 10:

[0073] f:W x → y (10)

[0074] Calculate the intersection of the feature word mapping label of the subject event to be detected and the feature word mapping label of the leader node, and the intersection set l of the mapping category labels of the current two subject events g The expression of is shown in formula 11:

[0075]

[0076] Calculate the union of the feature word mapping label of the subject event to be detected and the feature word mapping label of the leader node, and the mapping category label union set l of the current two subject events h The expression of is shown in formula 12:

[0077]

[0078] By analyzing financial news, by classifying the feature words and categorizing the feature word classification labels, the differences of the feature words can be distinguished more accurately. Therefore, this embodiment analyzes the feature word attributes of the subject event and obtains the subject event attribute relevance calculation formula as shown in formula 13

[0079]

[0080] In the above formula, the numerator represents the sum of the characteristic coefficients of the same category of the two main events, and the denominator represents the sum of the characteristic coefficients of all categories of the two main events.

[0081] In this step 4, the higher the feature word relevance, the more similar the category, the closer the FW value is to 1, and vice versa. Therefore, a feature word relevance threshold Z is selected to determine the relevance within a certain range as the required high feature word relevance, that is, when FW belongs to (0, Z), the feature word relevance is low, and when FW belongs to (Z, 1), the feature word relevance is high. The FS and FW calculated in the first stage are used for the subject event to be detected. Historical data of the chart Make a judgment and distinguish based on the following thresholds to determine whether to connect to an existing connected subgraph or to build a new connected subgraph.

[0082] Step 4: Determine whether the similarity FS value and the attribute relevance FW value are within the set threshold range. If so, output the duplicate label; otherwise, construct a new connected subgraph and give it a non-duplicate label.

[0083] In this embodiment, corresponding thresholds are set for FS and FW. After comparing the text similarity and attribute relevance between the subject event and the leader node with the threshold, duplicate labels are attached to the subject event and the connected subgraph is updated.

[0084] Specifically, if the FS and FW of two main events are both greater than the threshold, the two main events are repeated events, as shown in Formula 14.

[0085] FS>α and FW>β y→True (14)

[0086] If any one of the FS and FW values ​​of the two main events is not greater than this value, then the two main events are non-repetitive events, as shown in Formula 15.

[0087] FS≤ α or FW≤β y→False (15)

[0088] Update the connected subgraph under the subject according to the obtained label. The specific flow chart of the first stage is as follows: Figure 5 shown.

[0089] Step 5: Utilize the development of the main events and extract time-related features to calculate the connected main events The spatiotemporal distance SP value of the graph node to which it is connected; according to the change in the importance of the main event, the connected main event is calculated The importance of the graph nodes it is connected to changes in SI values.

[0090] From the above, we can see that the same connected subgraph is a similar main event. However, when the event has developed but the feature words are similar and mapped to the same category, it is impossible to accurately distinguish by simply calculating the similarity and feature word relevance. The main reason for this is that events are developing and have an evolving process, but this feature was not selected in the first stage. The above-mentioned duplicate detection results temporarily locate suspected duplicate events. The next step is to extract the feature that the event is evolving and developing, and use this feature to optimize the node judgment.

[0091] In this embodiment, time-related features are extracted, and the data crawled in step 1 will record the data release time and the data crawling time. When calculating time-related features, if the data release time does not exist, the data crawling time is used as its data release time. Calculate the time difference between the main event and the leader node event, and use the time difference result to calculate the spatiotemporal distance of the two data. According to the selection range of historical data, the two parameters in its spatiotemporal distance formula are selected from the following parameter pairs: (7, 0.7), (14, 0.3), (21, 0.1). To calculate the importance change value of the main event and the leader node, it is necessary to add the importance scores of the feature words of the main event and the leader node to obtain the importance score of the event. Finally, take the ratio of the two as the importance change value.

[0092] Specifically, the time difference between the data to be detected and the leader node data is calculated according to their release time. The time difference d(ti,tj) between the two main events is expressed as shown in formula 16:

[0093]

[0094] When doing repeated data comparison, the corresponding time attenuation coefficient γ is set according to the time interval value T of reading historical data. In this embodiment, the values ​​of T and γ are changed in pairs, the purpose is to control the error caused by excessive time difference, and the set parameter pairs are (7, 0.7), (14, 0.4), (21, 0.1) respectively.

[0095] In the second stage, this embodiment extracts the spatiotemporal distance characteristics of the main event development. The spatiotemporal distance is used to determine whether the space of the main event has changed and the degree of change. The calculation formula for the spatiotemporal distance of the main event is SP as shown in formula (17).

[0096]

[0097] In the above formula, ti and tj represent The time of occurrence.

[0098] When there is a time difference between the two events, the time-space distance between the two main events is calculated. By setting the time-space distance threshold U, we determine that when the time-space distance value of the two main events is within a certain range, the main event at this time has developed, that is, when the time-space distance SP belongs to (0, U), the two main events have a time-space distance, and the main event to be detected is developing. If SP belongs to (U, 1), the two main events do not have a time-space distance, and the main event to be detected has not developed; when there is no time difference between the two events, the time-space distance SP value is 1.

[0099] The temporal and spatial distance of the main event alone cannot accurately determine the development of the event. Therefore, this embodiment extracts the characteristics of the importance of the event development, scores the importance of the event according to the level of the feature word, and divides the importance of the main event according to the score value. The importance score Score corresponding to each feature word can be found in the financial field feature library LE. The importance formula r of its event attribute is expressed as shown in Formula 18:

[0100]

[0101] The importance scores of the feature words of the data to be detected and the leading node are counted, and their ratio is calculated to express the importance change SI of the two. The calculation formula of SI is shown in formula (19).

[0102]

[0103] If SI is greater than or less than 1, the importance of the two main events will change; otherwise, the importance of the two main events will not change.

[0104] The SP and SI calculated in the second stage are used to judge the connected main events and the main events of the graph history data.

[0105] Step 6: Determine whether the spatiotemporal distance SP value and the importance change SI value are within the set threshold range. If so, modify If it is a non-duplicate label, the connection is broken; otherwise, the original node connection is maintained.

[0106] In this embodiment, corresponding thresholds are set for SP and SI. In the second stage, the deduplication labels and connected subgraphs of the main events are updated according to the spatiotemporal distances and importance change values ​​between the main events and the leading nodes.

[0107] Specifically, if the SP of the two main events belongs to (0, U], and SI is not equal to 1, then the two main events are repeated events, as shown in Formula 20:

[0108] 0<SP≤U and SI≠1 y→True (20)

[0109] If the SP of these two main events belongs to (U, 1] and SI is not equal to 1, then these two main events are non-repeating events, as shown in Equation 21;

[0110] U<SP≤1 and SI≠1 y→False (21)

[0111] If the label is found to have changed through calculation, it is considered that the event is changing and developing with the event, and the connected subgraph is updated according to the new label. The specific process of the second stage is as follows Figure 6 shown.

[0112] Furthermore, in this embodiment, by comparing the labels obtained by the method with the labels obtained by the domain data analyst in the original data, the accuracy, recall rate and F1-SCORE value of the method are obtained.

[0113] It has been verified that the technical solution of this application can realize real-time deduplication detection and add relevant features of financial entity events in subsequent calculations to achieve the goal of improving the accuracy of deduplication.

[0114] Embodiment 2

[0115] In this second embodiment, the deduplication process is described by taking the subject Contemporary Amperex Technology Co., Ltd. as an example. The overall process is as follows: Figure 7 As shown:

[0116] 1. For CATL, 1,898 pieces of data were extracted for testing by subject attribute matching. The crawled news format and the extracted test data are shown in Tables 1 and 2.

[0117] Table 1

[0118]

[0119] Table 2

[0120]

[0121] According to the above data, the specific implementation includes the following steps:

[0122] 2. Construct a connected subgraph g based on the input data and the deduplicated historical data. One of the connected subgraphs is as follows: Figure 8For example, the algorithm code input is represented as an adjacency matrix A, which is obtained by removing the duplicate relationship between each data in the historical data.

[0123] A 2 =[...], ..., A max_group =[...] (22)

[0124] example:

[0125] A 2 =[...], ..., A max_group =[...] (23)

[0126] The clustering coefficient of each node in each connected subgraph is calculated using the adjacency matrix obtained above, and then the leader node of each connected subgraph is found through the sorting algorithm. Taking one of the connected subgraphs as an example, its leader node set is calculated, and the leader node selection method of the remaining connected subgraphs is the same.

[0127] The neighbor set N(u) of node u is a set of nodes, which can be expressed by the following formula:

[0128] N(u)={v∈V|(u, v)∈A} (24)

[0129] Example: N(id 1 )={id 2 , id 4 , id 5 , id 6 , id 7 , id 8 , id 9 , id 10 , id 17}

[0130] Z(u, x) represents the number of triangles formed by nodes u and x and their adjacent nodes.

[0131] example:

[0132] x=id 2 ,Z(id 1 , id 2 )=1

[0133] x=id 4 ,Z(id 1 , id 4 )=3 ...

[0135] x=id 17 ,Z(id 1 , id 17 )=1

[0136] deg(x) represents the degree of node x.

[0137] Example: deg(id 2 )=3,deg(id 4 )=5,...,deg(id 17 )=3

[0138] Generate the clustering coefficient of node u, whose value is related to its neighbor nodes and the degree of neighbor nodes. The clustering coefficient is obtained by the number of stable triangle structures and the degree of neighbor nodes calculated above.

[0139]

[0140] example:

[0141] The clustering coefficient of each node in the connected subgraph calculated through the above steps is stored in a dictionary.

[0142] Example: {id 1 :4.02,id 2 :0.77,id 4 :1.87,id 5 :1.33,id 6 :2.43,...,id 17 :0.75}

[0143] The bubble sort algorithm is used to obtain the top k nodes in terms of clustering coefficient value and use them as the leading nodes for subsequent comparison.

[0144] Example: The top 2 leader nodes {id 1 , id 6}

[0145] After calculating the leader node of a connected subgraph, the leader nodes of all connected subgraphs under the topic are calculated repeatedly through step 2 and stored in the form of a dictionary.

[0146] Example: {group 1 :{id 1 , id 6}, group 2 :{...},...,group max :{...}}

[0147] Traverse the leader nodes of each connected subgraph, compare the undetected data with the leader nodes, and output the deduplicated labels.

[0148] 3. The steps for calculating the similarity between new data and the leader node are as follows:

[0149] The TF-IDF method is used to extract keywords and related semantic information from the full text content and event feature sentences of new data and historical data, and store them in the form of a confusion matrix.

[0150] example:

[0151]

[0152]

[0153] Convert the obtained confusion matrix into a vector representation

[0154] example:

[0155]

[0156]

[0157] The cosine similarity between the data to be detected and the leader node is calculated through the vectorized subject events.

[0158]

[0159] example:

[0160] FS{[0,0.038848657002,...,0.0466913581897,...,0.089370787890,...,0.1369495173304,...,0.094115528654,...,0],[ 0, 0.2013813042871,..., 0.1427978469519,..., 0.25144172143,..., 0.139658726358,..., 0.1928200194744,..., 0]}=0.85

[0161] The feature words are mapped to the first-level labels of the expert-generated feature word library. The mapping rules are as follows:

[0162] f:W x → y (27)

[0163] Wx is a single feature word in the subject event feature word set, and each Wx corresponds to a first-level label in the expert feature word library, namely, a category label.

[0164] Example: f(stock price strengthens) → stock price is positive

[0165] When comparing new data with the leader node, take the intersection of the feature words of the two for label mapping and calculate the total number of mapped labels. Then take the union of the feature words of the two for label mapping and calculate the total number of union label mappings.

[0166]

[0167] example:

[0168] l g =[‘Stock price positive’, ‘Production advantage’, ‘Good capital flow’, ‘Macro positive’] ∩[‘Production advantage’, ‘Macro positive’, ‘Good capital flow’] = [‘Production advantage’, ‘Good capital flow’, ‘Macro positive’]

[0169] l h =[‘Stock price is favorable’, ‘Production advantage’, ‘Good capital flow’, ‘Macro favorable’]∪[‘Production advantage’, ‘Macro favorable’, ‘Good capital flow’]=[Stock price is favorable’, ‘Production advantage’, ‘Macro favorable’, ‘Good capital flow’]

[0170] FW = len(['production advantage', 'good capital flow', 'favorable macroeconomics']) / len('favorable stock price', 'production advantage', 'favorable macroeconomics', 'good capital flow') = 0.75

[0171] 4. The similarity value FS and the feature word association degree FW obtained in the previous step are compared with the initial threshold. If the FS and FW values ​​of the two are greater than the threshold, the two are considered similar and a duplicate label y = True is attached to the new data.

[0172] Example: FS=0.85>0.7 or FW=0.75>0.7→True

[0173] In the first stage, the connected subgraph is constructed, the leader node is selected, and the text similarity and feature word relevance are used to obtain duplicate labels and update the connected subgraph. Next, the updated edges are optimized.

[0174] 5. Take the release time of the news event and calculate the time difference between the new data and the comparison leader node data. If the new data release time ti is greater than or equal to the leader node data tj, the value of the time difference d(ti, tj) between the two is ti-tj. Otherwise, take a constant value of 0.

[0175]

[0176] Example: d(2021-01-05 11:00:00, 2021-01-05 16:46:00) = 286

[0177] The release time of a news event will cause a time-space difference between two news data. Its value has a certain relationship with the release time difference between the two. The adjustment parameter T,γ is used to calculate the time difference.

[0178]

[0179] There is a pairing relationship between the time interval T of the selected historical data and the parameter γ. In this method, (7, 0.7), (14, 0.4), and (21, 0.1) are selected as the relationship values ​​between the two.

[0180] example:

[0181] SP(2021-01-05 11:00:00, 2021-01-05 16:46:00)=3.794703235298559e -13

[0182] Read the importance scores of each feature word label from the financial domain vocabulary, add the importance scores of the data to be detected and the main event features of the leader node, and take the ratio of the two values ​​as the importance change

[0183]

[0184] example:

[0185] SI(['Production advantage', 'Good capital flow', 'Macro-positive'], ['Stock price positive', 'Production advantage', Good capital flow', 'Macro-positive']) = 0.43

[0186] The spatiotemporal distance SP and the importance change SI obtained in this step are compared with the initial threshold. If the SP of the two is greater than the threshold and the SI is not equal to 1, it is considered that the two are not similar, and y=False.

[0187] example:

[0188] SP=3.794703235298559e -13 <0.7 and SI('Stock price positive', 'Production advantage', 'Macro positive', 'Good capital flow'], ['Production advantage, 'Macro positive', 'Good capital flow']) = 0.43 ≠ 1y → True

[0189] 6. Update the connected subgraph according to the duplicate labels obtained in the above steps. If y=True, build an edge between the new data and the leader node. Otherwise, compare it with the leader nodes in other connected subgraphs until the data is no longer similar to any leader node, and put the data into the new connected subgraph.

[0190] 7. Select 1898 valid financial news data of CATL after data preprocessing. The data set is divided into four parts: small, midlle, big, and lager. The financial news data is selected as the data set within 7 days as the small data set, the data set within 14 days as the middle data set, the data set within 21 days as the big data set, and all the data as the lager data set. The method of this application predicts a repeated label for this batch of data, and the accuracy (Acc), recall (Recall) and F1-SCORE value of the method are calculated by comparing the repeated labels determined by professional data personnel and the labels predicted by the method, as shown in Table 3. By comparing with the original labels, the accuracy, recall and F1-SCORE values ​​of the method are obtained.

[0191] Table 3

[0192]

[0193] The embodiments described above are only descriptions of the preferred methods of the present application, and are not intended to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made to the technical solutions of the present application by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present application.

Claims

1. A method for deduplicating main events based on graph node selection and optimization, characterized in that: The steps include: Obtain subject event text data in the financial field, extract subject events in the subject event text data through professional vocabulary and vocabulary ratings and labels, and generate subject events to be detected; Constructing a main event connected subgraph, obtaining a node clustering coefficient of each node of each connected subgraph, and obtaining TOP-K leader nodes according to the node clustering coefficient; Calculating the similarity between the leader node in each of the subject event connected subgraphs and the subject event to be detected, and calculating the attribute relevance based on the professional vocabulary; Determine whether the calculation result of the similarity and the calculation result of the attribute correlation belong to the preset threshold range. If so, determine that the subject event to be detected is a suspected repeated event. If not, establish a new connected subgraph and obtain a non-repeated label; Calculating the temporal and spatial distance of the development between the suspected repeated event and the connected node and the degree of importance change; Preset threshold judgments are respectively performed on the spatiotemporal distance and the importance change degree to determine whether the suspected repeated event and the connected node are clear repeated events.

2. The method for deduplicating main events based on graph node selection and optimization according to claim 1, characterized in that: The subject event includes a characteristic word set related to the financial field, a sentence set of characteristic words, an entity of the subject event, remaining entities of the subject event and time.

3. The method for deduplicating main events based on graph node selection and optimization according to claim 2 is characterized in that: The words in the professional word library are divided into positive words, negative words and neutral words. When a word cannot be determined to be a positive word or a negative word, it is classified as a neutral word; The professional vocabulary includes three parts. The first part is the label, which indicates the maximum relevant information of the feature word. The second part is the feature word, which indicates the feature word of the special proof sentence. The third part is the level, which indicates the degree of each feature word itself.

4. The method for deduplicating main events based on graph node selection and optimization according to claim 1, characterized in that: Using the constructed main event connected subgraph, the degree of each node in the graph and the number of stable triangle structures formed with its neighboring nodes are counted to calculate the node clustering coefficient of each node; According to the node clustering coefficient, TOP-K leader nodes are selected through a sorting algorithm.

5. The method for deduplicating main events based on graph node selection and optimization according to claim 1, characterized in that: The vector representation of the subject event to be detected and the subject event of the leader node is obtained by TF-IWF calculation, and the cosine similarity is used to represent the similarity between the two. According to the label mapping relationship in the professional vocabulary, the intersection and union between the feature word mapping label of the subject event to be detected and the feature word mapping label of the leader node are obtained, and the attribute relevance is obtained based on the intersection and the union.

6. The method for deduplicating main events based on graph node selection and optimization according to claim 1, characterized in that: When the similarity calculation results and the attribute correlation calculation results of the two subject events to be detected are both greater than a preset threshold, the subject event to be detected is determined to be a suspected repeated event; When any one of the similarity calculation results and the attribute correlation calculation results of the two subject events to be detected is not greater than a preset threshold, the subject event to be detected is determined to be a non-repetitive event.

7. The method for deduplicating main events based on graph node selection and optimization according to claim 1, characterized in that: The time difference between the suspected duplicate event and the release time of the leader node is calculated, and when there is a time difference between the two events, the time-space distance is calculated.

8. The method for deduplicating main events based on graph node selection and optimization according to claim 1, characterized in that: The importance of event development is extracted, and the importance of events is scored according to the levels of feature words. The importance scores of each feature word of the suspected repeated event and the importance scores of each feature word of the leading node are calculated, and the importance change degree is further calculated.

Images