News text analysis method based on graph analysis technique

Abstract

The application provides a news text analysis method based on a graph analysis technique, collects mass news text data, calculates a multi-level mixed similarity based on a plurality of natural language processing methods, uses a graph analysis method to quantize weights of nodes and edges and constructs a time-varying news knowledge graph, uses a text classification method and information decay evaluation to construct a time sequencing index, finally forms a classified and time-sequencing text index, and on this basis, realizes quantitative measurement of influence and attributes of events represented by the text index on a field related index by using a sliding window causal test, wavelet analysis and other methods; the application solves the technical problem that it is difficult to accurately identify, extract and process influence factors of a market of a field to be analyzed based on mass network information, thereby deepening cognition of a market influence system of the field to be analyzed and better realizing application of big data in construction of the market influence system of the field to be analyzed.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a news text analysis method based on graph analysis technology. Background Technology

[0002] In the context of the big data era, how to combine massive amounts of unstructured internet data, use appropriate methods to process and extract effective information from it, and thus analyze relevant information in specific fields to better provide data support for decision-making, is a key focus in research fields such as statistics and finance. Research fields such as energy, finance, and economics are characterized by a wide variety of influencing factors, complex mechanisms of action, and significant research and practical implications. Their data foundations, data structures, and analytical methods have good points of integration and research foundation with modern information technology and big data.

[0003] Currently, technologies similar to this invention include "Text Data Cleaning Method, Apparatus, Computer Equipment and Medium for Energy Equipment" (CN 115878742 A) and "Structured Energy Text Data Based on Relation Extraction Method and Apparatus" (CN116383339 A). The embodiments disclosed in "Text Data Cleaning Method, Apparatus, Computer Equipment and Medium for Energy Equipment" disclose a text data cleaning method, apparatus, computer equipment and medium for energy equipment. The method includes: acquiring a text dataset to be processed; performing data cleaning processing on the text dataset to obtain cleaned text data to form a cleaned text dataset; concatenating the cleaned text data in the cleaned text dataset to obtain text samples to form text data; and generating a text vector table of the text dataset to be processed based on the text data. The paper "A Method and Apparatus for Structuring Energy Text Data Based on Relation Extraction" discloses a method and apparatus for structuring energy text data based on relation extraction. The method involves acquiring a first energy text dataset, extracting entity features from the energy text data in the first dataset, and establishing entity relationships between the energy text data to obtain a second energy text dataset. Based on the energy type of the energy text data, data augmentation is performed on the energy text data in the second dataset to obtain a third energy text dataset. Based on the third energy text dataset, a preset neural network model is iteratively trained until the preset neural network model reaches a preset convergence condition to obtain a data structuring model. This data structuring model is then used to structure the unstructured energy text data, generating structured data reports.

[0004] Based on the investigation of existing technologies, the limitations of existing technologies similar to this invention are reflected in the following aspects: (1) Existing technologies generally remain at the level of data processing, structuring and preliminary extraction of information, without further discussion on in-depth information mining and property-based discrimination in massive data; (2) Existing text data structuring methods are mostly based on supervised learning theory, often relying on a large amount of labeled data, which requires a high amount of time and manpower; (3) Existing technologies lack a theoretical framework for further application of the extracted information in their construction and description, especially the quantitative application of information extracted from specific fields. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a news text analysis method based on graph analysis technology. It constructs an integrated model framework for processing, analyzing, representing, quantifying, and applying exogenous unstructured information to the measurement task of the market dynamics influence system in the field to be analyzed. This can deepen the understanding of the market influence system in the field to be analyzed and better realize the application of big data in the construction of the energy market influence system.

[0006] A news text analysis method based on graph analysis technology includes the following steps:

[0007] S1: Assemble the news texts in the domain to be analyzed into a total set {N1, N2, ..., N} according to their publication time. n}, where N i ={h i ,t i}, i = 1, 2, ..., n, h i ,t i The news text N is respectively i Title and body text;

[0008] S2: Using a time window of a set length, extract multiple subsets from the total set of news texts according to a set step size. The number of news texts contained in each subset is not exactly the same.

[0009] S3: After obtaining the mixed similarity between any two news texts in each subset, construct the knowledge graph corresponding to each subset based on the mixed similarity. Each news text is a node in the knowledge graph, and there are edges between nodes with a mixed similarity of not 0. The weights of each node are not exactly the same.

[0010] S4: After merging the named entities and keywords of all news texts in the total set, a topic word hybrid set is obtained, and a co-occurrence graph is constructed based on the topic word hybrid set. The named entities and keywords in the topic word hybrid set are nodes in the co-occurrence graph, and there are edges between the nodes corresponding to two named entities or keywords that co-occur in the news text.

[0011] S5: Randomly cluster all nodes in the co-occurrence graph to obtain a set of categories C consisting of multiple categories with different numbers of nodes. Calculate the modularity of category set C, and adjust the number of categories and the number of nodes in each category according to the modularity until the optimal set of categories C with the minimum modularity is obtained. * and the category set C * The named entities and keywords corresponding to the nodes in each category form the topic word vector;

[0012] S6: Obtain the cosine similarity between each news text and each topic word vector in the overall news text set;

[0013] S7: Take the latest publication time of the news text contained in each subset as each specific time T. For each specific time T, obtain the time-series index sequence of news information intensity corresponding to that specific time T.

[0014] S8: Record the time series covered by all the news information intensity time series obtained in step S7 as the time series to be tested, and obtain the return series of the field to be analyzed within the time series to be tested.

[0015] S9: Evaluate the causal relationship between the time-series index sequence of news information intensity and the return rate sequence according to the set indicators;

[0016] S10: Evaluate the coherence between the time-series index series of news information intensity and the return series according to the set indicators.

[0017] Furthermore, the method for obtaining the time-series index sequence of news information intensity corresponding to each specific time T in step S7 is as follows:

[0018] S71: Extract the knowledge graph corresponding to the subset at the current specific time T;

[0019] S72: Obtain the cosine similarity between each news text and each topic word vector contained in the current knowledge graph, and the weight of the node corresponding to each news text contained in the current knowledge graph;

[0020] S73: For each current news text contained in the current knowledge graph, obtain the categorized weight of the current news text in each category based on the cosine similarity between the current news text and each topic word vector, and the corresponding node weight in the current knowledge graph.

[0021] S74: Based on the categorization weights of all news texts contained in the subset corresponding to the current specific time T under each category, obtain the time-series index sequence of news information intensity corresponding to the current specific time T.

[0022] Furthermore, the method for obtaining the cosine similarity between each news text and each topic word vector in the overall news text set in step S6 is as follows:

[0023]

[0024] in, For the i-th news text and the c-th... h The topic word vectors corresponding to each category The normalized proportion of cosine similarity between the categories, h = 1, 2, ..., H, where H is the set of categories C. * The total number of categories, For the i-th news text and the c-th... h The topic word vectors corresponding to each category Cosine similarity between them Let be the sum of the cosine similarities between the i-th news text and all topic word vectors.

[0025] Furthermore, the method for obtaining the categorical weights of the current news text under each category in step S73 is as follows:

[0026]

[0027] in, For the i-th news text at the c-th... h Categorical weights under each category For the i-th news text and the c-th... h The topic word vectors corresponding to each category The cosine similarity between the categories, h = 1, 2, ..., H, where H is the set of categories C. * The total number of categories, PR(t) is the sum of the cosine similarities between the i-th news text and all topic word vectors. i ) represents the weight of the i-th news text in the knowledge graph of the subset corresponding to the current specific time T.

[0028] Furthermore, the method for obtaining the time-series index sequence of news information intensity corresponding to the current specific time T in step S74 is as follows:

[0029] The subset corresponding to the current specific time T at the cth moment is obtained using the following formula. h News Information Intensity Time-Series Index under Each Category

[0030]

[0031] in, For the c-th time, all news texts in the subset corresponding to the current specific time T are... hThe sum of the categorical weights under each category, s = 1, 2, ..., T-1, where s is a specific time before the current specific time T. For all news texts in the subset corresponding to a specific time s at time c h The sum of the categorical weights under each category, where λ is the set exponential weight;

[0032] The time-series index of news information intensity of the subset corresponding to the current specific time T under each category This forms a time-series index sequence of news information intensity.

[0033] Furthermore, let N denote any two news texts in the subset. u and N v Then the news text N u And news text N v Mixed similarity W(N) between u N v The method to obtain ) is as follows:

[0034]

[0035] Wherein, L(t) u ,t v (N) is the news text. u The main text t u And news text N v The main text t v Keyword weighted similarity between them, D(h) u ,h v (N) is the news text. u Title h u And news text N v Title h v Field similarity between them, M(t) u ,t v (N) is the news text. u The main text t u And news text N v The main text t v Named entity co-occurrence similarity between them;

[0036] Field similarity D(h) u ,h v The calculation formula for ) is as follows:

[0037]

[0038] Among them, edit(h u ,h v |A,B) is the title h u With title h vEdit distance between them, A is the title h u The number of words in the title h, B is the title h. v The number of words in the edit(h) u ,h v A, B) are obtained recursively from the following formula:

[0039]

[0040]

[0041] Among them, edit(h u ,h v |a,b) is the title h u The a-th word in the title h v The edit distance between the b-th words in the text, where a∈[0,A], b∈[0,B], and edit[a-1][b] represents the title h. u The (a-1)th word in the title h v The edit distance between the b-th words in the text, edit[a][b-1] is the title h. u The a-th word in the title h v The edit distance between the (b-1)th words in the text, where edit[a-1][b-1] represents the title h. u The (a-1)th word in the title h v Edit distance between the (b-1)th words in the text, flag is the flag value, h u [a] is the title h u The a-th word in the text, h v [b] is the title h v The edit distances of the b-th word in the code are set to the initial values ​​of edit[0][0], edit[1][0], and edit[0][1].

[0042] Named entity co-occurrence similarity M(t) u ,t v The calculation method for ) is as follows:

[0043]

[0044] Where, sim(t) u ,t v ) represents the main text t u With the main text t v The number of identical named entities between them, count(t) u ) represents the main text t u The total number of named entities in the table, count(t) v ) represents the main text t v The total number of named entities in the text;

[0045] Keyword weighted similarity L(t) u ,t v The calculation method for ) is as follows:

[0046]

[0047] Among them, V p For text t to appear simultaneously u With the main text t v The keyword with the largest weight p, where p = 1, 2, ..., P, and P is at least 10. For the keyword V p In the main text t u The weights in For the keyword V p In the main text t v Weights in;

[0048] Current keyword V r The weight WS(V) of the main body text of the current news article r The calculation method for ) is as follows:

[0049]

[0050] Among them, In(V r () is to find the keyword V in the current text. r V is a set of adjacent words that have an adjacency relationship. q For the set of adjacent words In(V) r Keywords in ) In(V q To find the keyword V in the current text. q V is a set of adjacent words that have an adjacency relationship. k For the set of adjacent words In(V) q The keyword in ) is W qr For the keyword V q With the current keyword V r The weight of the connecting edge, W kq For the keyword V k With keyword V q The weight of the connecting edge, d is the damping coefficient, WS(V q () is the keyword V q The weight of the main body text of the current news article.

[0051] Furthermore, the method for setting the weights of nodes in each knowledge graph in step S3 is as follows:

[0052] The weight calculation formula for each node in the current knowledge graph is defined as follows:

[0053]

[0054] Where PR(N) is the weight of the node corresponding to the current news text N in the current knowledge graph, S(X) is the set of news texts with a non-zero mixed similarity to the news text N corresponding to the current node, N′ is the news text in set S(X), PR(N′) is the weight of the node corresponding to the news text N′ in the current knowledge graph, W(N,N′) is the mixed similarity between the current news text N and the news text N′, and n l The number of news texts with a mixed similarity of non-zero with news text N′;

[0055] By combining the weight calculation formulas of all nodes in the current knowledge graph, the weight of each node in the current knowledge graph can be calculated.

[0056] Furthermore, the method for calculating the modularity in step S5 is as follows:

[0057] Calculate the modularity between any two nodes in the co-occurrence graph using the following formula:

[0058]

[0059] in, For node w in the current node combination f With node w g The weights of connecting edges in a co-occurrence graph. For nodes connected to node w in the co-occurrence graph f All edges The sum of weights, l is the number of nodes connected to w in the co-occurrence graph. f The number of all edges, For nodes connected to node w in the co-occurrence graph g All edges The sum of weights, where n is the number of nodes connected to w in the co-occurrence graph. g The number of all edges, For node w f The category to which it belongs, For node w g The category to which node w belongs, where if node w f With node w g If they belong to the same category, then the flag function Otherwise, the flag function The sum of the weights of the edges connecting all combinations of nodes in the co-occurrence graph;

[0060] Summing the modularity of all node combinations in the co-occurrence graph, we obtain the modularity Q of all nodes in the co-occurrence graph under the current category set as follows:

[0061]

[0062] Furthermore, the evaluation method for causality in step S9 is as follows:

[0063] Construct a bivariate VAR model between the time-series index sequence of total news information intensity and the return rate sequence corresponding to all specific time T;

[0064] Obtain the maximum F-statistic Sup-F, the average F-statistic Mean-F, and the exponential F-statistic Exp-F for the bivariate VAR model;

[0065] Based on Sup-F, Mean-F, and Exp-F, determine whether the bivariate VAR model corresponding to the current specific time T has a causal relationship. Among them, when Sup-F is less than a set value or when Mean-F and Exp-F are both less than the set value, there is a causal relationship between the time series index sequence of news information intensity and the return series in the time series under test.

[0066] Furthermore, the method for evaluating coherence in step S10 is as follows:

[0067] Obtain the cross-wavelet coherence, multiple wavelet coherence, and partial wavelet coherence between the total news information intensity time-series index sequence and the return sequence for all specific time T.

[0068] The correlation between the news information intensity time-series index sequence and the return sequence at a specific time T is determined based on cross-wavelet coherence, multiple wavelet coherence, and partial wavelet coherence. Specifically, if any one of the cross-wavelet coherence, multiple wavelet coherence, and partial wavelet coherence is greater than a set value, then the news information intensity time-series index sequence and the return sequence in the time series under test are coherent.

[0069] Beneficial effects:

[0070] 1. This invention provides a news text analysis method based on graph analysis technology. By collecting massive amounts of news text data, it calculates multi-level hybrid similarity using various natural language processing methods, quantifies the weights of nodes and edges using graph analysis methods, constructs a time-varying news knowledge graph, and uses text classification methods and information decay assessment to construct a time-series index. Finally, it forms a categorized and time-series text index. Furthermore, it uses sliding window causality tests, wavelet analysis, and other methods to quantify the influence and attributes of the events represented by the text index on relevant indices in the domain. This invention solves the technical problem of accurately identifying, extracting, and processing the influencing factors of the market in the analyzed domain based on massive amounts of network information, thus assisting researchers and investors in establishing qualitative understanding and making quantitative predictions for specific domains. This deepens the understanding of the market influence system in the analyzed domain and better realizes the application of big data in the construction of the market influence system in the analyzed domain.

[0071] 2. This invention provides a news text analysis method based on graph analysis technology. Compared with traditional text index generation methods, this invention uses knowledge graphs as a carrier and applies natural language processing tools to achieve deep extraction of text information. It also combines text classification and information decay analysis methods to generate categorized text indices while fully considering their timeliness, relevance, and evolutionary relationships. This can help researchers use news information to generate quantitative understanding of the market impact of the field under analysis and further implement applications. Attached Figure Description

[0072] Figure 1 A flowchart of the energy field text index generation and analysis method based on graph analysis technology provided by the present invention;

[0073] Figure 2(a) is a sub-graph with the theme "Demand";

[0074] Figure 2(b) is a sub-graph on the theme of "Emergency Events";

[0075] Figure 2(c) is a sub-graph on the theme of "International Trade";

[0076] Figure 2(d) is a sub-graph with the theme "Energy";

[0077] Figure 2(e) is a subplot with the theme "Supply";

[0078] Figure 2(f) is a sub-graph with the theme "Technology";

[0079] Figure 3(a) shows the news index for the theme "demand";

[0080] Figure 3(b) shows the news index for the theme "Breaking News";

[0081] Figure 3(c) shows the news index for the theme "International Trade";

[0082] Figure 3(d) shows the news index for the theme "Energy";

[0083] Figure 3(e) shows the news index for the theme "supply";

[0084] Figure 3(f) shows the news index for the theme "Technology";

[0085] Figure 4 This invention provides a distributed modeling framework for a text index generation and analysis system in the energy field. Detailed Implementation

[0086] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0087] This invention provides a news text analysis method based on graph analysis technology, which mainly includes the following steps: preprocessing news text data, calculating multi-level hybrid similarity based on multiple natural language processing methods, quantifying the weights of nodes and edges using the PageRank graph analysis method, classifying using the KeyGraph+ method, introducing an exponential decay process to construct a time-series index, using the time-series text index as exogenous information, analyzing its time-series causal relationship with the energy market and its time-frequency domain coherence attributes, and encapsulating it into an energy text index construction and analysis system with data acquisition, processing, analysis and display based on a distributed system modeling framework.

[0088] Based on this, such as Figure 1 As shown, a news text analysis method based on graph analysis technology includes the following steps:

[0089] S1: Assemble the news texts in the domain to be analyzed into a total set {N1, N2, ..., N} according to their publication time. n}, where N i ={h i ,t i}, i = 1, 2, ..., n, h i ,t i The news text N is respectively i Title and body text;

[0090] It should be noted that when acquiring news texts, news text data of the domain to be analyzed can be collected from various information systems. Natural language processing methods can then be used to preprocess the collected data and extract its features, as detailed below:

[0091] S1.1): Use web crawling methods to obtain news texts in the field to be analyzed, such as energy-related fields.

[0092] S1.2): Perform part-of-speech restoration, capitalization conversion, stemming, and spell checking through format normalization.

[0093] S1.3): Remove noise information that is not textual information by using pattern matching methods and regular expressions, and segment long characters in the text into smaller segments such as words and phrases.

[0094] S1.4): Based on the stop word lexicon, functional words, numbers, punctuation marks, and special characters that do not contain actual meaning in the English text are deleted, resulting in a preprocessed set of news texts {N1, N2, ..., N...}. n}, where N i ={h i ,t i}, h i ,t i The news text N is respectively i The title and body text.

[0095] S2: Using a time window of a set length, extract multiple subsets from the total set of news texts according to a set step size. The number of news texts contained in each subset is not exactly the same.

[0096] For example, suppose the earliest publication date of each news text in the overall news text collection is January 1, 2024, and the latest publication date is March 31, 2024. The news texts are arranged in the overall collection in ascending order of publication date. Also suppose the time window is 30 days with a step size of 2 days. Then, the first time window will extract the news texts published between January 1, 2024 and January 30, 2024 as the first subset. The first time window will extract the news texts published between January 3, 2024 and February 1, 2024 as the second subset, and so on, resulting in multiple subsets. At the same time, since the publication dates of the collected news texts are not necessarily uniformly distributed, the number of news texts contained in each subset will not be exactly the same.

[0097] S3: After obtaining the mixed similarity between any two news texts in each subset, construct the knowledge graph corresponding to each subset based on the mixed similarity. Each news text is a node in the knowledge graph, and there are edges between nodes with a mixed similarity of not 0. The weight of the edge is the mixed similarity value of the two nodes that make up the edge, and the weights of each node are not exactly the same.

[0098] It should be noted that each subset corresponds to a time window. If any two news texts in the subset are denoted as N... u and N vThen the news text N u And news text N v Mixed similarity W(N) between u N v The method to obtain ) is as follows:

[0099]

[0100] Wherein, L(t) u ,t v (N) is the news text. u The main text t u And news text N v The main text t v Keyword weighted similarity between them, D(h) u ,h v (N) is the news text. u Title h u And news text N v Title h v Field similarity between them, M(t) u ,t v (N) is the news text. u The main text t u And news text N v The main text t v The mixed similarity is the co-occurrence similarity of named entities between the two texts. Therefore, the mixed similarity is the average of keyword weighted similarity, field similarity, and named entity co-occurrence similarity. If the mixed similarity between two news texts is 0, it means that the weight of the edge between the two news texts is 0, which means that there is no edge between the two news texts.

[0101] Furthermore, the field similarity D(h) u ,h v The calculation formula for ) is as follows:

[0102]

[0103] Among them, edit(h u ,h v |A,B) is the title h u With title h v Edit distance between them, A is the title h u The number of words in the title h, B is the title h. v The number of words in the edit(h) u ,h v A, B) are obtained recursively from the following formula:

[0104]

[0105]

[0106] Among them, edit(h u ,h v |a,b) is the title h u The a-th word in the title h v The edit distance between the b-th words in the text, where a∈[0,A], b∈[0,B], and edit[a-1][b] represents the title h. u The (a-1)th word in the title h v The edit distance between the b-th words in the text, edit[a][b-1] is the title h. u The a-th word in the title h v The edit distance between the (b-1)th words in the text, where edit[a-1][b-1] represents the title h. u The (a-1)th word in the title h v Edit distance between the (b-1)th words in the text, flag is the flag value, h u [a] is the title h u The a-th word in the text, h v [b] is the title h v The edit distances of the b-th word in the code are set to the initial values ​​of edit[0][0], edit[1][0], and edit[0][1].

[0107] It should be noted that edit(h) u ,h v In |a,b), a and b decrease from A and B to 0 respectively, while edit[0][0], edit[1][0], and edit[0][1] are set initial values. Therefore, edit(h) can be recursively obtained. u ,h v The values ​​of |A,B) clearly indicate that the field similarity D(h) u ,h v () is the reciprocal of the edit distance.

[0108] Furthermore, the named entity co-occurrence similarity M(t) u ,t v The calculation method for ) is as follows:

[0109]

[0110] Where, sim(t) u ,t v ) represents the main text t u With the main text t v The number of identical named entities between them, count(t) u ) represents the main text t uThe total number of named entities in the table, count(t) v ) represents the main text t v The total number of named entities in the text;

[0111] It should be noted that named entities in the main text can be extracted based on open-source tag sequences such as the Stanford NER Named Entity Repository, as detailed below:

[0112]

[0113] Where t is the news text to be named entity tagged, Y is the calculated tag type, P(Y|t) is the conditional probability of a specific tag sequence Y given the observation sequence t, δ is the transition feature function defined at two adjacent tag positions in the observation sequence, θ is the state feature function defined at the tag position in the observation sequence, λ,μ are parameters in the StanfordNER named entity library, and Z is the normalization factor.

[0114] Furthermore, keyword weighted similarity L(t) u ,t v The calculation method for ) is as follows:

[0115]

[0116] Among them, V p For text t to appear simultaneously u With the main text t v The keyword with the largest weight p, where p = 1, 2, ..., P, and P is at least 10. For the keyword V p In the main text t u The weights in For the keyword V p In the main text t v Weights in;

[0117] In other words, when calculating keyword weighting similarity, this invention statistically analyzes the main text t. u With the main text t v The text contains identical keywords, and the top 10 keywords with the highest weight are extracted. These 10 keywords are then recorded in the main text. u With the main text t v The smaller of the weights is used, and the weights of the top 10 identical keywords selected are summed to form the main text t. u With the main text t v Keyword weighted similarity.

[0118] It should be noted that the keyword V p As the current keyword Vr Then the current keyword V r The weight WS(V) of the main body text of the current news article r The calculation method for ) is as follows:

[0119]

[0120] Among them, In(V r () is to find the keyword V in the current text. r V is a set of adjacent words that have an adjacency relationship. q For the set of adjacent words In(V) r Keywords in ) In(V q To find the keyword V in the current text. q V is a set of adjacent words that have an adjacency relationship. k For the set of adjacent words In(V) q The keyword in ) is W qr For the keyword V q With the current keyword V r The weight of the connecting edge, W kq For the keyword V k With keyword V q The weights of the connecting edges (usually defined by co-occurrence frequency and semantic dependency), d is the damping coefficient, taking values ​​of (0,1), WS(V q () is the keyword V q The weight of the main body text of the current news article.

[0121] Meanwhile, the method for setting the weights of nodes in each knowledge graph in step S3 is as follows:

[0122] The weight calculation formula for each node in the current knowledge graph is defined as follows:

[0123]

[0124] Where PR(N) is the weight of the node corresponding to the current news text N in the current knowledge graph, S(X) is the set of news texts with a non-zero mixed similarity to the news text N corresponding to the current node, N′ is the news text in set S(X), PR(N′) is the weight of the node corresponding to the news text N′ in the current knowledge graph, W(N,N′) is the mixed similarity between the current news text N and the news text N′, and n l The number of news texts with a mixed similarity of non-zero with news text N′, which is also the degree of the node corresponding to news text N′;

[0125] By combining the weight calculation formulas of all nodes in the current knowledge graph, the weight of each node in the current knowledge graph can be calculated.

[0126] S4: After merging the named entities and keywords of all news texts in the total set, a topic word hybrid set is obtained, and a co-occurrence graph is constructed based on the topic word hybrid set. The named entities and keywords in the topic word hybrid set are used as nodes in the co-occurrence graph, and there is an edge between the nodes corresponding to two named entities or keywords that co-occur in the news text. The weight of the edge is the number of times the keywords or named entities corresponding to the two nodes that make up the edge co-occur in the total set of news texts.

[0127] For example, for all news texts, the Stanford NER named entity library in step S3 is used to extract named entities for each news text. Simultaneously, based on whether the weight of the keywords in the news text in step S3 is greater than a set value, it is determined whether the candidate topic words in the news text are keywords. The named entities and keywords of all news texts are then merged, and the union is taken to obtain a mixed set of topic words {w} obtained by combining keywords and named entities. r}, r=1,2,…,R, where R is the set {w r The number of keywords in the text.

[0128] Then, set {w r All keywords in} are paired up, and the keywords in each pair are denoted as w. f ,w g Determine w f ,w g Does it appear repeatedly in any news text within the overall collection of news texts? If it does, then w f ,w g There is an edge between them, and the weight of the edge is w. f ,w g The total number of times a keyword appears repeatedly in all news texts is used to obtain the keyword co-occurrence map KG = {w,e}.

[0129] For example, Figures 2(a) to 2(f) A schematic diagram of categorized keyword maps constructed using different corpora.

[0130] S5: Randomly cluster all nodes in the co-occurrence graph to obtain a set of categories C consisting of multiple categories with different numbers of nodes. Calculate the modularity of category set C, and adjust the number of categories and the number of nodes in each category according to the modularity until the optimal set of categories C with the minimum modularity is obtained. * and the category set C * The named entities and keywords corresponding to the nodes in each category form the topic word vector;

[0131] For example, the category set C *Each category is denoted as {c1, c2, ..., c h ,…,c H}, where H is the set of categories C * The total number of categories in the data, then category c h The named entities and keywords corresponding to the included nodes form the topic word vector, which can be denoted as: That is, the current category set C * Category C h Includes the named entities and keywords corresponding to the nodes.

[0132] Furthermore, the modularity Q of all nodes in the co-occurrence graph under the current category set is calculated as follows:

[0133] Calculate the modularity between any two nodes in the co-occurrence graph using the following formula:

[0134]

[0135] in, For node w in the current node combination f With node w g The weights of connecting edges in a co-occurrence graph. For nodes connected to node w in the co-occurrence graph f All edges The sum of weights, l is the number of nodes connected to w in the co-occurrence graph. f The number of all edges, For nodes connected to node w in the co-occurrence graph g All edges The sum of weights, where n is the number of nodes connected to w in the co-occurrence graph. g The number of all edges, For node w f The category to which it belongs, For node w g The category to which node w belongs, where if node w f With node w g If they belong to the same category, then the flag function Otherwise, the flag function The sum of the weights of the edges connecting all combinations of nodes in the co-occurrence graph;

[0136] Summing the modularity of all node combinations in the co-occurrence graph, we obtain the modularity Q of all nodes in the co-occurrence graph under the current category set as follows:

[0137]

[0138] S6: Obtain the cosine similarity between each news text and each topic word vector in the overall news text set;

[0139] Furthermore, the method for obtaining the cosine similarity between each news text and each topic word vector is as follows:

[0140]

[0141] in, For the i-th news text and the c-th... h The topic word vectors corresponding to each category The normalized proportion of cosine similarity between the categories, h = 1, 2, ..., H, where H is the set of categories C. * The total number of categories, For the i-th news text and the c-th... h The topic word vectors corresponding to each category Cosine similarity between them Let be the sum of the cosine similarities between the i-th news text and all topic word vectors.

[0142] S7: Take the latest publication time of the news text contained in each subset as each specific time T. For each specific time T, obtain the time-series index sequence of news information intensity corresponding to that specific time T.

[0143] The method for obtaining the time-series index sequence of news information intensity at each specific time T includes the following steps:

[0144] S71: Extract the knowledge graph corresponding to the subset at the current specific time T;

[0145] S72: Obtain the cosine similarity between each news text and each topic word vector contained in the current knowledge graph, and the weight of the node corresponding to each news text contained in the current knowledge graph;

[0146] In this process, the cosine similarity between each news text and each topic word vector in the current knowledge graph has been calculated in step S6, and the weight of the node corresponding to each news text in the current knowledge graph has been calculated in step S3.

[0147] S73: For each current news text contained in the current knowledge graph, obtain the categorized weight of the current news text in each category based on the cosine similarity between the current news text and each topic word vector, and the corresponding node weight in the current knowledge graph.

[0148] Furthermore, the method for obtaining the categorical weights of the current news text under each category is as follows:

[0149]

[0150] in, For the i-th news text at the c-th... h Categorical weights under each category For the i-th news text and the c-th... h The topic word vectors corresponding to each category The cosine similarity between the categories, h = 1, 2, ..., H, where H is the set of categories C. * The total number of categories, PR(t) is the sum of the cosine similarities between the i-th news text and all topic word vectors. i ) represents the weight of the i-th news text in the knowledge graph of the subset corresponding to the current specific time T.

[0151] S74: Based on the categorization weights of all news texts contained in the subset corresponding to the current specific time T under each category, obtain the time-series index sequence of news information intensity corresponding to the current specific time T.

[0152] Furthermore, the method for obtaining the time-series index sequence of news information intensity corresponding to a specific time T is as follows:

[0153] The subset corresponding to the current specific time T at the cth moment is obtained using the following formula. h News Information Intensity Time-Series Index under Each Category

[0154]

[0155] in, For the c-th time, all news texts in the subset corresponding to the current specific time T are... h The sum of the categorical weights under each category, s = 1, 2, ..., T-1, where s is a specific time before the current specific time T. For all news texts in the subset corresponding to a specific time s at time c h The sum of the categorical weights under each category, where λ is the set exponential weight; thus, it can be seen that... For category c h The information influence value at time T. Let be the historical information decay influence value when the time value is (0, T-1), and when s>T.

[0156] The time-series index of news information intensity of the subset corresponding to the current specific time T under each category Composition of news information intensity time-series index sequence H is the set of categories C * The total number of categories.

[0157] like Figures 3(a) to 3(f) For based on Figures 2(a) to 2(f) A schematic diagram of categorized text indices generated from categorized keyword maps constructed using different corpora.

[0158] S8: Record the time series covered by all the news information intensity time series obtained in step S7 as the time series to be tested, and obtain the return series of the field to be analyzed within the time series to be tested.

[0159] For example, assuming that the earliest publication time of each news text in the total collection of news texts is January 1, 2024, and the latest publication time is March 31, 2024, then the yield series of the domain to be analyzed from January 1, 2024 to March 31, 2024 can be obtained.

[0160] S9: Evaluate the causal relationship between the time-series index series of news information intensity and the return series according to the set indicators, specifically:

[0161] Construct a bivariate VAR model between the time-series index sequence of total news information intensity and the return rate sequence corresponding to all specific time T;

[0162] Obtain the maximum F-statistic Sup-F, the average F-statistic Mean-F, and the exponential F-statistic Exp-F for the bivariate VAR model;

[0163] Based on Sup-F, Mean-F, and Exp-F, determine whether the bivariate VAR model corresponding to the current specific time T has a causal relationship. Among them, when Sup-F is less than a set value or when Mean-F and Exp-F are both less than the set value, there is a causal relationship between the time series index sequence of news information intensity and the return series in the time series under test.

[0164] Based on this, the specific procedures for causal relationship assessment are as follows:

[0165] S91: Construct a bivariate VAR model for the text index and return based on the following formula:

[0166]

[0167] Where R(t) is the return series within the time series to be tested, and I(t) is the calculated time-series index of total news information intensity. For the VAR model coefficients, i,j∈{1,2}, k is defined as the larger of i and j plus 1, representing the lag operator.

[0168] S92: Imposing constraints on the model established in S91 and To examine the causal relationship between returns and time-series indices, and to evaluate the effectiveness of the generated time-series indices;

[0169] S93: Calculate the parameter stability of the VAR model according to the following formula to determine the time-varying nature of the dependence between the text index and the return rate:

[0170]

[0171]

[0172]

[0173] in represents the F-statistic of the VAR model, ESS represents the regression sum of squares of the VAR model, RSS represents the residual sum of squares of the VAR model, and n is the number of variables in the VAR model. That is, n=2. When the Sup-F value is greater than a certain value, it indicates that the model established by S91 has a structural mutation and the model is unstable; when the Mean-F and Exp-F values ​​are greater than a certain value, it indicates that the parameters of the model established by S91 are time-varying and cannot be evaluated using a single model, so there is a need to establish a time-varying model.

[0174] S94: Determine an appropriate sampling sliding window size, and use the RB-based modified LR statistic to calculate the dynamic causal relationship between the text serialization index and the yield dependence, based on the strength and polarity of the causal relationship.

[0175] It should be noted that, assuming there are 10 specific moments T, the total news information intensity time-series index sequence is composed of 10 news information intensity time-series index sequences. If the three F statistics (Sup-F, Mean-F, Exp-F) calculated by the VAR model constructed from the total news information intensity time-series index sequence are all large, and the dynamic causal relationship between the sequence index and the return rate cannot be analyzed, a small segment of the 10 news information intensity time-series index sequences is extracted using a sliding window. These 10 segments form a new total news information intensity time-series index sequence. The bivariate VAR model is then reconstructed based on the newly generated total news information intensity time-series index sequence, and the three F statistics are recalculated to evaluate the causal relationship.

[0176] S10: Evaluate the coherence between the time-series index series of news information intensity and the return series according to the set indicators, specifically:

[0177] Obtain the cross-wavelet coherence, multiple wavelet coherence, and partial wavelet coherence between the total news information intensity time-series index sequence and the return sequence for all specific time T.

[0178] The correlation between the news information intensity time-series index sequence and the return sequence at a specific time T is determined based on cross-wavelet coherence, multiple wavelet coherence, and partial wavelet coherence. Specifically, if any one of the cross-wavelet coherence, multiple wavelet coherence, and partial wavelet coherence is greater than a set value, then the news information intensity time-series index sequence and the return sequence in the time series under test are coherent.

[0179] Specifically, the cross-wavelet coherence R is used to evaluate the impact of the time-series index on returns in different time and frequency domains. 2 The formula for calculating (R,I) is as follows:

[0180]

[0181] Where S is the smoothing factor, W R,I (λ,μ) are the model parameters for cross-wavelet coherence weighting, representing the cross-wavelet power spectrum of the return and text index, W. R (λ,μ) and W I (λ,μ) represent the continuous wavelet transform forms of the return and text index, respectively. Under specific time periods and cycles, the cross-wavelet coherence is relatively high, indicating that the return and text index sequences measured under those conditions have a strong mutual influence.

[0182] Multiple wavelet coherence used to assess the combined impact of textual indices and influencing factors on returns. The calculation formula is as follows:

[0183]

[0184] If the coherence of multiple wavelets is relatively high under specific time periods and cycles, it indicates that the combined influence of the time-series index and the influencing factor sequence on the return series is strong under those conditions.

[0185] Partial wavelet coherence used to evaluate the independent impact of the text index on returns after removing influencing factors. The calculation formula is as follows:

[0186]

[0187] Under specific time periods and cycles, the coherence of the smaller wavelets is relatively high, indicating that under the conditions of that time period and cycle, the time-series index has a strong influence on the return series after excluding the series of influencing factors.

[0188] Based on the above-described news text analysis method using graph analysis technology, this invention can construct a distributed system based on this news text index construction and analysis method, including functions such as data acquisition, processing, analysis, and display. Figure 4 As shown, the details are as follows:

[0189] A distributed database layer is constructed to store the data required for the operation of the distributed system. Specifically, this includes acquiring information from the internet and building a database containing news texts in the energy sector. Simultaneously, by linking energy databases and open information sites, an energy market-related index database is built, storing information related to the energy market and its influencing factors. Furthermore, a news knowledge graph database containing event tags is established and stored.

[0190] Based on the knowledge graph construction method, the text index generation method, the sliding window causality test method, and the wavelet coherence analysis method, a basic framework and content for the distributed system technology layer are constructed, and the above algorithms are encapsulated to provide support for the main operating modules of the system.

[0191] Based on a web development framework, a control layer for a distributed system is constructed. Specifically, this includes establishing links between the distributed system and external information sources on the Internet to ensure timely information acquisition; and establishing a front-end and back-end interaction framework to call different algorithms based on user-input model parameters and feed the results back to the front-end interface.

[0192] Based on the front-end development framework, the application layer of the distributed system is built, which specifically includes designing interfaces for user login and management, parameter setting and timestamp adjustment, knowledge graph display, and display of causal relationships and wavelet analysis results.

[0193] In summary, compared with traditional text index generation methods, this invention uses knowledge graphs as a carrier and applies natural language processing tools to achieve deep extraction of text information. It also combines text classification and information decay analysis methods to fully consider the timeliness, relevance, and evolutionary relationship of the generated categorized text indices.

[0194] Secondly, existing technologies only focus on verifying the generated indices, with little discussion on their application methods and expected effects. This invention, by introducing a sliding window causal relationship discrimination method and a multi-factor wavelet coherence analysis method, not only determines the existence of the causal relationship and influence of the generated indices on the energy market, but also explores their polarity, intensity, timeliness, and independence attributes. This can help researchers gain a quantitative understanding of the impact of news information on the energy market for further application.

[0195] Finally, current related inventions rarely provide guidance for the practical application of the proposed method from the perspective of system modeling. This invention discusses a distributed system framework that takes the proposed method as its core, includes a database, interactive algorithms and a front-end interface, and extends its application in a distributed system manner, thereby realizing the acquisition, processing, analysis and display of data, thus meeting this practical need.

[0196] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A news text analysis method based on graph analysis technology, characterized in that, Includes the following steps: S1: Assemble the news texts in the domain to be analyzed into a total set {N1, N2, ..., N} according to their publication time. n }, where N i ={h i ,t i }, i = 1, 2, ..., n, h i ,t i The news text N is respectively i Title and body text; S2: Using a time window of a set length, extract multiple subsets from the total set of news texts according to a set step size. The number of news texts contained in each subset is not exactly the same. S3: After obtaining the mixed similarity between any two news texts in each subset, construct the knowledge graph corresponding to each subset based on the mixed similarity. Each news text is a node in the knowledge graph, and there are edges between nodes with a mixed similarity of not 0. The weights of each node are not exactly the same. S4: After merging the named entities and keywords of all news texts in the total set, a topic word hybrid set is obtained, and a co-occurrence graph is constructed based on the topic word hybrid set. The named entities and keywords in the topic word hybrid set are nodes in the co-occurrence graph, and there are edges between the nodes corresponding to two named entities or keywords that co-occur in the news text. S5: Randomly cluster all nodes in the co-occurrence graph to obtain a set of categories C consisting of multiple categories with different numbers of nodes. Calculate the modularity of category set C, and adjust the number of categories and the number of nodes in each category according to the modularity until the optimal set of categories C with the minimum modularity is obtained. * and the category set C * The named entities and keywords corresponding to the nodes in each category form the topic word vector; S6: Obtain the cosine similarity between each news text and each topic word vector in the overall news text set; S7: Take the latest publication time of the news text contained in each subset as each specific time T. For each specific time T, obtain the time-series index sequence of news information intensity corresponding to that specific time T. S8: Record the time series covered by all the news information intensity time series obtained in step S7 as the time series to be tested, and obtain the return series of the field to be analyzed within the time series to be tested. S9: Evaluate the causal relationship between the time-series index sequence of news information intensity and the return rate sequence according to the set indicators; S10: Evaluate the coherence between the time-series index series of news information intensity and the return series according to the set indicators.

2. The news text analysis method based on graph analysis technology as described in claim 1, characterized in that, The method for obtaining the time-series index sequence of news information intensity corresponding to each specific time T in step S7 is as follows: S71: Extract the knowledge graph corresponding to the subset at the current specific time T; S72: Obtain the cosine similarity between each news text and each topic word vector contained in the current knowledge graph, and the weight of the node corresponding to each news text contained in the current knowledge graph; S73: For each current news text contained in the current knowledge graph, obtain the categorized weight of the current news text in each category based on the cosine similarity between the current news text and each topic word vector, and the corresponding node weight in the current knowledge graph. S74: Based on the categorization weights of all news texts contained in the subset corresponding to the current specific time T under each category, obtain the time-series index sequence of news information intensity corresponding to the current specific time T.

3. The news text analysis method based on graph analysis technology as described in claim 2, characterized in that, The method for obtaining the cosine similarity between each news text and each topic word vector in the overall news text set in step S6 is as follows: in, For the i-th news text and the c-th... h The topic word vectors corresponding to each category The normalized proportion of cosine similarity between the categories, h = 1, 2, ..., H, where H is the set of categories C. * The total number of categories, For the i-th news text and the c-th... h The topic word vectors corresponding to each category Cosine similarity between them Let be the sum of the cosine similarities between the i-th news text and all topic word vectors.

4. The news text analysis method based on graph analysis technology as described in claim 2, characterized in that, The method for obtaining the categorical weights of the current news text under each category in step S73 is as follows: in, For the i-th news text at the c-th... h Categorical weights under each category For the i-th news text and the c-th... h The topic word vectors corresponding to each category The cosine similarity between the categories, h = 1, 2, ..., H, where H is the set of categories C. * The total number of categories, PR(t) is the sum of the cosine similarities between the i-th news text and all topic word vectors. i ) represents the weight of the i-th news text in the knowledge graph of the subset corresponding to the current specific time T.

5. The news text analysis method based on graph analysis technology as described in claim 2, characterized in that, The method for obtaining the time-series index sequence of news information intensity corresponding to the current specific time T in step S74 is as follows: The subset corresponding to the current specific time T at the cth moment is obtained using the following formula. h News Information Intensity Time-Series Index under Each Category in, For the c-th time, all news texts in the subset corresponding to the current specific time T are... h The sum of the categorical weights under each category, s = 1, 2, ..., T-1, where s is a specific time before the current specific time T. For all news texts in the subset corresponding to a specific time s at time c h The sum of the categorical weights under each category, where λ is the set exponential weight; The time-series index of news information intensity of the subset corresponding to the current specific time T under each category This forms a time-series index sequence of news information intensity.

6. The news text analysis method based on graph analysis technology as described in claim 1, characterized in that, Let N be any two news texts in the subset. u and N v Then the news text N u And news text N v Mixed similarity W(N) between u N v The method to obtain ) is as follows: Wherein, L(t) u ,t v (N) is the news text. u The main text t u And news text N v The main text t v Keyword weighted similarity between them, D(h) u ,h v (N) is the news text. u Title h u And news text N v Title h v Field similarity between them, M(t) u ,t v (N) is the news text. u The main text t u And news text N v The main text t v Named entity co-occurrence similarity between them; Field similarity D(h) u ,h v The calculation formula for ) is as follows: Among them, edit(h u ,h v |A,B) is the title h u With title h v Edit distance between them, A is the title h u The number of words in the title h, B is the title h. v The number of words in edit(h) u ,h v A, B) are obtained recursively from the following formula: Among them, edit(h u ,h v |a,b) is the title h u The a-th word in the title h v The edit distance between the b-th words in the text, where a∈[0,A], b∈[0,B], and edit[a-1][b] represents the title h. u The (a-1)th word in the title h v The edit distance between the b-th words in the text, edit[a][b-1] is the title h. u The a-th word in the title h v The edit distance between the (b-1)th words in the text, where edit[a-1][b-1] represents the title h. u The (a-1)th word in the title h v Edit distance between the (b-1)th words in the text, flag is the flag value, h u [a] is the title h u The a-th word in the text, h v [b] is the title h v The edit distances of the b-th word in the code are set to the initial values ​​of edit[0][0], edit[1][0], and edit[0][1]. Named entity co-occurrence similarity M(t) u ,t v The calculation method for ) is as follows: Where, sim(t) u ,t v ) represents the main text t u With the main text t v The number of identical named entities between them, count(t) u ) represents the main text t u The total number of named entities in the table, count(t) v ) represents the main text t v The total number of named entities in the text; Keyword weighted similarity L(t) u ,t v The calculation method for ) is as follows: Among them, V p For text t to appear simultaneously u With the main text t v The keyword with the largest weight p, where p = 1, 2, ..., P, and P is at least 10. For the keyword V p In the main text t u The weights in For the keyword V p In the main text t v Weights in; Current keyword V r The weight WS(V) of the main body text of the current news article r The calculation method for ) is as follows: Among them, In(V r () is to find the keyword V in the current text. r V is a set of adjacent words that have an adjacency relationship. q For the set of adjacent words In(V) r Keywords in ) In(V q To find the keyword V in the current text. q V is a set of adjacent words that have an adjacency relationship. k For the set of adjacent words In(V) q The keyword in ) is W qr For the keyword V q With the current keyword V r The weight of the connecting edge, W kq For the keyword V k With keyword V q The weight of the connecting edge, d is the damping coefficient, WS(V q () is the keyword V q The weight of the main body text of the current news article.

7. The news text analysis method based on graph analysis technology as described in claim 1, characterized in that, The method for setting the weights of nodes in each knowledge graph in step S3 is as follows: The weight calculation formula for each node in the current knowledge graph is defined as follows: Where PR(N) is the weight of the node corresponding to the current news text N in the current knowledge graph, S(X) is the set of news texts with a non-zero mixed similarity to the news text N corresponding to the current node, N′ is the news text in set S(X), PR(N′) is the weight of the node corresponding to the news text N′ in the current knowledge graph, W(N,N′) is the mixed similarity between the current news text N and the news text N′, and n l The number of news texts with a mixed similarity of non-zero with news text N′; By combining the weight calculation formulas of all nodes in the current knowledge graph, the weight of each node in the current knowledge graph can be calculated.

8. The news text analysis method based on graph analysis technology as described in claim 1, characterized in that, The method for calculating modularity in step S5 is as follows: Calculate the modularity between any two nodes in the co-occurrence graph using the following formula: in, For node w in the current node combination f With node w g The weights of connecting edges in a co-occurrence graph. For nodes connected to node w in the co-occurrence graph f All edges The sum of weights, l is the number of nodes connected to w in the co-occurrence graph. f The number of all edges, For nodes connected to node w in the co-occurrence graph g All edges The sum of weights, where n is the number of nodes connected to w in the co-occurrence graph. g The number of all edges, For node w f The category to which it belongs, For node w g The category to which node w belongs, where if node w f With node w g If they belong to the same category, then the flag function Otherwise, the flag function The sum of the weights of the edges connecting all combinations of nodes in the co-occurrence graph; Summing the modularity of all node combinations in the co-occurrence graph, we obtain the modularity Q of all nodes in the co-occurrence graph under the current category set as follows:

9. The news text analysis method based on graph analysis technology as described in claim 1, characterized in that, The evaluation method for causality in step S9 is as follows: Construct a bivariate VAR model between the time-series index sequence of total news information intensity and the return rate sequence corresponding to all specific time T; Obtain the maximum F-statistic Sup-F, the average F-statistic Mean-F, and the exponential F-statistic Exp-F for the bivariate VAR model; Based on Sup-F, Mean-F, and Exp-F, determine whether the bivariate VAR model corresponding to the current specific time T has a causal relationship. Among them, when Sup-F is less than a set value or when Mean-F and Exp-F are both less than the set value, there is a causal relationship between the time series index sequence of news information intensity and the return series in the time series under test.

10. The news text analysis method based on graph analysis technology as described in claim 1, characterized in that, The method for evaluating coherence in step S10 is as follows: Obtain the cross-wavelet coherence, multiple wavelet coherence, and partial wavelet coherence between the total news information intensity time-series index sequence and the return sequence for all specific time T. The correlation between the news information intensity time-series index sequence and the return sequence at a specific time T is determined based on cross-wavelet coherence, multiple wavelet coherence, and partial wavelet coherence. Specifically, if any one of the cross-wavelet coherence, multiple wavelet coherence, and partial wavelet coherence is greater than a set value, then the news information intensity time-series index sequence and the return sequence in the time series under test are coherent.

Images