A classical poetry binary poetic language automatic extraction and semantic network construction method and system based on two-stage NPMI screening

Abstract

This invention relates to a method and system for automatic extraction of binary poetic phrases and construction of semantic networks in classical poetry based on a two-stage NPMI screening approach, belonging to the field of computational analysis technology for classical poetry. This method addresses the problems of high noise in poetic phrase extraction, neglect of inter-poetic phrase relationships, and lack of overall quality control in semantic network construction found in existing technologies. The technical solution quantifies the cohesion of poetic meaning through a first-stage NPMI screening to extract high-quality core poetic phrases. The second-stage NPMI screening quantifies the strength of poetic phrase pairings to identify significant poetic pairings, thereby constructing a weighted undirected semantic network and performing topological analysis and community detection. The technical advantages include significantly improved accuracy in poetic phrase extraction, clear and reliable poetic pairing relationships, powerful overall analysis capabilities, high automation, and reproducible results.

Description

Technical Field

[0001] This invention belongs to the field of computational analysis technology of classical poetry, and relates to a method and system for automatic extraction of binary poetic language and construction of semantic network in classical poetry based on two-stage NPMI screening. Background Technology

[0002] In computational analysis of classical poetry, existing technologies mainly suffer from the following three problems.

[0003] First, the quality of automatic extraction of binary poetic language is relatively low. Classical poetry is highly condensed, and binary character combinations often serve as the smallest units of poetic meaning, such as "bright moon" and "spring breeze." Existing methods mostly rely on modern Chinese word segmentation tools or simple... n Metagram frequency statistics. Modern Chinese word segmentation tools are not suitable for classical Chinese grammatical structures, but simpler ones... n Metagrammatical statistics generate a large number of high-frequency but poetically worthless invalid combinations, such as "one person" and "there is". These methods lack a quantitative assessment of the poetic cohesion, resulting in noisy and inaccurate extraction results.

[0004] Secondly, the research granularity is limited, neglecting the connections between poetic phrases. Most existing studies focus only on the frequency or distribution of individual binary words, failing to systematically explore and quantify the stable connections between different binary poetic phrases. For example, studies might separately analyze "gentle breeze" and "bright moon," but lack a macro-level measurement of the intensity of classic pairings like "gentle breeze" and "bright moon." This isolated research perspective cannot reveal the inherent recurring patterns of poetic pairings in poetic language.

[0005] Third, semantic network construction lacks a holistic perspective and quality control. The few studies attempting to construct semantic networks are mostly limited to single poems or individual authors, making it difficult to build a semantic network reflecting the entire poetic genre's imagery system at a macro level. More importantly, when constructing the network, the original co-occurrence frequency is usually used directly as the edge weights, lacking statistical significance control over node quality (the poetic language itself) and edge quality (the correlations between poetic phrases). This results in high network noise and a vague structure, making the credibility and interpretability of advanced analytical results such as community detection poor.

[0006] In summary, existing technologies cannot simultaneously solve the problems of high noise in poetic language extraction, insufficient quantification of correlation relationships, and low quality of network construction. There is an urgent need for a technical solution that can automatically filter high-quality binary poetic language, systematically quantify poetic pairings, and construct a pure semantic network. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a method and system for automatic extraction of binary poetic language and construction of semantic networks in classical poetry based on a two-stage NPMI screening. This invention, through systematic innovation, solves the key problems of high noise in poetic language extraction, insufficient quantification of correlation relationships, and a lack of a holistic perspective in network construction.

[0008] The primary objective of this invention is to provide an automatic extraction and high-quality filtering method for binary poetic phrases. This method quantifies the poetic cohesion through a first-stage normalized point mutual information calculation. This first-stage normalized point mutual information calculation filters the set of results from the byte-to-byte encoding algorithm, accurately identifying core poetic phrase units with high semantic cohesion from massive character combinations. This step fundamentally filters out high-frequency but low-cohesion invalid combinations, significantly improving the quality of the underlying data.

[0009] Another objective of this invention is to provide a quantitative measurement and screening method for semantic associations between binary poetic phrases. Through a second-stage normalized point mutual information calculation, based on intra-poetic co-occurrence statistics, poetic pairings with significant statistical associations are evaluated and screened. This design expands the analytical perspective from isolated poetic phrases to the network of associations between poetic phrases, revealing stable imagery pairing patterns in classical poetry.

[0010] The third objective of this invention is to provide a method for constructing and analyzing macroscopic semantic networks based on high-quality poetic nodes and significant poetic pairings. This method applies complex network analysis techniques to achieve network topology analysis and automatic discovery of semantic communities, thereby revealing the holistic organizational structure of the classical poetry imagery system.

[0011] To achieve the above objectives, this invention adopts the following technical solution. The entire process comprises three core steps, sequentially realizing the entire process from poetic extraction, pairing and linking to network construction. The first step is core poetic extraction, involving data preprocessing, high-frequency candidate set generation, and first-stage normalized point mutual information screening. The second step is poetic pairing measurement, including document-level co-occurrence statistics and second-stage normalized point mutual information screening. The third step is semantic network construction and discovery, covering network construction, topology analysis, and community discovery.

[0012] A method for automatic extraction of binary poetic language and construction of semantic networks in classical poetry based on two-stage normalized pointwise mutual information (NPMI) filtering includes the following steps: The data preprocessing step involves cleaning, standardizing, and structuring the original poetry text to output a clean collection of verses. The core poetic phrase extraction step generates a high-frequency binary poetic phrase candidate set from the pure poetic phrase set, and quantifies the poetic coherence based on the first-stage NPMI screening to obtain a high-quality core poetic phrase set. The poetic pairing measurement step involves performing document-level co-occurrence statistics based on the high-quality core poetic phrase set, and quantifying the semantic association strength between poetic phrases based on the second-stage NPMI screening to obtain a significant poetic pairing edge set. The semantic network construction and discovery steps involve constructing a weighted undirected semantic network using the high-quality core poetic phrases as the node set and the significant poetic pairing edge set as the edge set, and then performing topology analysis and community discovery.

[0013] Furthermore, the data preprocessing step includes: The text cleaning sub-step removes non-poetic content from the original text, retaining only the main text of the poem. The character standardization sub-step unifies variant characters and traditional characters into simplified characters; The poem segmentation sub-step divides the poem into independent lines using punctuation marks as boundaries.

[0014] Furthermore, the core poetic extraction step includes: The binary poem candidate set generation sub-step applies the Byte Pair Encoding (BPE) algorithm to traverse the pure poem line set, counts the frequency of all adjacent character pairs, and selects the highest frequency candidate lines. K A set of binary combinations is used as a high-frequency candidate set; The core binary poetic phrase generation sub-step involves generating each candidate binary poetic phrase from the high-frequency candidate set. phrase =( c 1, c 2) Calculate the first-stage NPMI value to quantify the poetic cohesion, and retain the poetic phrases with NPMI values ​​greater than the threshold Θ1 to form the high-quality core poetic phrase set.

[0015] Furthermore, the NPMI value in the first stage is obtained through the formula... Calculation, where phrase It is a binary poetic language, composed of Chinese characters. c 1 and c Composed of 2 components, P ( c 1) For characters c 1. The probability of a single character appearing in the corpus. P ( c 1, c 2) is a binary combination ( c 1, c 2) Joint probability of occurrence in the corpus.

[0016] Furthermore, the poetic pairing measurement step includes: The co-occurrence statistics sub-step iterates through each poem in the corpus and counts the poetic language. phrase 1 and phrase 2. The number of poems that appear together in the same poem; The probability calculation sub-step is based on the total number of poems. M Calculate Poetic Language phrase Document probability of 1 P _ doc ( phrase 1) and poetic parallelism ( phrase 1, phrase 2) Co-occurrence probability P _ co ( phrase 1, phrase 2); The second stage, the NPMI screening sub-step, calculates the value of any two poetic phrases. NPMI _ association The value is used to quantify the pairing association strength, and poetic pairs with NPMI values ​​greater than the threshold Θ2 are retained to form the significant poetic pairing edge set.

[0017] Furthermore, the NPMI value in the second stage is obtained through the formula... Calculation, where For poetic language The document probability, For poetic language The co-occurrence probability.

[0018] Furthermore, the semantic network construction and discovery steps include: The network construction sub-step uses the aforementioned high-quality core poetry collection as the node set. V The significant poetic pairing edge set is an edge set. E The NPMI_association value is a weight. W Constructing a weighted undirected semantic network G =( V , E , W ); The topology analysis sub-step calculates the basic topology attributes of the network, including modularity Q, average clustering coefficient C, and average path length L; the community detection sub-step applies the Louvain algorithm to divide the network into communities and optimize the modularity Q.

[0019] Furthermore, the community discovery sub-step also includes a theme interpretation sub-step, which analyzes the high-frequency poetic phrases and core nodes within each community and assigns theme tags based on the poetic background.

[0020] Furthermore, the thresholds Θ1 and Θ2 are determined based on predefined evaluation metrics; Θ1 is selected by optimizing the curves of precision and recall as a function of the threshold; Θ2 through modularity Q Average clustering coefficient C and average path length L The comprehensive evaluation selection specifically includes: conducting tests within the range of Θ2=0.15 to Θ2=0.55 with a preset step size, and recording the network size and module degree corresponding to each Θ2 value. Q Average clustering coefficient C and average path length L ; Choose to make modular Q The optimal threshold is defined as the Θ² value that achieves a value in the range of 0.1 to 0.3 while maintaining the small-world property of the network and without losing nodes; where the modularity... Q Through formula calculate, For nodes i and j Edge weights between them For nodes i The weighting degree, m The total weight of all edges in the network. For nodes i Community number to which it belongs The Kronecker function is used; the average clustering coefficient C is obtained through the formula... calculate, For nodes i The local clustering coefficient, n Total number of nodes; average path length L Through formula calculate, For nodes i arrive j The shortest path length.

[0021] A system for automatic extraction of binary poetic language and construction of semantic networks in classical poetry based on two-stage NPMI screening includes: The data preprocessing module is used to clean, standardize, and structure the original poetry text, outputting a clean collection of verses; The core poetic phrase extraction module is used to generate a high-frequency binary poetic phrase candidate set from the pure poetic phrase set, and to quantify the poetic coherence based on the first-stage NPMI screening to obtain a high-quality core poetic phrase set. The Poetic Pairing Measurement Module is used to perform document-level co-occurrence statistics based on the high-quality core poetic phrase set, and to quantify the semantic association strength between poetic phrases based on the second-stage NPMI screening to obtain significant poetic pairing edge sets. A semantic network construction and discovery module, which is used to construct a weighted undirected semantic network with the high-quality core poetic phrase set as the node set and the significant poetic pairing edge set as the edge set, and perform topological analysis and community discovery.

[0022] The beneficial effects of the present invention are as follows: (1) Through the quantification and screening of "poetic solidity degree" by the first-stage normalized point mutual information, the present invention can accurately identify core poetic phrase units with literary value from high-frequency character combinations. This method effectively filters out invalid combinations with high frequency but low cohesion, such as "one person" and "there is", and raises the accuracy of poetic phrase extraction to a new height. Compared with the traditional methods that rely on modern Chinese word segmentation tools or simple n meta-grammar statistics, the present invention significantly improves the quality of basic data and lays a reliable foundation for subsequent analysis.

[0023] (2) Through the measurement of the association strength between poetic phrases by the second-stage normalized point mutual information, the present invention realizes a paradigm shift from isolated poetic phrase analysis to associated network mining. The poetic pairings screened by this method represent stable association patterns (such as the classic combination of "gentle breeze - bright moon") throughout the poem library, rather than accidental co-occurrences. The constructed network edge set is pure and has clear meanings, completely changing the limitation of the prior art that ignores the association relationship between poetic phrases, and providing a new perspective for exploring the internal rules of the classical poetry image system.

[0024] (3) Based on the macroscopic semantic network constructed by high-quality nodes and edges, the present invention demonstrates excellent overall analysis capabilities. The network has a clear community structure (the modularity reaches 0.183 in the embodiment), and can automatically separate semantic communities such as "spring scenery and boudoir feelings" and "traveling and homesickness". This modular structure intuitively shows the overall layout and theme division of the classical poetry image system, solves the fragmentation problem of the prior research limited to single poems or individual authors, and provides an unprecedented global quantitative perspective for literary research.

[0025] (4) The parameters of the entire process are clear, without manual annotation or intervention, and can automatically and efficiently process massive corpora such as "Complete Tang Poems". The two-stage screening thresholds are determined through scientific evaluation, and the analysis method has high reproducibility, meeting the rigorous requirements of computational humanities research. This standardized process not only improves the research efficiency, but also ensures that different scholars can obtain consistent and reliable results when applying this method, promoting the standardized development of computational analysis of classical poetry.

[0026] Other advantages, objectives and features of the present invention will be described in some aspects in the subsequent specification, and in some aspects, will be obvious to those skilled in the art based on the study of the following text, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following specification. Attached Figure Description

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 The overall flowchart for the two-stage NPMI screening process; Figure 2 System module block diagram; Figure 3 Example diagram of the network semantic effect of core words in Tang poetry; Figure 4 The data results are for an example using the Complete Tang Poems. Figure 5 A heatmap showing the strength of connections between communities; Figure 6 It is a self-centered network based on the image of "willow". Detailed Implementation

[0028] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0029] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0030] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0031] I. Methods and Systems This invention proposes a "two-stage NPMI screening" framework, whose overall process includes three core steps, sequentially realizing the entire process from poetic language extraction, pairing and connection to network construction. For example... Figure 1 As shown.

[0032] Step 1: Extraction of core poetic phrases (including the first stage NPMI screening) 1. Data preprocessing: Clean the corpus of "Complete Tang Poems" and other sources, remove non-poetic content, and retain the main text of the poems.

[0033] 2. High-frequency candidate set generation: Apply the Byte-Pair Encoding (BPE) algorithm to the entire corpus, count the global frequency of all binary character combinations, and select the highest frequency candidate set. K One as a candidate set C BPE .

[0034] 3. First-stage NPMI screening (quantitative poetic consolidation): targeting C BPE Each candidate binary poetic phrase The normalized point mutual information value is calculated to measure the tightness of the combination of the two characters into a stable poetic unit.

[0035]

[0036] Here, "phrase" is a binary poetic term, consisting of two individual Chinese characters. and Composition, such as "bright moon" which is composed of "bright" and "moon"; Chinese characters The probability of a single character appearing in the entire corpus, specifically, =(character) (Total number of occurrences in the corpus) / (Total number of characters in the corpus); It is a binary combination The joint probability of occurrence of the whole in the corpus, specifically, =(binary combination) (Total frequency of occurrences in the corpus) / (Total number of combinations of BPE-generated binary combinations in the corpus). Set a threshold. ,reserve The poetic language constitutes a high-quality collection of core poetic language. This step fundamentally filters out ineffective combinations that are high-frequency but low-cohesion.

[0037] Step Two: Poetic Pairing Measurement (corresponding to the second stage NPMI screening) 1. Document-level co-occurrence statistics: Traverse every poem in the corpus. For each poem, traverse every binary word combination from beginning to end, and identify all occurrences belonging to the set... Poetic language. Statistical analysis of poetic language. The total number of poems that appear at least once. Statistical Poetry and The total number of poems that appear together in the same poem .

[0038] 2. Probability Calculation: Let the total number of poems in the corpus be... M Poetic Language Document probability: Poetic language Co-occurrence probability:

[0039] 3. Second-stage NPMI screening (quantitative pairing association strength): for any two poems... and Calculate its semantic association strength:

[0040]

[0041] 4. Significant Pair Filtering: Set a threshold (e.g., 0.30), only retain The poetic phrases form a significant poetic pairing edge set. E .

[0042] Step 3: Semantic Network Construction and Discovery 1. Network Construction: A collection of high-quality poetic language For node set V With a strikingly poetic pairing of edge sets E The edge set is weighted by the NPMI_association value. W Constructing a weighted undirected semantic network G =( V , E , W ).

[0043] 2. Topology Analysis and Community Detection: Calculate the basic topological properties of the network (such as density, average path length, and clustering coefficient). Apply the Louvain community detection algorithm to the network. G Divide and optimize modularity Q It automatically identifies semantic communities within the network.

[0044] 3. Theme Interpretation: For each discovered community, analyze its high-frequency characters and core poetic phrases with high internal connectivity, and combine them with the poetic background to assign interpretable theme tags to the community (such as "landscape and pastoral", "traveling to the frontier").

[0045] like Figure 2 The diagram shown is a system module block diagram.

[0046] 1. Data Preprocessing Module Function: Cleans, standardizes, and structures the original poetry text to provide clean and standardized input data for subsequent analysis.

[0047] Input: Raw text, such as "Complete Tang Poems" (including punctuation, author information, and other non-textual content).

[0048] Processing flow: (1) Text cleaning: Remove non-poetic content such as HTML tags, numbers, English letters, volume numbers, and author information, and retain only the main text of the poem.

[0049] (2) Character standardization: unify variant characters and traditional characters to simplified characters for mainland China to ensure consistency of character shapes.

[0050] (3) Segmentation of poem lines: The poem is divided into independent lines by using “.”, “?”, “!”, etc., while retaining the original line structure.

[0051] Output: A clean, structured collection of verses for use in subsequent modules.

[0052] 2. Core Poetic Language Extraction Module Submodule 1: Binary Poetic Candidate Set Generation Submodule Function: Extract high-frequency binary character combinations from preprocessed text to form a preliminary candidate set.

[0053] Input: A collection of cleaned poems.

[0054] Processing flow: (1) Use the byte pair encoding (BPE) algorithm to traverse the entire corpus and count the frequency of occurrence of all adjacent character pairs.

[0055] (2) Sort by frequency in descending order and select the top N The first 1000 highest frequency binary combinations are used as the initial candidate set to reduce low-frequency noise interference.

[0056] Output: A candidate set of high-frequency binary poetic phrases (e.g., 1000), which serves as the basis for poetic selection.

[0057] Submodule 2: Core Binary Poetic Expression Generation Submodule Function: Selects core binary poetic phrases with true poetic value from high-frequency candidate sets.

[0058] Input: A high-frequency binary poetic phrase candidate set (e.g., 1000).

[0059] Processing flow: (1) For each candidate binary combination (e.g., “bright moon”), calculate its first NPMI value to measure the “poetic cohesion” between the two characters.

[0060] (2) Set threshold (e.g.) θ =1=0.25), retain combinations with NPMI greater than the threshold, and remove high-frequency combinations that have no poetic value (such as "one person" and "there is").

[0061] Output: A high-quality set of core poetic phrases (e.g., 668), which will serve as nodes for subsequent network construction.

[0062] 3. Poetic Pairing Measurement Module Function: Quantify the semantic association strength between different binary poetic phrases and filter out statistically significant "poetic pairings".

[0063] Input: A collection of core poetic phrases and cleaned poetry texts.

[0064] Processing flow: (1) Co-occurrence statistics: Traverse each poem. For each poem, traverse each binary word combination from beginning to end and detect the co-occurrence of poetic phrases in the same poem.

[0065] (2) Probability calculation: Based on co-occurrence statistics, calculate the co-occurrence probability and independent probability of the poetic pairs.

[0066] (3) Second NPMI calculation: used to quantify the semantic association strength between two poetic phrases: (4) Threshold screening: retain NPMI> θ 2 (e.g., 0.30) of poetic couplets, filtering out accidental co-occurrence.

[0067] Output: A set of saliently poetic paired edges, which serve as edges in the semantic network.

[0068] 4. Semantic Network Construction Module Function: Construct a weighted undirected semantic network by using core poetic phrases as nodes and poetic pairings as edges.

[0069] Input: Core set of poetic phrases (nodes), set of poetic matching edges (edges and their weights).

[0070] Processing flow: (1) Using core poetic phrases as nodes V Using significant pairings as edge sets ENPMI correlation value as weight W , construct graph G =( V , E , W ).

[0071] (2) Networks can be used for topology analysis, such as calculating density, average path length, clustering coefficient, etc., to verify their small-world properties.

[0072] Output: A structured semantic network graph that can be used for visualization and analysis.

[0073] 5. Community Discovery and Thematic Interpretation Module Function: Divide semantic networks into communities and automatically generate and interpret topic tags based on community content.

[0074] Input: Semantic network graph.

[0075] Processing flow: (1) Community detection: The Louvain algorithm is used to divide the network into communities and optimize the module degree Q.

[0076] (2) Feature extraction: For each community, count the high-frequency poems and core nodes (high internal connectivity).

[0077] (3) Theme generation: Combining high-frequency words, core poetic phrases and poetics knowledge, the community is given interpretable theme tags (such as "spring scenery and boudoir feelings" and "homesickness while traveling").

[0078] Output: The segmented semantic communities and their topic tags, used for holistic analysis of the imagery system, such as... Figure 3 As shown.

[0079] Using the Complete Tang Poems as an example, the original BPE results (the first 1000) without NPMI filtering were directly fed into the second stage NPMI calculation for pairing. The results are shown in Tables 1 to 4.

[0080] After algorithmic calculation, four communities were generated. Table 1 shows the top 10 poems and their semantic network attributes in Community 1; Table 2 shows the top 10 poems and their semantic network attributes in Community 2; Table 3 shows the top 10 poems and their semantic network attributes in Community 3; and Table 4 shows the top 10 poems and their semantic network attributes in Community 4. Bound phrases containing 2-gram poetic terms with low semantic relevance to the community, as well as non-poetic phrases, are displayed in bold to indicate their "inconsistency" within the community.

[0081] Table 1

[0082] Table 2

[0083] Table 3

[0084] Table 4

[0085] As can be seen from Tables 1 to 4, directly inputting the raw BPE results without NPMI filtering into the second-stage NPMI calculation and pairing results caused "noise" in the community.

[0086] The data results of the embodiment of the automatic extraction of binary poetic language and semantic network construction process of classical poetry using the two-stage NPMI screening method designed in this invention, taking the Complete Tang Poems as an example, are as follows: Figure 4 As shown in the figure. The results of the four communities generated are shown in Table 5.

[0087] Table 5

[0088] Key technical points of this invention: 1. Dual NPMI Quality Control System: This system creatively applies Normalized Point Mutual Information (NPMI) twice to the same analytical process: the first stage of NPMI screening ensures node (poetic) quality (poetic cohesion), while the second stage of NPMI screening ensures edge (pairing association) quality (association strength). This forms the statistical foundation for constructing high-quality, highly interpretable semantic networks.

[0089] 2. Paradigm innovation from isolated units to interconnected networks: This invention systematically proposes a macro-level measurement, screening, and network construction method for the pairing relationships of binary poetic phrases in classical poetry, realizing a paradigm shift from frequency analysis of isolated poetic phrases to the mining of stable association patterns between poetic phrases.

[0090] 3. Structured Macro-Poetic Discovery Process: This invention defines three clear and coherent core steps: "extraction, measurement, and discovery," forming a complete automated analysis pipeline. Through two-stage statistical filtering, this process can robustly extract the macro-modular structure of the classical poetic imagery system from the original text.

[0091] II. Implementation Examples This invention is implemented using the Complete Tang Poems as the object of analysis.

[0092] 1. Step 1 (Core Poetic Expression Extraction): The top 1000 pairs of frequency were selected by BPE, and then 668 high-quality core poetic expressions were obtained through the first-stage NPMI screening with θ1=0.25.

[0093] 2. Step Two (Poetic Pairing Measurement): Statistically analyze the co-occurrence of poetic phrases within all poems. Through the second-stage NPMI screening with θ2=0.30, extract 24,077 significant poetic pairing edges from a massive number of possible combinations.

[0094] 3. Step Three (Network Construction and Discovery): Construct a semantic network (668 nodes, 24077 edges). This network exhibits small-world characteristics (average clustering coefficient 0.145, average path length 2.83) and a clear modular structure (modularity 0.183). The Louvain algorithm automatically identified four main communities, which, after analysis, were identified as the themes of "Springtime Romance," "Homesickness While Traveling," "Life's Encounters," and "Seclusion in Nature."

[0095] 4. Validation of Results: This embodiment validates that the method of the present invention achieves high-quality poetic language extraction, significant poetic pairing discovery, and macro-network construction through three sequential steps. The automatically discovered thematic communities highly align with traditional poetic classifications, demonstrating the effectiveness and practicality of the two-stage NPMI screening framework in revealing the macro-imagery organizational structure of classical poetry.

[0096] III. Basis for Determining Algorithm Parameters and Thresholds (1) Regarding θ1, the first 1000 results of the initial BPE generation were manually labeled to construct a gold standard set. Then, the evaluation index was used to evaluate it.

[0097] a. The Gold Standard Based on 1000 candidate words from the BPE results, three university Chinese language teachers were invited to annotate them. The final labels were determined by majority voting. 670 positive examples and 330 negative examples were labeled "poetic language," forming the gold standard set. G The consistency rate among the three annotators was 0.82 (Cohen's Kappa), indicating that the annotation quality is reliable.

[0098] b. Evaluation Indicators The following quantitative metrics were used to evaluate the quality of the sorted lists returned by each method at multiple cutoff points (K): Precision@K: Precision@K = (#Relevant results in Top K) / K Recall@K: Recall@K = (#relevant results in Top K) / |G + |, where |G + | is the total number of positive examples in the gold standard set (670).

[0099] c. Result, such as Figure 4 As shown.

[0100] (2) Regarding Θ2, the three indicators of modularity (Q), average clustering coefficient (C), and average path length (L) are used for comprehensive evaluation, as shown in Table 6.

[0101] Table 6

[0102] a. The formula for calculating modularity (Q) is:

[0103] in, Represents a node i and j The edge weight between them (0 if there is no edge); Represents a node i The weighted degree (i.e., the sum of the weights of all edges); This represents the total weight of all edges in the network; Represents a node i The community number to which it belongs; Represent the Kronecker function, when It is 1 if it is true, otherwise it is 0.

[0104] b. Average Clustering Coefficient C Calculation formula:

[0105]

[0106] in, Represents a node i The local clustering coefficient; Represents a node i and j Edge weights between them; n This represents the total number of nodes.

[0107] c. Formula for calculating the average shortest path length (L):

[0108] in, Represents a node i arrive j The shortest path length (number of edges); in weighted networks, weights are usually converted into distances (e.g., ...). d =1 / w or d = log (w )).

[0109] Experimental data shows: Test Θ2=0.15...√ Nodes: 668, Edges: 119505, Q: 0.069, C: 0.170, L: 4.700 Test Θ2=0.16...√ Nodes: 668, Edges: 115758, Q: 0.071, C: 0.166, L: 4.700 Test Θ2=0.17...√ Nodes: 668, Edges: 111290, Q: 0.072, C: 0.161, L: 4.700 Test Θ2=0.18...√ Nodes: 668, Edges: 106004, Q: 0.077, C: 0.156, L: 4.701 Test Θ2=0.19...√ Nodes: 668, Edges: 99908, Q: 0.081, C: 0.149, L: 4.704 Test Θ2=0.20...√ Nodes: 668, Edges: 92863, Q: 0.085, C: 0.142, L: 4.713 Test Θ2=0.21...√ Nodes: 668, Edges: 84897, Q: 0.086, C: 0.134, L: 4.730 Test Θ2=0.22...√ Nodes: 668, Edges: 76607, Q: 0.094, C: 0.125, L: 4.755 Test Θ2=0.23...√ Nodes: 668, Edges: 68307, Q: 0.105, C: 0.117, L: 4.787 Test Θ2=0.24...√ Nodes: 668, Edges: 60406, Q: 0.114, C: 0.108, L: 4.823 Test Θ2=0.25...√ Nodes: 668, Edges: 52494, Q: 0.123, C: 0.099, L: 4.866 Test Θ2=0.26...√ Nodes: 668, Edges: 45078, Q: 0.132, C: 0.090, L: 4.911 Test Θ2=0.27...√ Nodes: 668, Edges: 38762, Q: 0.144, C: 0.082, L: 4.953 Test Θ2=0.28...√ Nodes: 668, Edges: 33208, Q: 0.156, C: 0.074, L: 4.996 Test Θ2=0.29...√ Nodes: 668, Edges: 28475, Q: 0.167, C: 0.067, L: 5.037 Test Θ2=0.30...√ Nodes: 668, Edges: 24077, Q: 0.183, C: 0.061, L: 5.085 Test Θ2=0.31...√ Nodes: 668, Edges: 20114, Q: 0.195, C: 0.055, L: 5.148 Test Θ2=0.32...√ Nodes: 668, Edges: 16431, Q: 0.204, C: 0.050, L: 5.255 Test Θ2=0.33...√ Nodes: 668, Edges: 13194, Q: 0.215, C: 0.046, L: 5.423 Test Θ2=0.34...√ Nodes: 668, Edges: 10368, Q: 0.246, C: 0.041, L: 5.672 Test Θ2=0.35...√ Nodes: 668, Edges: 8087, Q: 0.265, C: 0.038, L: 5.969 Test Θ2=0.36...√ Nodes: 668, Edges: 6242, Q: 0.290, C: 0.035, L: 6.295 Test Θ2=0.37...√ Nodes: 668, Edges: 4844, Q: 0.310, C: 0.033, L: 6.662 Test Θ2=0.38...√ Nodes: 668, Edges: 3785, Q: 0.347, C: 0.032, L: 7.128 Test Θ2=0.39...√ Nodes: 663, Edges: 2843, Q: 0.399, C: 0.028, L: 7.757 Test Θ2=0.40...√ Nodes: 654, Edges: 2125, Q: 0.455, C: 0.027, L: 8.569 Test Θ2=0.41...√ Nodes: 627, Edges: 1547, Q: 0.529, C: 0.030, L: 9.663 Test Θ2=0.42...√ Nodes: 588, Edges: 1184, Q: 0.600, C: 0.031, L: 10.830 Test Θ2=0.43...√ Nodes: 531, Edges: 879, Q: 0.671, C: 0.029, L: 12.269 Test Θ2=0.44...√ Nodes: 440, Edges: 637, Q: 0.730, C: 0.031, L: 13.184 Test Θ2=0.45...√ Nodes: 348, Edges: 452, Q: 0.773, C: 0.031, L: 14.188 Test Θ2=0.46...√ Nodes: 268, Edges: 333, Q: 0.801, C: 0.034, L: 16.437 Test Θ2=0.47...√ Nodes: 163, Edges: 184, Q: 0.809, C: 0.031, L: 16.305 Test Θ2=0.48...√ Nodes: 101, Edges: 106, Q: 0.795, C: 0.010, L: 16.390 Test Θ2=0.49...√ Nodes: 38, Edges: 38, Q: 0.683, C: 0.000, L: 11.264 Test Θ2=0.50...√ Nodes: 26, Edges: 25, Q: 0.618, C: 0.000, L: 9.384 Test Θ2=0.51...√ Nodes: 13, Edges: 12, Q: 0.463, C: 0.000, L: 6.755 Test Θ2=0.52...√ Nodes: 8, Edges: 7, Q: 0.313, C: 0.000, L: 3.869 Test Θ2=0.53...√ Nodes: 6, Edges: 8, Q: -0.000, C: 0.536, L: 2.447 Test Θ2=0.54...√ Nodes: 6, Edges: 8, Q: -0.000, C: 0.536, L: 2.447 Test Θ2=0.55...√ Nodes: 5, Edges: 7, Q: 0.000, C: 0.677, L: 2.087 The choice of Θ=0.3 is based on its optimal balance in parameter testing: it retains all 668 poetic nodes without losing any analytical objects, while keeping the network size manageable at 24,000 edges. At this threshold, the modularity Q reaches 0.183, and the community structure begins to clearly emerge (according to empirical criteria, Q=0.1~0.3 indicates a relatively obvious community structure), while the network still maintains the small-world characteristics of a true semantic network (average path length is only 5.085). This threshold filters out 82% of weak noise associations, conforms to the Pareto principle in text mining, offers excellent computational efficiency and visualization, and is consistent with the commonly used range (0.2~0.4) in similar studies, making it an adaptive choice for the concise language characteristics of Tang poetry.

[0110] IV. Boundaries of Technical Solutions and Alternative Means The technical solution described in this invention has clear boundaries and reasonable alternatives, all of which fall within the scope of implementation that can be naturally conceived or substituted by those skilled in the art after reading this invention: 1. Substitutability of word segmentation and sub-word unit generation methods: This invention preferably employs Byte-Pair Encoding (BPE) as a method for extracting high-frequency binary combinations. However, those skilled in the art can also achieve the same purpose using WordPiece, statistical n-gram extraction, or dictionary-based matching methods. These methods all fall under the technical category of "identifying high-frequency substrings from continuous character sequences" and do not affect the core innovation of this invention—namely, the two-stage NPMI screening framework.

[0111] 2. The substitutability of community detection algorithms: In this embodiment of the invention, the Louvain algorithm is used for community detection based on its efficiency and modularity optimization capabilities. However, community detection or graph clustering methods such as Infomap, Label Propagation Algorithm (LPA), spectral clustering, and GN algorithm are also applicable to the network structure of this invention and can all achieve automatic semantic community division.

[0112] All of the above alternative methods are common knowledge in the field and do not affect the novelty and inventiveness of the core framework of the "dual NPMI screening" of this invention.

[0113] V. Specific details regarding the pretreatment steps This invention performs systematic cleaning and standardization operations tailored to the characteristics of classical poetry texts during the data preprocessing stage, ensuring the accuracy and reproducibility of subsequent algorithmic processing. 1. Text cleaning: Remove all numbers, HTML characters, English characters, poem titles, volume numbers, author information, etc. Note: punctuation marks within the poem must be preserved. That is, retain only the poem text with punctuation marks to ensure the analysis object is a pure sequence of poetic lines. Specifically, filter poem titles, volume numbers, and author information by recognizing special punctuation marks; for example, those containing "【" represent volume numbers, and those containing ",.", or "!" represent the poem text, and so on. For non-Chinese characters such as numbers, HTML characters, and English characters, use regular expressions for recognition and filtering. If a poem contains such characters, it is considered dirty data and needs to be filtered out.

[0114] 2. Character standardization: Using the OpenCC tool + custom rules method, unify variant Chinese characters and traditional Chinese characters, such as converting "羣" to "群" and "陞" to "升", and convert all characters to simplified Chinese characters in Mainland China. The custom dictionary is placed in the attachment "Dictionary.txt for Unifying Variant and Traditional Chinese Characters".

[0115] The above preprocessing steps can be naturally implemented by those skilled in the art according to the characteristics of classical texts and have been clearly described in the embodiments of the present invention.

[0116] Use double NPMI to process Tang poems and then construct a network. After community division, the heat map of the association strength between communities is as Figure 5 shown, where C0, C1, C2, and C3 correspond to four communities.

[0117] 1. Defined the "theme territory" and "core channels" of Tang poems: Four communities (C0 - C3): Through network analysis, the vast vocabulary and images involved in the entire "Complete Tang Poems" are clearly converged into four core theme modules. This can be regarded as the "four major emotion - image clusters" of Tang poems in cultural psychology.

[0118] The connections between them: These association strengths are the "cultural highways" connecting these core clusters. n =9194 This path (spring scenery and boudoir feelings life and official career) is the busiest and most core main road. This means that when a Tang poet wants to express "official career", the most natural and classic path in his mind is to express it indirectly through the image of "spring scenery".

[0119] 2. Revealed the collective unconsciousness and shared rhetorical library of Tang poets: All association strengths are so high (0.336 - 0.353) and the differences are not significant, proving that these four themes are highly interconnected in the universe of Tang poems.

[0120] Any one theme cannot exist in isolation. When wanting to write about "autumn journey and homesickness" (C1), people are very likely to naturally associate with "life reflections" (C2) because the connection channel between them ( n =5700) is very strong. This is the shared thinking template or rhetorical inertia of Tang poets.

[0121] 3. Quantified the core cultural contradiction of "official career" and "seclusion": The weakest connection between C2 and C3 (official career seclusion) is precisely the most interesting cultural discovery. It accurately depicts the eternal tug - of - war in the hearts of Chinese scholars with data.

[0122] The low value (0.336) indicates that, at the level of direct textual expression, poets rarely juxtapose "seeking officialdom" and "retirement." However, the connection still exists. n =1205), indicating that this tension is always latent and may be expressed more through a more roundabout way (such as first writing about setbacks in official career C2, and then turning to the Zen-like feeling of mountains and rivers C3).

[0123] Detailed description of the network: The network contains 668 nodes (poetic phrases) and 24,077 weighted edges. The network density is 0.1081, and the average weighted degree is 72.09, indicating extensive and close semantic connections between the poetic phrases. Table 7 shows the basic topological indicators of the poetic phrase network.

[0124] Table 7

[0125] To verify the small-world property of the network, networks with the same number of nodes ( n =668) and connection probability Erd s – Ré nyi Based on a random graph model, the connection probability of a random graph is taken as the density of the actual network. ,in The number of edges is 24077. n The number of nodes is 668, therefore we get p =0.1081.

[0126] The expected clustering coefficient of a random graph is C_random= p =0.1081. For the average shortest path length, the classic approximation formula for the ER random graph is used. Calculation, where With an average degree of 72.09, L_random ≈ 1.52. This represents the clustering coefficient of a real-world network. C =0.145, average shortest path length L =2.83.

[0127] Comparison yields: relative clustering coefficient C / C _ random ≈1.34>1 relative path length L / L _ random ≈1.86 This network satisfies two key characteristics of small-world networks: its clustering coefficient is significantly higher than that of random networks, while its average path length is on the same order of magnitude as that of random networks. This indicates that the Tang poetry imagery system forms tight semantic clusters locally (high clustering coefficient) while maintaining efficient semantic accessibility globally (short average path).

[0128] For the word "willow," an association test was conducted, such as... Figure 6 As shown.

[0129] 1. There are 141 nodes connected to "Yangliu", with an average NPMI value of 0.348, and 4 connected communities.

[0130] 2. "Cross-community connection count" refers to the number of different communities Yangliu's neighbors come from, not the number of communities Yangliu herself belongs to. The information for the four neighboring communities is as follows: Community 1 has 97 nodes, Community 2 has 28 nodes, Community 3 has 9 nodes, and Community 4 has 7 nodes.

[0131] 3. The three words with the strongest association are: spring breeze, parting, and spring water. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for automatic extraction of binary poetic language and construction of semantic network in classical poetry based on two-stage NPMI screening, characterized in that: Includes the following steps: The data preprocessing step involves cleaning, standardizing, and structuring the original poetry text to output a clean collection of verses. The core poetic phrase extraction step generates a high-frequency binary poetic phrase candidate set from the pure poetic phrase set, and quantifies the poetic coherence based on the first-stage NPMI screening to obtain a high-quality core poetic phrase set. The poetic pairing measurement step involves performing document-level co-occurrence statistics based on the high-quality core poetic phrase set, and quantifying the semantic association strength between poetic phrases based on the second-stage NPMI screening to obtain a significant poetic pairing edge set. The semantic network construction and discovery steps involve constructing a weighted undirected semantic network using the high-quality core poetic phrases as the node set and the significant poetic pairing edge set as the edge set, and then performing topology analysis and community discovery.

2. The method for automatic extraction of binary poetic language and construction of semantic network in classical poetry based on two-stage NPMI screening as described in claim 1, characterized in that: The data preprocessing steps include: The text cleaning sub-step removes non-poetic content from the original text, retaining only the main text of the poem. The character standardization sub-step unifies variant characters and traditional characters into simplified characters; The poem segmentation sub-step divides the poem into independent lines using punctuation marks as boundaries.

3. The method for automatic extraction of binary poetic language and construction of semantic network in classical poetry based on two-stage NPMI screening as described in claim 1, characterized in that: The core poetic extraction steps include: The binary poetic candidate set generation sub-step involves applying the byte pair encoding (BPE) algorithm to traverse the pure poetic line set, counting the frequency of all adjacent character pairs, and selecting the highest frequency candidate lines. K A set of binary combinations is used as a high-frequency candidate set; The core binary poetic phrase generation sub-step involves generating each candidate binary poetic phrase from the high-frequency candidate set. phrase =( c 1, c 2) Calculate the first-stage NPMI value to quantify the poetic cohesion, and retain the poetic phrases with NPMI values ​​greater than the threshold Θ1 to form the high-quality core poetic phrase set.

4. The method for automatic extraction of binary poetic language and construction of semantic network in classical poetry based on two-stage NPMI screening as described in claim 3, characterized in that: The first stage NPMI value is obtained through the formula Calculation, where phrase It is a binary poetic language, composed of Chinese characters. c 1 and c Composed of 2 components, P ( c 1) For characters c 1. The probability of a single character appearing in the corpus. P ( c 1, c 2) is a binary combination ( c 1, c 2) Joint probability of occurrence in the corpus.

5. The method for automatic extraction of binary poetic language and construction of semantic network in classical poetry based on two-stage NPMI screening as described in claim 1, characterized in that: The poetic pairing measurement steps include: The co-occurrence statistics sub-step iterates through each poem in the corpus and counts the poetic language. phrase 1 and phrase 2. The number of poems that appear together in the same poem; The probability calculation sub-step is based on the total number of poems. M Calculate Poetic Language phrase 1. phrase Document probability of 2 P _ doc ( phrase 1) P _ doc ( phrase 2) and poetic couplets ( phrase 1, phrase 2) Co-occurrence probability P _ co ( phrase 1, phrase 2); The second stage, the NPMI screening sub-step, calculates the value of any two poetic phrases. NPMI _ association The value is used to quantify the pairing association strength, and poetic pairs with NPMI values ​​greater than the threshold Θ2 are retained to form the significant poetic pairing edge set.

6. The method for automatic extraction of binary poetic language and construction of semantic network in classical poetry based on two-stage NPMI screening as described in claim 5, characterized in that: The second-stage NPMI value is obtained through the formula Calculation, where For poetic language The document probability, For poetic language The co-occurrence probability.

7. The method for automatic extraction of binary poetic language and construction of semantic network in classical poetry based on two-stage NPMI screening as described in claim 1, characterized in that: The semantic network construction and discovery steps include: The network construction sub-step uses the aforementioned high-quality core poetry collection as the node set. V The significant poetic pairing edge set is an edge set. E The NPMI_association value is a weight. W Constructing a weighted undirected semantic network G =( V , E , W ); The topology analysis sub-step calculates the basic topology attributes of the network, including modularity Q, average clustering coefficient C, and average path length L; the community detection sub-step applies the Louvain algorithm to divide the network into communities and optimize the modularity Q.

8. The method for automatic extraction of binary poetic language and construction of semantic network in classical poetry based on two-stage NPMI screening as described in claim 7, characterized in that: The community discovery sub-step also includes a theme interpretation sub-step, which analyzes the high-frequency poetic phrases and core nodes within each community and assigns theme tags based on the poetic background.

9. The method for automatic extraction of binary poetic language and construction of semantic network in classical poetry based on two-stage NPMI screening according to claim 3 or 5, characterized in that: The thresholds Θ1 and Θ2 are determined based on predefined evaluation metrics; Θ1 is selected by optimizing the curves of precision and recall as a function of the threshold; Θ2 through modularity Q Average clustering coefficient C and average path length L The comprehensive evaluation selection specifically includes: conducting tests within the range of Θ2=0.15 to Θ2=0.55 with a preset step size, and recording the network size and module degree corresponding to each Θ2 value. Q Average clustering coefficient C and average path length L ; Choose to make modular Q The optimal threshold is defined as the Θ² value that achieves a value between 0.1 and 0.3 while maintaining the small-world property of the network and without losing nodes; where the modularity Q is calculated using the formula... calculate, For nodes i and j Edge weights between them For nodes i The weighting degree, m The total weight of all edges in the network. For nodes i Community number to which it belongs The Kronecker function is used; the average clustering coefficient C is obtained through the formula... calculate, For nodes i The local clustering coefficient, n Total number of nodes; average path length L Through formula calculate, For nodes i arrive j The shortest path length.

10. A system for automatic extraction of binary poetic language and construction of semantic network in classical poetry based on two-stage NPMI screening, characterized in that: include: The data preprocessing module is used to clean, standardize, and structure the original poetry text, outputting a clean collection of verses; The core poetic phrase extraction module is used to generate a high-frequency binary poetic phrase candidate set from the pure poetic phrase set, and to quantify the poetic coherence based on the first-stage NPMI screening to obtain a high-quality core poetic phrase set. The poetic pairing measurement module is used to perform document-level co-occurrence statistics based on the high-quality core poetic phrase set, and to quantify the semantic association strength between poetic phrases based on the second-stage NPMI screening, so as to obtain significant poetic pairing edge sets and pairing strength sets. The semantic network construction and discovery module is used to construct a weighted undirected semantic network using the high-quality core poetic phrases as the node set, the significant poetic pairing edge set as the edge set, and the pairing strength set as the weight set, and to perform topology analysis and community discovery.

Images