Information density calculation method and application system thereof

Abstract

The invention belongs to the technical field of natural language processing, and discloses an information density calculation method and an application system thereof, and the system comprises a semantic unit extraction module, a multi-attribute filling module, an information density scoring module, a density distribution modeling module and a downstream application interface module. The semantic unit extraction module identifies subject-object structures, proper noun combinations and verb-object phrases from an original text through fusion word segmentation, syntactic dependency analysis and semantic role labeling, and outputs a structured semantic unit list in combination with unified term expression of a domain dictionary and a concept ontology. The invention provides a new method for modeling information density from a semantic structure level, which is different from a word frequency statistics or black box probability method, emphasizes structure interpretability, attribute combinability and application suitability, is especially suitable for scenes such as intelligent abstracts, document compression, semantic annotation, knowledge graph construction and the like, and has a wide application prospect. And the method has definite novelty, practicability and generalizability.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and more specifically, to an information density calculation method and an application system thereof. Background Art

[0002] With the development of natural language processing technology, information density, as an important indicator to measure the compactness of text expression, information carrying capacity and content saturation, has been increasingly used in tasks such as intelligent summarization, knowledge graph construction, educational content compression, and intelligent question and answer.

[0003] The current mainstream information density calculation methods mainly include:

[0004] 1. Techniques based on word frequency statistics (such as TF-IDF and keyword density analysis) only measure the frequency of word occurrence in a document and cannot identify semantic structure and inter-word relationships, lacking an understanding of semantic content.

[0005] 2. Technologies based on language model entropy (such as token perplexity and surprise) only measure the predictability of word order probability, but lack the ability to interpret structure, which is not conducive to understanding the composition and expression efficiency of information.

[0006] Therefore, existing technologies generally lack an information density calculation method that has the ability to abstract semantic structures, is highly interpretable, supports multi-attribute annotations, and is oriented towards machine processing. Summary of the Invention

[0007] The purpose of the present invention is to solve the shortcomings of the prior art and to propose an information density calculation method and an application system thereof.

[0008] In order to solve the above problems, the present invention adopts the following technical solutions:

[0009] A method for calculating information density includes the following steps:

[0010] 1) Semantic Unit Extraction: This process integrates word segmentation, syntactic dependency analysis, and semantic role labeling to identify subject-verb-object structures, proper noun combinations, and verb-object phrases from the original text. It then unifies terminology using domain dictionaries and concept ontologies, and outputs a structured list of semantic units.

[0011] 2) Multi-attribute annotation: Dynamically attach multi-dimensional semantic attributes to each semantic unit, and dynamically adapt attribute labels through predefined attribute templates or knowledge graph reasoning;

[0012] 3) Density score: Based on the rarity and attribute richness of semantic units, the total information content of all semantic units within a unit of text length is calculated;

[0013] 4) Distribution Modeling: Using a sliding window or paragraph segmentation strategy, we generate a text information density distribution map, identify high-density and low-density areas, and generate a density heat map and interval markers.

[0014] 5) Application interface: Apply density scores and distribution results to downstream applications and provide density data services.

[0015] As a further solution of the present invention: in the semantic unit extraction step, multilingual text processing and multi-layer nested structure analysis are supported, and the semantic integrity of candidate phrases is verified through a deep model.

[0016] As a further solution of the present invention: the semantic attributes include: entity type, semantic category, hyponym and hyponym structure, spatiotemporal annotation and subject area.

[0017] As a further solution of the present invention: the calculation formula of the density scoring step is as follows:

[0018]

[0019] U={u1,u2,...,u n} is a set of semantic units, w(u i )=α·IDF(u i )+β·|A(u i )| is the semantic unit weight, L is the unit text length;

[0020] Among them, IDF(u i ) is the inverse document frequency of the semantic unit, which measures its rarity in the corpus; |A(u i )| is the number of attributes annotated in the unit; α and β are adjustable weights used to balance the contribution of semantic scarcity and attribute information.

[0021] As a further solution of the present invention: the sliding window size and step size in the distribution modeling step support dynamic adjustment, and the high-density area and low-density area thresholds are configured according to the application scenario.

[0022] This solution also provides an information density application system based on semantic abstraction units and multi-attribute fusion, including:

[0023] Semantic unit extraction module, which is used to extract units with independent semantic meaning from the original text and output semantic units with structural labels;

[0024] Multi-attribute annotation module: used to dynamically add multi-dimensional semantic attributes to each semantic unit;

[0025] Information density scoring module: Generates text density scores based on the rarity and attribute richness of semantic units;

[0026] Density distribution modeling module: Models information density changes in text using sliding windows or paragraph structure strategies, generating density heat maps and interval markers;

[0027] Downstream application interface module: used to apply density scoring and distribution results to downstream applications and provide density data services.

[0028] As a further solution of the present invention: the downstream application interface module supports the priority extraction of high-density paragraphs or the rearrangement of document structure and the elimination of redundant content based on a heat map.

[0029] As a further solution of the present invention: the downstream application interface includes a visualization interface, which displays density distribution with an interactive heat map and supports dynamic adjustment of thresholds.

[0030] Compared with the prior art, the advantages of the present invention are:

[0031] 1. Strong structure: Based on semantic units rather than word frequency statistics, it has the ability to express abstract semantics;

[0032] 2. Strong interpretability: Outputs each semantic unit and its attribute structure, and the results are reproducible;

[0033] 3. Strong scalability: supports multi-language and multi-attribute templates, and is easy to integrate with knowledge graph systems;

[0034] 4. High application value: Suitable for information extraction, summary optimization, and visual analysis tasks in text-intensive fields such as education, medicine, and scientific research;

[0035] In summary, the present invention proposes a new method for modeling information density from the semantic structure level. Different from word frequency statistics or black box probability methods, it emphasizes structural interpretability, attribute combination and application adaptability. It is particularly suitable for scenarios such as intelligent summarization, document compression, semantic annotation, and knowledge graph construction, and has clear novelty, practicality and generalizability. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the process of the present invention;

[0037] Figure 2 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] See also Figure 1 、 2 , an information density calculation method and application system, comprising:

[0040] 1. Semantic Unit Extraction: The Semantic Unit Extraction module integrates word segmentation, syntactic dependency analysis, and semantic role labeling from the original text to identify subject-verb-object structures, proper noun combinations, and verb-object phrases. It then integrates domain dictionaries and concept ontologies to unify terminology and output a structured list of semantic units. It supports multilingual text processing and parsing of multi-layer nested structures, and uses deep models to verify the semantic integrity of candidate phrases.

[0041] Specifically, in this embodiment, units with independent semantic meanings are extracted from the original text, which are called "semantic units". The extraction strategy includes:

[0042] Integrate word segmentation, syntactic dependency, and semantic role labeling to identify subject-verb-object structures, proper noun combinations, verb-object phrases, etc.

[0043] Introducing domain dictionaries and concept ontologies to unify terminology and resolve issues such as synonyms and nested structures;

[0044] The output is: a set of structured semantic units (such as: ["Tsinghua University", "carry out", "artificial intelligence research"]).

[0045] Traditional methods usually align the segmented sentences with the reference language sentences, and extract the semantic units of the target sentences from the part-of-speech tagging results of the reference language. However, this solution pays more attention to semantic coherence in different contexts, and can achieve more refined segmentation and merging. The function of the semantic unit extraction module is to identify semantically complete phrases or conceptual units from the segmentation results. For example, "Peking University" is merged into an entity unit "Peking University", or multiple words involving professional terms are retained as a whole. Its workflow may include: combining the vocabulary / ontology, syntactic rules or deep models to analyze syntactic structure, identifying possible entity names, term phrases, subject-verb structures, etc. as candidate units; verifying the semantic integrity of candidate phrases (for example, through context consistency or knowledge base retrieval); and finally outputting a list of semantic units.

[0046] Compared to traditional word frequency-based segmentation, this algorithm overcomes problems such as polysemy and incorrect segmentation of proper nouns, and can generate semantically consistent, standardized segments. By extracting semantic units, it significantly reduces the number of elements that make up a sentence, making the syntactic structure clearer and subsequent analysis more accurate.

[0047] 2. Multi-attribute annotation: Dynamically add multi-dimensional semantic attributes to each semantic unit through the multi-attribute annotation module, and dynamically adapt attribute labels through predefined attribute templates or knowledge graph reasoning.

[0048] Specifically, in this embodiment, multi-dimensional semantic attributes are added to each semantic unit, including but not limited to:

[0049] Entity type (person, organization, place, event, etc.);

[0050] semantic categories (causal, contextual, conclusion, temporal, etc.);

[0051] Hyponym and hyponym structures (e.g., “deep learning” → “artificial intelligence”);

[0052] Spatiotemporal annotation (time, place);

[0053] Subject area (medicine, law, education, etc.).

[0054] The Multi-Attribute Annotation module supports adaptively configured attribute templates, allowing users to customize or automatically infer them based on the knowledge graph. This refers to a mechanism for adding multi-dimensional semantic attributes to extracted semantic units. The system annotates each semantic unit with attribute labels such as entity type, semantic category, hyponym, subject area, and time and location information.

[0055] Among them, the attribute system design is flexible: the label set can be dynamically adapted according to the type and context of the semantic unit. For example, if the semantic unit is identified as an "entity-person", the system will automatically match attributes such as occupation and identity; if it is identified as a "product concept", it will add attributes such as category, brand, and function; if it is a scientific and technological term, it can be annotated with the subject to which it belongs, the superordinate concept, etc. This process can be implemented with the help of an ontology or knowledge graph: the system predefines attribute templates corresponding to different entity types, and when a semantic unit is classified into a specific type, it automatically retrieves or infers its desirable attributes and annotates them. For the hierarchical relationship, the system can identify the superordinate category of the new concept through semantic analysis.

[0056] This solution also similarly determines the relationship between concepts and superordinate concepts based on the context, and records it as an attribute. The significance of the multi-attribute mechanism is to enrich information representation: a semantic unit carries multiple attributes at the same time, which can provide more delicate semantic distinctions in retrieval or evaluation. In addition, the system provides flexible configuration: users can define attribute lexicons and hierarchies, such as adjusting the application of the "disease-symptom-treatment" hierarchical relationship, or expanding new attributes (such as emotional tendencies) to adapt to different application scenarios. In short, the flexible multi-attribute annotation system embeds semantic units into a richer semantic network, so that the same unit can obtain corresponding annotations in different fields / levels, improving the accuracy of semantic understanding and the relevance of retrieval.

[0057] 3. Density Score: The information density score module calculates the total information content of all semantic units within a unit of text length based on the rarity and attribute richness of the semantic units. The calculation formula is as follows:

[0058]

[0059] U={u1,u2,...,u n} is a set of semantic units, w(u i )=α·IDF(u i )+β·|A(u i )| is the semantic unit weight, L is the unit text length (which can be calculated by the number of characters, words, or sentences);

[0060] Among them, IDF(u i ) is the inverse document frequency of the semantic unit, which measures its rarity in the corpus; |A(u i )| is the number of attributes annotated in the unit; α and β are adjustable weights used to balance the contribution of semantic scarcity and attribute information.

[0061] This score takes into account structure, semantic rarity, and attribute richness, and is a density quantification model for semantic interpretation. Specifically, within a vertical system, the information content of a semantic unit is proportional to its rarity and attribute richness. Using this definition, units with more rare concepts and more additional attributes contribute more information.

[0062] The information density logic embodied in the above formula is "the total weight of valid semantic units contained in a unit length of text." Furthermore, the formula can be fine-tuned to meet application needs, such as assigning different weights to different attribute types or introducing a semantic strength function. Unlike methods that only consider word frequency, this scoring method integrates the attribute diversity and global rarity of semantic units. Building on this foundation, this method utilizes semantic abstraction and attribute assignment to achieve greater interpretability and a higher degree of matching. This method, directly based on interpretable semantic units and attribute calculations, avoids the drawback of purely traditional probabilistic entropy, which lacks a clear correspondence to specific semantics.

[0063] Fourth, Distribution Modeling: The density distribution modeling module uses a sliding window or paragraph segmentation strategy to generate a text information density distribution map and identify high- and low-density areas. The sliding window size and step size in the density distribution modeling step support dynamic adjustment, and the high- and low-density thresholds are configurable based on the application scenario. The density distribution modeling module outputs: local density highlighted paragraphs; low-density area identification; and an information density heatmap visualization interface.

[0064] The output forms include: density heat map, highlighted peak interval (>threshold σ), and low-density area mark (<threshold δ).

[0065] 5. Application Interface: Density scoring and distribution results are applied to downstream applications through the downstream application interface module, which can serve the following tasks:

[0066] Automatic summarization and sentence selection (prioritizing high-density segments);

[0067] Document navigation and structure rearrangement (jumping according to information peaks);

[0068] Content compression and course streamlining (removing redundant and low-density paragraphs);

[0069] Information visualization and knowledge graph docking.

[0070] Based on the above, the present invention uses semantic unit extraction and multi-attribute annotation to identify phrase-level or concept-level keywords and assign them rich semantic tags, eliminating the need for simplistic segmentation of important concepts and enhancing the semantic expressiveness of retrieval and analysis. For example, different expressions of the same concept ("Traditional Chinese Medicine" and "Traditional Medicine") are classified as the same semantic unit or associated attribute, rather than as two independent words in TF-IDF.

[0071] Secondly, although the language model entropy of existing technologies (such as the probabilistic entropy based on word sequences) can measure the uncertainty of text, as a black box measurement, it is difficult to explain which specific information is dominant. It also requires a large amount of unsupervised corpus and model training, and has limited adaptability to domains.

[0072] The present invention directly measures the extracted concept-level information, avoiding the overhead of complex models, and the scoring formula can be freely adjusted according to the application, more flexibly meeting different needs.

[0073] In summary, this solution overcomes the semantic void of TF-IDF and the interpretability weaknesses of pure entropy metrics by calculating information density from a semantic and multi-attribute perspective, offering significant advantages in key term extraction and domain adaptation. Furthermore, incorporating information density calculations into the annotation system enables more refined sorting and filtering strategies, such as prioritizing attribute-rich and highly rare paragraphs in RAG searches, improving recall quality.

[0074] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any person skilled in the art who, within the technical scope disclosed by the present invention, makes equivalent substitutions or modifications based on the technical solutions and improved concepts of the present invention shall be covered by the scope of protection of the present invention.

Claims

1. A method for calculating information density, characterized in that: The following steps are involved: 1) Semantic Unit Extraction: This process integrates word segmentation, syntactic dependency analysis, and semantic role labeling to identify subject-verb-object structures, proper noun combinations, and verb-object phrases from the original text. It then unifies terminology using domain dictionaries and concept ontologies, and outputs a structured list of semantic units. 2) Multi-attribute annotation: Dynamically attach multi-dimensional semantic attributes to each semantic unit, and dynamically adapt attribute labels through predefined attribute templates or knowledge graph reasoning; 3) Density score: Based on the rarity and attribute richness of semantic units, the total information content of all semantic units within a unit of text length is calculated; 4) Distribution Modeling: Using a sliding window or paragraph segmentation strategy, we generate a text information density distribution map, identify high-density and low-density areas, and generate a density heat map and interval markers. 5) Application interface: Apply density scores and distribution results to downstream applications and provide density data services.

2. The information density calculation method according to claim 1, characterized in that: The semantic unit extraction step supports multilingual text processing and multi-layer nested structure analysis, and verifies the semantic integrity of candidate phrases through a deep model.

3. The information density calculation method according to claim 1, wherein: The semantic attributes include: entity type, semantic category, hyponym structure, spatiotemporal annotation and subject area.

4. The information density calculation method according to claim 1, wherein: The calculation formula of the density scoring step is as follows: U={u1,u2,...,u n } is a set of semantic units, w(u i )=α·IDF(u i )+β·|A(u i )| is the semantic unit weight, L is the unit text length; Among them, IDF(u i ) is the inverse document frequency of the semantic unit, which measures its rarity in the corpus; |A(u i )| is the number of attributes annotated in the unit; α and β are adjustable weights used to balance the contribution of semantic scarcity and attribute information.

5. The information density calculation method according to claim 1, wherein: The sliding window size and step size in the distribution modeling step support dynamic adjustment, and the high-density area and low-density area thresholds are configured according to the application scenario.

6. The information density application system based on semantic abstraction unit and multi-attribute fusion according to any one of claims 1 to 5, characterized in that: include: Semantic unit extraction module, which is used to extract units with independent semantic meaning from the original text and output semantic units with structural labels; Multi-attribute annotation module: used to dynamically add multi-dimensional semantic attributes to each semantic unit; Information density scoring module: Generates text density scores based on the rarity and attribute richness of semantic units; Density distribution modeling module: Models information density changes in text using sliding windows or paragraph structure strategies, generating density heat maps and interval markers; Downstream application interface module: used to apply density scoring and distribution results to downstream applications and provide density data services.

7. The information density application system based on semantic abstraction unit and multi-attribute fusion according to claim 6, characterized in that: The downstream application interface module supports the priority extraction of high-density paragraphs or the rearrangement of document structure and the elimination of redundant content based on a heat map.

8. The information density application system based on semantic abstraction unit and multi-attribute fusion according to claim 6, characterized in that: The downstream application interface includes a visualization interface that displays density distribution with an interactive heat map and supports dynamic adjustment of thresholds.

Images