A three-dimensional compression method for long text oriented to large language models

Abstract

This invention discloses a three-dimensional compression method for ultra-long texts oriented towards large language models, comprising: receiving ultra-long text data and segmenting it into initial word sequences; extracting three-dimensional features from the entire text based on these sequences to obtain word frequency, sentence, and contextual features; merging the three-dimensional features to obtain corresponding word frequency, sentence, and contextual compression results; integrating the three compression results, performing a semantic integrity check on the resulting preliminary compressed text, and supplementing missing information from the original text based on the check results to obtain the final compressed text. This invention improves the information fidelity and logical consistency of compressed text through collaborative extraction of three-dimensional features and integrity constraints, solving the problems of severe information loss and low usability caused by insufficient semantic understanding and logical breaks in existing methods.

Description

Technical Field

[0001] This invention relates to the field of text recognition technology, and in particular to a three-dimensional compression method for ultra-long texts with large language models. Background Technology

[0002] With the rapid development of large language model technology, its applications in text understanding, information extraction, and intelligent question answering are becoming increasingly widespread. However, current mainstream large language models are generally limited by their context window length. Even advanced models are typically limited to processing text lengths of several thousand to tens of thousands of tags. This inherent limitation makes it difficult for models to perform effective global understanding and in-depth analysis when faced with extremely long texts, such as lengthy books, in-depth research reports, academic papers, product evaluation documents, interview transcripts, legal contracts, medical case studies, financial statement analyses, technology patents, news reports, and meeting minutes—documents that can easily reach tens or even hundreds of thousands of words.

[0003] To address the technical challenge of processing extremely long texts, existing technologies mainly offer the following solutions: One is the simple truncation method, which directly extracts a limited portion of the text from the beginning and inputs it into the model for processing. While this method is simple to operate, it completely discards a large amount of crucial information from the following text, resulting in a severe lack of understanding of the overall document structure and core content. Another method is the sliding window method, which divides the long text into multiple overlapping paragraphs, processes them separately, and then concatenates the results. While this method can expand the processing scope to some extent, it struggles to effectively capture and restore the global semantic coherence of the document, and the concatenation process between paragraphs is inefficient, easily leading to information fragmentation. Additionally, there are summarization-based methods, which automatically summarize the original text before inputting it into the model. However, traditional summarization algorithms often rely on word frequency statistics or simple positional heuristics for information filtering, such as focusing only on high-frequency words or extracting only sentences from the beginning and end of the text. This single-dimensional information extraction approach lacks the ability to understand the deep semantics and logical structure of the text, resulting in compressed text often losing a large amount of crucial factual information and logical connections, making it difficult to support high-quality, in-depth analysis and summarization tasks.

[0004] Specifically, at the lexical level, existing technologies mostly employ statistical methods such as TF-IDF for keyword extraction. These methods measure word importance solely from the perspective of word frequency and document frequency, failing to identify the deeper semantic meaning of words. This leads to problems such as redundant synonyms and unclear expression of core concepts, and the fixed number of extracted words is difficult to adapt to the dynamic needs of different document types and content densities. At the sentence level, existing methods mostly score and filter based on simple features such as sentence position, length, or whether specific words are contained, lacking a comprehensive consideration of the functional roles of sentences and their logical dependencies. This results in a set of extracted sentences that is not very representative and makes it difficult to reconstruct the original text's argumentative structure. At the context level, existing technologies often use fixed entity extraction templates, such as forcibly extracting "person-time-place-event-result." Such fixed rules are only applicable to specific types of narrative text. For document types with diverse structures and complex logic, such as technical papers, research reports, and legal documents, they appear rigid and inefficient, failing to effectively capture the unique core dimensions and structured information of the document. More importantly, existing methods generally lack constraints on the logical consistency and semantic integrity of compressed text. Problems such as broken causal chains, missing comparison benchmarks, unclear terminology definitions, and invalid data references frequently occur in the compression results, which seriously affect the usability and comprehensibility of compressed text. Summary of the Invention

[0005] The purpose of this invention is to provide a three-dimensional compression method for ultra-long texts oriented towards large language models. By deeply integrating the collaborative extraction of three-dimensional features of word frequency, sentence, and context with integrity constraints, the method improves the information fidelity and logical consistency of compressed texts, and solves the problems of severe information loss and low usability of compression results caused by insufficient semantic understanding and logical breaks in existing methods.

[0006] To address the aforementioned technical problems, a first aspect of this invention provides a method for three-dimensional compression of ultra-long texts for large language models, comprising the following steps:

[0007] Receive extremely long text data, perform word segmentation on the extremely long text data, and generate an initial text word sequence;

[0008] Based on the initial text word sequence, three-dimensional feature extraction processing is performed on the entire ultra-long text to obtain word frequency dimension features, sentence dimension features and context dimension features respectively;

[0009] The word frequency dimension feature, the sentence dimension feature and the context dimension feature are respectively merged to obtain the word frequency dimension compression result, the sentence dimension compression result and the context dimension compression result;

[0010] The compression results of word frequency dimension, sentence dimension, and context dimension are integrated. The semantic integrity of the preliminarily compressed text obtained after integration is checked. Based on the check results, missing information is filled in based on the original text of the ultra-long text to obtain compressed text data.

[0011] Furthermore, the step of performing three-dimensional feature extraction processing on the entire ultra-long text based on the initial text word sequence includes:

[0012] An improved TF-IDF method based on semantic vector clustering is used to perform word frequency dimension processing on the entire text of ultra-long texts to obtain word frequency dimension features;

[0013] A composite scoring mechanism based on sentence function classification and dependency modeling is used to perform sentence dimension processing on the entire text of ultra-long texts to obtain sentence dimension features;

[0014] An LLM-driven rule generation engine is used to perform contextual dimension processing on the entire text of ultra-long texts to obtain contextual dimension features. The LLM-driven rule generation engine first identifies the document type, then generates targeted information extraction rules based on the document type, and finally applies the information extraction rules to extract contextual information.

[0015] Furthermore, the step of performing word frequency dimension processing on the entire ultra-long text using the improved TF-IDF method based on semantic vector clustering includes:

[0016] The semantic vector of each word in the initial text word sequence is obtained using a pre-trained language model;

[0017] K-means clustering is performed on the semantic vectors of all words, and words with similar semantics are aggregated into concept nodes. Each concept node selects one output word.

[0018] Calculate the improved semantic TF-IDF weight for each output word, which is related to word frequency, inverse document frequency, and semantic center distance;

[0019] The output words are sorted and filtered according to the improved semantic TF-IDF weights to obtain word frequency dimension features.

[0020] Furthermore, the composite scoring mechanism based on sentence function classification and dependency modeling is used to process the sentence dimension of the entire text, including:

[0021] Each sentence in the entire long text is classified by function, and the function weight of each type of sentence is different.

[0022] Identify sentence pairs that have a relationship, and label the relationship type of each sentence pair. The relationship types include causal relationship, contrast relationship, parallel relationship, progressive relationship, complementary relationship and reference relationship. Based on the labeling results, establish a sentence dependency graph.

[0023] Based on the functional weight of each sentence and the degree of each sentence in the sentence dependency graph, key sentences are extracted to obtain sentence dimension features.

[0024] Furthermore, the LLM-driven rule generation engine performs contextual processing on the entire text, including:

[0025] The document type is determined by analyzing the structural features, vocabulary style, and paragraph organization of the entire text using LLM analysis.

[0026] LLM is used to generate information extraction rules corresponding to the document type. The extraction rules include core dimensions, weight allocation, keyword library and positional bias.

[0027] The aforementioned extraction rules are applied to extract contextual information from the entire text of the ultra-long text, resulting in contextual dimension features.

[0028] Furthermore, the feature merging of the word frequency dimension features, sentence dimension features, and context dimension features includes:

[0029] A position-weight dual-dimensional adaptive compression strategy is adopted to merge word frequency dimension features. The position weight configuration is determined according to the document type. The ultra-long text includes a front core region, a rear core region, and a middle region. The retention rate and sampling rate of each region are different. The comprehensive weight of each output word is calculated based on the position weight, the improved semantic TF-IDF weight, and the sentence function weight. The words are sorted in descending order of comprehensive weight and the retained words are selected to generate the word frequency dimension compression result.

[0030] The information association graph is constructed to merge the sentence dimension features. Each key sentence in the sentence dimension features is used as a node, and the relationship between key sentences is used as an edge to construct a directed weighted graph. The shortest semantic path algorithm is used to retain the core nodes and the nodes on their shortest path, and to delete isolated nodes and redundant branches to generate the sentence dimension compression result.

[0031] The entities in the context dimension features are deduplicated and merged, and the merged entities are filtered based on the information association graph to generate the context dimension compression result.

[0032] Furthermore, the method of merging word frequency dimension features using a position-weight dual-dimensional adaptive compression strategy includes:

[0033] The position weight configuration is determined based on the document type, and the position weight configuration includes the front core region weight, the rear core region weight, the middle region weight, and the equal sampling rate;

[0034] The extremely long text is divided into a front core region, a rear core region, and a middle region according to its position.

[0035] High-weight retention is applied to the front core area and the rear core area, with a retention rate of the first preset ratio value;

[0036] Perform proportional sampling in the middle region, and perform uniform sampling according to the sampling rate determined by the document type;

[0037] The comprehensive weight of each output word is calculated by combining positional weight, improved semantic TF-IDF weight, and sentence function weight. The words are then sorted in descending order of comprehensive weight and selected for retention.

[0038] Furthermore, the construction of the information association graph merges the sentence-dimensional features, including:

[0039] A directed weighted graph is constructed using each key statement in the statement dimension features as a node and the relationships between key statements as edges; the relationships include causal relationships, comparative relationships, parallel relationships, progressive relationships, supplementary relationships, and referential relationships.

[0040] The core nodes are determined from the nodes based on sentence function weights. The core nodes and the nodes on their shortest paths are retained by the shortest phrase semantic path algorithm, while isolated nodes and redundant branches are deleted.

[0041] Multiple nodes expressing the same semantics are merged to form a streamlined information association network.

[0042] Furthermore, the process of deduplicating and merging entities in the contextual dimension features, and filtering the merged entities based on the information association graph, includes:

[0043] For entities of the same category, deduplication is performed based on the entity name or identifier;

[0044] For each type of entity after deduplication, its importance score is calculated in the information association graph. The importance score is determined based on the entity's centrality, association strength and frequency of occurrence in the graph.

[0045] For each type of entity after sorting, core entities are retained based on importance score thresholds, while low-importance entities are deleted.

[0046] Furthermore, the initial compressed text obtained after integration undergoes a semantic integrity check, and based on the check results, missing information is supplemented from the original long text to obtain compressed text data, including:

[0047] Perform a causal integrity check: If the preliminary compressed text retains the result description but lacks the corresponding cause description, then extract the cause description from the original long text and add it in;

[0048] Perform a comparison integrity check: If the preliminary compressed text retains the comparison conclusion but lacks the definition of the object being compared, then extract the definition of the object being compared from the original long text and add it in;

[0049] Perform a data integrity check: If the preliminary compressed text contains performance improvement data but lacks a baseline value, then extract the baseline value from the original long text and add it in;

[0050] Perform a citation integrity check: If the preliminary compressed text retains charts or literature citations but lacks corresponding citation source descriptions, then extract the citation source descriptions from the original long text and add them in;

[0051] Perform a definition integrity check: If the preliminary compressed text contains technical terms and the definition sentence is missing at the first occurrence of the term, then extract the definition sentence from the original long text and insert it into the first occurrence position;

[0052] The data, after integrity checks and completion, is output as the compressed text data.

[0053] Accordingly, a second aspect of the present invention provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the above-described method for ultra-long text three-dimensional compression for large language models.

[0054] Accordingly, a third aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described method for three-dimensional compression of ultra-long texts for large language models.

[0055] The above-described technical solutions of the embodiments of the present invention have the following beneficial technical effects:

[0056] 1. By extracting and deeply integrating features from three dimensions—word frequency, sentence structure, and context—a multi-layered, three-dimensional text information capture mechanism is constructed. The word frequency dimension employs semantic vector clustering technology to overcome the limitations of traditional word frequency statistics, aggregating synonyms into concept nodes to achieve accurate identification of core concepts. The sentence structure dimension accurately grasps the functional role and logical positioning of each sentence through functional classification and dependency modeling. The context dimension leverages an LLM-driven rule generation engine to dynamically generate targeted information extraction rules based on document type, enabling the compression process to adapt to the characteristics of different text types. This three-dimensional collaborative feature extraction approach fundamentally solves the problems of insufficient semantic understanding and one-sided information capture in traditional single-dimensional compression methods, ensuring complete coverage from local vocabulary to global semantics.

[0057] 2. A position-weight dual-dimensional adaptive compression strategy is adopted. Differentiated weights and sampling rates are configured for the initial core area, the later core area, and the middle area according to the document type. This strengthens the preservation of key positional information such as the opening background and the conclusion, while the middle argument is effectively condensed through intelligent sampling. At the same time, by constructing an information association graph, with key sentences as nodes and logical relationships such as cause and effect, comparison, and progression as edges, the shortest semantic path algorithm is used to preserve the core argument chain and delete isolated and redundant nodes. This ensures that the compressed text not only retains key information units but also more completely restores the logical context and argument structure of the original text, effectively solving the problems of information fragmentation and poor logical coherence in traditional compression methods.

[0058] 3. A comprehensive semantic integrity constraint mechanism is introduced before the compressed text is generated. Causal integrity checks ensure the complete correspondence between conclusions and causes; comparative integrity checks ensure the clarity and traceability of the definitions of both sides being compared; data integrity checks maintain the correlation between performance indicators and benchmark values; citation integrity checks ensure the traceability of figures and documents; and definition integrity checks ensure that technical terms have clear explanations when they first appear. This allows the compressed text to maintain an unbroken causal chain, complete data benchmarks, and unambiguous terminology definitions while significantly reducing its length. This significantly improves the usability and understandability of the compressed text and provides a high-quality input foundation for large language models to process ultra-long texts. Attached Figure Description

[0059] Figure 1 This is a flowchart of a three-dimensional compression method for ultra-long texts oriented towards large language models provided in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present invention.

[0061] Please refer to Figure 1 , the first aspect of the embodiment of the present invention provides a three-dimensional compression method for ultra-long texts for large language models, including the following steps:

[0062] Step S100: Receive ultra-long text data, perform word segmentation processing on the ultra-long text data, and generate an initial text word sequence.

[0063] After receiving the ultra-long text data, first perform Chinese word segmentation on the ultra-long text. For example, use a word segmentation tool adapted to Chinese to split the continuous character sequence into independent words, while preserving the integrity of professional terms such as new energy vehicles and power batteries, avoiding incorrect splitting, and filtering out stop words such as "de" and "le" that contribute less to semantics. Finally, generate an initial text word sequence containing all valid words as the basic input unit for subsequent feature extraction.

[0064] Step S200: Perform three-dimensional feature extraction processing on the entire ultra-long text based on the initial text word sequence to obtain word frequency dimension features, sentence dimension features, and context dimension features respectively.

[0065] Three-dimensional feature extraction was performed on the entire research report based on the initial text word sequence. In the word frequency dimension, a pre-trained language model was used to generate semantic vectors for each word. Then, K-means clustering was used to aggregate semantically similar words such as "driving range", "charging time", and "energy density" into concept nodes. From each node, the most representative output word was selected, and its improved semantic TF-IDF (Term Frequency-Inverse Document Frequency) weight was calculated. This weight comprehensively considers the frequency of the word's local occurrence, the distribution specificity in the entire document set, and the proximity of the word to the center of its concept node. Finally, word frequency dimension features that can represent the core technical theme of the report were selected based on the weight ranking. At the statement level, each sentence in the report is functionally categorized: sentences explaining the background are classified as "background sentences," those describing problems as "problem sentences," those introducing methods as "method sentences," those presenting data as "data sentences," and those summarizing viewpoints as "conclusion sentences." Different functional weights are assigned to different types. Simultaneously, sentence pairs exhibiting causal, comparative, or other logical relationships are identified, and each sentence pair is labeled with its specific relationship type. Based on these labels, a sentence dependency graph reflecting the internal logical structure of the report is constructed. Subsequently, based on the functional weights of sentences and their connectivity in the dependency graph, statement-dimensional features encompassing core arguments and key evidence are extracted. At the context level, a large language model is used to analyze the overall structure, vocabulary style, and paragraph organization of the report, determining its document type to be an "industry research report." The large language model then generates dynamic extraction rules for this type, specifying core dimensions such as "market background," "technology trends," "competitive landscape," "policy impact," and "future outlook," as well as the corresponding keyword database and positional biases for each dimension. These rules are applied to extract structured information from the entire text, obtaining context-dimensional features containing entities and their relationships.

[0066] Step S300: The word frequency dimension feature, sentence dimension feature and context dimension feature are merged respectively to obtain the word frequency dimension compression result, sentence dimension compression result and context dimension compression result.

[0067] After obtaining the features across the three dimensions, they are merged to generate compression results for each dimension. For the word frequency dimension, the positional weight configuration is determined based on the document type of the research report. The original text is divided into a front core region (usually containing the abstract and background), a rear core region (usually containing the conclusion and recommendations), and a middle region. Different retention rates and sampling rates are applied to different regions. The comprehensive weight of each output word is calculated by combining its positional weight, improved semantic TF-IDF weight, and the functional weight of the sentence it belongs to. The words are then sorted in descending order to form the word frequency dimension compression result. For the sentence dimension, the extracted key sentences are used as nodes, and the labeled logical relationships between sentences are used as edges to construct a directed weighted graph. The core nodes in the graph are determined based on the functional weight threshold of the sentences (e.g., background sentences, question sentences, data sentences, and conclusion sentences with high functional weights). These core nodes and the nodes on the shortest path connecting them are retained using the shortest phrase semantic path algorithm. At the same time, isolated and unrelated nodes and semantically repetitive branch nodes are deleted from the graph, forming a sentence dimension compression result that retains the main argument of the report. For contextual dimension features, the extracted entities (such as "solid-state battery" and "lithium iron phosphate") are first deduplicated and merged by category. Then, based on the constructed information association graph, the importance score of each merged entity is calculated. This score comprehensively considers the entity's centrality in the graph, the strength of its association with other entities, and the frequency of its appearance in the whole text. Based on the score, core entities are retained and low-importance entities are filtered out, finally obtaining a simplified contextual dimension compression result.

[0068] Step S400: Integrate the compression results of word frequency dimension, sentence dimension, and context dimension, perform semantic integrity checks on the preliminarily compressed text obtained after integration, and fill in the missing information based on the original text of the ultra-long text according to the check results to obtain compressed text data.

[0069] The compression results from the three dimensions of word frequency, sentence structure, and context are organically integrated. Using key sentences from the sentence structure dimension as the basic framework, core vocabulary from the word frequency dimension is incorporated to replace redundant expressions. The paragraph order is then organized according to the structured information extracted from the context dimension, forming a preliminary compressed text. This preliminary compressed text then undergoes a comprehensive semantic integrity check: if the text retains a conclusion about improved technical performance but lacks a benchmark value for comparison, the benchmark data is automatically supplemented from the original text; if a policy document is cited but its name or issuing agency is missing, the citation source is extracted and added from the original text; if the technical term "solid-state battery" is not defined when it first appears, its definition is retrieved from the original text and inserted at its first occurrence; simultaneously, the causal chain is checked for breaks, and the definitions of both sides in the comparison are checked for completeness. After the above checks and supplementation, a final compressed text is generated that significantly reduces the length while maintaining logical consistency and complete information.

[0070] This invention's technical solution achieves comprehensive information capture from vocabulary concepts to sentence functions and discourse structure through collaborative feature extraction across three dimensions: word frequency, sentence structure, and context. By employing position-weight dual-dimensional adaptive compression and information association graph construction, it restores the logical flow of the original text while preserving core content. Furthermore, through semantic integrity checks and automatic completion mechanisms, it ensures the causal integrity, data traceability, and terminology clarity of the compressed results. This provides high-fidelity input for large language models processing ultra-long texts, significantly improving the model's understanding accuracy and inference efficiency.

[0071] Furthermore, step S200, which involves performing three-dimensional feature extraction on the entire ultra-long text based on the initial text word sequence, includes:

[0072] Step S210: The improved TF-IDF method based on semantic vector clustering is used to perform word frequency dimension processing on the entire long text to obtain word frequency dimension features.

[0073] First, a pre-trained language model is used to generate semantic vectors for each word in the initial text word sequence. These vectors reflect the semantic information of the words in their context. Then, K-means clustering is performed on the semantic vectors of all words, aggregating semantically similar words into multiple concept nodes. For example, in a research report on the new energy vehicle industry, words such as "range," "mileage," and "range capability" might be clustered together to form a concept node representing "range performance." One output word is selected from each concept node as its representative, chosen either by selecting the word closest to the cluster center or the word with the highest frequency. Next, the improved semantic TF-IDF weight for each output word is calculated. This weight considers not only the word frequency in the text block and the inverse document frequency in the entire document set but also introduces the distance between the word and the center of its concept node as a semantic center distance factor, giving higher weights to words closer to the core concept. Finally, all output words are sorted in descending order based on the improved semantic TF-IDF weight, and the word frequency dimension feature that best represents the core theme of the entire text is selected according to a preset retention ratio or quantity.

[0074] Step S220: A composite scoring mechanism based on sentence function classification and dependency modeling is used to perform sentence dimension processing on the entire long text to obtain sentence dimension features.

[0075] First, each sentence in the entire text is functionally categorized, such as definition sentences, question sentences, method sentences, data sentences, conclusion sentences, and transition sentences. Different functional weights are assigned to each category, with definition sentences and conclusion sentences, which are functionally important, receiving higher weights, while transition sentences receive lower weights. Then, syntactic analysis and keyword matching are used to identify sentence pairs with related relationships. Each sentence pair is labeled with its relationship type, including causal, comparative, parallel, progressive, complementary, and citation relationships. Based on these labels, a directed sentence dependency graph is constructed, where nodes represent sentences and edges represent logical connections between sentences. After obtaining the dependency graph, the importance of each sentence is comprehensively evaluated based on its functional weight and the degree of connection between it and other nodes in the graph (i.e., the number of edges). Sentences with high functional weights and high connection degrees typically carry core information; key sentences are extracted based on this, forming sentence-dimensional features.

[0076] In step S230, the LLM (Large Language Model) driven rule generation engine is used to perform context dimension processing on the entire long text to obtain context dimension features. The LLM-driven rule generation engine first identifies the document type, then generates targeted information extraction rules based on the document type, and finally applies the information extraction rules to extract context information.

[0077] First, a large language model is used to analyze the structural features, lexical style, and paragraph organization of the entire text to automatically determine the document type, such as whether it is an engineering academic paper, a product evaluation report, or meeting minutes. Then, the large language model is used to generate information extraction rules corresponding to the document type. These rules include core dimensions (such as "research question," "experimental method," and "performance comparison" in academic papers), weight allocation for each dimension, a keyword database, and the positional tendency of information within the text (e.g., "research question" tends to appear in the beginning). Finally, the generated extraction rules are applied to extract contextual information from the entire text, extracting entities and their attributes, relationships, and contextual fragments that meet the requirements of each core dimension. For example, from academic papers, structured contextual information such as descriptions of innovative points, experimental data, and conclusions is extracted to form contextual dimension features.

[0078] Through the above steps S210 to S230, the present invention achieves comprehensive semantic capture from vocabulary concepts, sentence functions to discourse structure. The word frequency dimension extracts core concept vocabulary through semantic clustering, the sentence dimension depicts the discourse context through functional classification and logical dependencies, and the context dimension extracts structured information through LLM dynamic rules. The three complement each other and jointly construct a three-dimensional feature representation of ultra-long texts, laying a solid foundation for subsequent high-fidelity compression.

[0079] Furthermore, the improved TF-IDF method based on semantic vector clustering in step S210 performs word frequency dimension processing on the entire ultra-long text, including:

[0080] Step S211 uses a pre-trained language model to obtain the semantic vector of each word in the initial text word sequence.

[0081] Pre-trained language models, through self-supervised learning on large-scale corpora, can map each word to a vector representation in a high-dimensional semantic space. This vector not only encodes the meaning of the word itself but also implies its semantic information in the context. For example, when processing a research report on new energy vehicle battery technology, the words "battery," "cell," "range," and "mileage" obtained after word segmentation will each have their corresponding semantic vectors after model processing. The geometric distance between these vectors can reflect the degree of semantic similarity between words.

[0082] Step S212 performs K-means clustering on the semantic vectors of all words, aggregating semantically similar words into concept nodes, and selecting one output word from each concept node.

[0083] The K-means clustering algorithm iteratively divides vectors in the semantic space into several clusters. Vectors within each cluster are close to each other, indicating semantic similarity, and thus are aggregated into a single concept node. For example, in a research report, terms like "range," "mileage," and "range capability" might be grouped into the same concept node because they all point to the core concept of "range performance"; while terms like "battery," "cell," and "power battery" might form another concept node. After clustering, an output word is selected from each concept node as its representative. The selection rule can be either choosing the word closest to the cluster center or choosing the word with the highest frequency in that node, ensuring that each core concept is represented by a unified word.

[0084] Step S213 calculates the improved semantic TF-IDF weight for each output word. The improved semantic TF-IDF weight is related to word frequency, inverse document frequency, and semantic center distance.

[0085] For each output word, its frequency of occurrence in the entire document is first calculated as the word frequency component. Next, its inverse document frequency (IVF) is calculated to measure the word's discriminative power across the entire document set; common words have lower IVFs, while domain-specific words have higher IVFs. Simultaneously, a semantic center distance factor is introduced, which is the distance between the output word and the center of its corresponding concept node. A closer distance indicates that the word better represents the core semantics of that concept node, thus contributing more significantly to the weight calculation. By comprehensively considering these three factors, the improved semantic TF-IDF weights can more accurately reflect the true importance of each output word as a core concept in the entire text.

[0086] Step S214 sorts and filters the output words based on the improved semantic TF-IDF weights to obtain word frequency dimension features.

[0087] All output words are sorted in descending order of their improved semantic TF-IDF weights. Then, based on a preset retention ratio or quantity, the top-ranked output words are selected as the word frequency dimension features. For example, the top 20% of output words with the highest weights can be retained, or the top 100 output words can be retained directly. These selected words represent the core set of concepts in the document. For example, key terms such as "driving range," "energy density," "charge / discharge rate," and "thermal management system" are selected from research reports, forming the core vocabulary used to strengthen thematic expression and replace redundant expressions in the subsequent compression process.

[0088] Through steps S211 to S214, word frequency processing achieves a refinement process from raw vocabulary to core concepts. Semantic vector clustering effectively solves the problem of synonym redundancy, grouping semantically similar words into single concept nodes, making the expression of core concepts clearer. Improved semantic TF-IDF weights further highlight the most representative words in concept nodes by introducing a semantic center distance factor. The word frequency features obtained by final sorting and filtering provide accurate lexical-level information support for subsequent multi-dimensional compression, ensuring that the compressed text is highly condensed at the conceptual level and semantically accurate.

[0089] Furthermore, step S220 employs a composite scoring mechanism based on sentence function classification and dependency modeling to perform sentence-level processing on the entire text, including:

[0090] Step S221: Classify each sentence in the entire long text according to its function, with different functional weights for each category of sentences.

[0091] In practical implementation, a text classifier based on a pre-trained language model can be used to perform functional identification on each sentence, such as classifying sentences into categories like definition sentences, problem sentences, method sentences, data sentences, conclusion sentences, and transition sentences. Definition sentences typically contain markers such as "refers to" or "defined as," used to clarify the meaning of terms; problem sentences are often introduced by words like "problem," "challenge," or "shortcoming," pointing out the current situation or difficulties to be solved; method sentences contain verbs such as "propose," "adopt," or "design," describing technical means or implementation paths; data sentences often contain numbers, percentages, and quantitative expressions such as "improve" or "reduce"; conclusion sentences are introduced by words like "therefore," "in summary," or "show," summarizing viewpoints or experimental results; transition sentences such as "in addition" or "furthermore" mainly serve a connecting function, with lower semantic contributions. Based on this, different functional weights are assigned to different types of sentences; for example, definition sentences, problem sentences, method sentences, data sentences, and conclusion sentences have higher functional weights, while transition sentences have lower functional weights. Taking a research report on new energy vehicle battery technology as an example, the sentence describing "solid-state batteries are a battery technology that uses solid electrodes and solid electrolytes" is classified as a definition sentence and given a higher weight; while "In addition, major manufacturers are also actively deploying" is identified as a transition sentence and given a lower weight. Through functional classification and weight allocation, each sentence has a basic importance score that matches its semantic role in subsequent processing.

[0092] Step S222: Identify sentence pairs with related relationships and label the relationship type for each sentence pair. The relationship types include causal relationship, contrast relationship, parallel relationship, progressive relationship, complementary relationship and reference relationship. Based on the labeling results, establish a sentence dependency graph.

[0093] This process is achieved collaboratively through multiple techniques, including dependency parsing, keyword matching, and semantic similarity calculation. Specifically, when two sentences contain causal markers such as "because...therefore..." or "leading to," a causal relationship is determined and the sentence is labeled as causal. If contrastive words such as "better than" or "compared to" are present, the sentence is labeled as a contrastive relationship. Parallel or supplementary markers such as "at the same time" or "in addition" are labeled as parallel or supplementary relationships, respectively. When a sentence contains progressive expressions such as "furthermore" or "more importantly," it is labeled as a progressive relationship. And citations such as "according to literature" are labeled as citation relationships. Taking the research report as an example, there is a contrastive relationship between the sentence describing "traditional liquid electrolyte batteries have safety hazards at high temperatures" and the subsequent sentence "solid-state batteries use inorganic solid electrolytes, which significantly improve thermal stability." However, there is a progressive relationship between the sentence "the energy density of solid-state batteries is expected to reach more than 500 watt-hours per kilogram" and the subsequent sentence "this is about 40% higher than current liquid batteries." After identifying and labeling all relationships, a sentence dependency graph is constructed, using each sentence as a node and the labeled relationships as directed edges, to reflect the logical structure of the entire document. The connection of nodes in the graph intuitively shows the interaction and dependence of each sentence in the textual discussion.

[0094] Step S223: Based on the functional weight of each sentence and the degree of each sentence in the sentence dependency graph, extract key sentences to obtain sentence dimension features.

[0095] The degree of a sentence in the dependency graph reflects the closeness of its connection with other sentences. A higher degree indicates that the sentence occupies a more central connection position in the text's logical network, often serving as a key node connecting preceding and following arguments. A comprehensive importance score is obtained by comprehensively considering the functional weight and degree of each sentence, such as by performing a weighted sum or product operation. Sentences with high functional weight and high degree, such as definition sentences and conclusion sentences, which are cited or related to multiple other sentences, have the highest comprehensive score and are prioritized for retention as key statements. For example, in a research report, the conclusion sentence describing "solid-state batteries, with their high energy density and high safety, are considered an important development direction for next-generation power batteries" not only has a high functional weight but may also be cited as an argument target by multiple method and data sentences in the dependency graph, thus resulting in a high degree and its extraction as one of the key statements. Through this step, a small number of core sentences that play important functional roles and occupy pivotal positions in the logical network are selected from a massive number of sentences, forming sentence-dimensional features that summarize the main argument of the entire text, providing high-quality sentence-level material for subsequent compression and fusion.

[0096] Through steps S221 to S223, sentence-level processing achieves in-depth analysis and refinement from isolated sentences to a logical network. The sentence function classification mechanism assigns each sentence a basic weight matching its semantic role, ensuring that sentences carrying core arguments, such as definitions, questions, methods, data, and conclusions, gain a first-mover advantage in subsequent screening. The construction of the sentence dependency graph transforms the text from a linear sequence into a three-dimensional logical network. By labeling various relationships such as causality, comparison, and progression, it fully reconstructs the argumentative context and interactions between sentences. Finally, the method of extracting key sentences by combining functional weights and graph degree not only retains sentences with prominent semantic importance but also ensures that these sentences occupy pivotal positions in the logical network, connecting the core argumentative chain of the entire document.

[0097] Furthermore, in step S230, the LLM-driven rule generation engine performs contextual dimension processing on the entire text, including:

[0098] Step S231: Determine the document type by analyzing the structural features, vocabulary style, and paragraph organization of the entire text using LLM analysis.

[0099] After receiving a very long full-text document, the large language model first identifies its macro-structure, such as whether the text contains standard components like abstracts, introductions, chapter titles, and references. Simultaneously, it analyzes the style of frequently occurring words in the text; for example, academic papers often use terms like "research," "experiment," and "analysis," while product review reports frequently use words like "performance," "experiment," and "comparison," and news reports often include expressions like "it is reported," "stated," and "reported." Furthermore, the model observes the organization of paragraphs, such as whether there is a clear logical progression between chapters in a paper, and whether the review text uses a general-specific-general structure. Based on the combined results of these multi-dimensional analyses, the large language model automatically determines the document type. For example, if a document of over 30,000 words contains complete chapters such as abstract, introduction, methods, experiments, and conclusions, and its vocabulary is predominantly technical, with paragraphs organized according to standard academic discourse logic, it is classified as an "engineering journal article." This document type determination provides a precise basis for generating targeted extraction rules.

[0100] Step S232: Generate information extraction rules corresponding to the document type using LLM. The extraction rules include core dimensions, weight allocation, keyword library, and positional bias.

[0101] After identifying the document type as an engineering journal article, the large language model generates a set of customized information extraction rules based on its understanding of this type of document structure. Regarding core dimensions, the rules explicitly require extraction of five dimensions: "Research Question," "Innovative Methods," "Experimental Data," "Performance Comparison," and "Conclusions and Outlook," each corresponding to a specific information block in the paper. In terms of weight allocation, the rules assign the highest weight coefficient to "Innovative Methods" to highlight technical contributions, while assigning the second highest weight to "Experimental Data" to ensure the preservation of quantitative results. Regarding the keyword library, the rules configure a typical keyword list for each core dimension; for example, the "Research Question" dimension includes words such as "existing research," "limitations," and "challenges," while the "Experimental Data" dimension includes words such as "experimental results," "performance," and "improvement." Regarding location bias, the rules specify that "Research Questions" are mainly located in the beginning of the paper, "Experimental Data" is concentrated in the middle experimental sections, and "Conclusions and Outlook" appears in the end. This set of rules comprehensively describes all the parameters and constraints required for contextual information extraction from an engineering journal article.

[0102] Step S233: Apply extraction rules to extract contextual information from the entire long text to obtain contextual dimension features.

[0103] Based on the customized rules generated in step S232, the system performs structured information extraction from the entire text. First, candidate paragraphs are located based on positional bias and a keyword database. For example, paragraphs containing keywords such as "challenge" and "problem" are searched in the beginning region to identify the research problem description; sentences containing words like "experiment" and "test" with specific numerical values ​​are located in the middle experimental section to extract experimental data; and paragraphs containing conclusion markers such as "therefore" and "in summary" are identified in the end region to extract conclusion and outlook information. Simultaneously, named entity recognition technology is used to extract key entities and their attributes from the relevant context, such as extracting algorithm names and model architectures from method descriptions, and performance indicators and comparison objects from experimental data. Taking engineering journal papers as an example, by applying rules, we can extract the following research questions: "existing Transformer models have too high computational complexity in long text processing", "propose a sparse attention mechanism" as an innovative method, "achieve an accuracy of 92.3% on the GLUE dataset" as experimental data, "improve by 15% compared to the baseline model" as performance comparison, and "this method provides a new idea for long text modeling" as a conclusion outlook. We can also extract core entities such as "Transformer", "sparse attention", "GLUE", and "accuracy" and their relationships to form a structured contextual dimension feature.

[0104] Through steps S231 to S233, contextual dimension processing achieves adaptive structured information extraction based on document type. The large language model first accurately determines the document type through multi-dimensional analysis, laying the foundation for rule generation. Then, based on type knowledge, it generates customized extraction rules containing core dimensions, weights, keywords, and positional biases, ensuring a high degree of matching between information capture and document characteristics. Finally, the rules are applied to extract structured contextual features covering key entities and their relationships from the text. This mechanism breaks through the limitations of traditional fixed-template extraction methods on document types, enabling the compression method to flexibly adapt to diverse text types such as academic papers, research reports, product reviews, and legal documents, ensuring the extraction of the most representative core contextual information from each document type.

[0105] Furthermore, step S300 involves merging the word frequency dimension features, sentence dimension features, and context dimension features, respectively, including:

[0106] Step S310: The word frequency dimension features are merged using a position-weight dual-dimensional adaptive compression strategy. The position weight configuration is determined according to the document type. The ultra-long text includes: the front core region, the rear core region, and the middle region. The retention rate and sampling rate of each region are different. The comprehensive weight of each output word is calculated based on the position weight, the improved semantic TF-IDF weight, and the sentence function weight. The words are sorted in descending order according to the comprehensive weight and the retained words are selected to generate the word frequency dimension compression result.

[0107] After determining the positional weights based on document type, the extremely long text is divided into a front core region, a rear core region, and a middle region, each with different retention and sampling rates. Taking an engineering academic paper as an example, the front core region typically contains the abstract and introduction, while the rear core region contains the conclusion and future prospects. These two regions have high information density and carry the core arguments, so a higher retention rate is set to ensure that key content is not over-compressed. The middle region contains method descriptions and experimental procedures, which are information-rich but relatively detailed, so a lower sampling rate is set for uniform sampling to control the compression ratio. Based on the region division, the comprehensive weight of each output word is calculated by combining the positional weight of its region, the improved semantic TF-IDF weight from the word frequency dimension, and the sentence function weight from the sentence dimension. For example, words located in the front core region that have both a high improved semantic TF-IDF weight and a high sentence function weight have a significantly higher comprehensive weight than other words. Then, all output words are sorted in descending order according to their comprehensive weight, and the top-ranked words are selected and retained according to the preset compression target, resulting in a word frequency dimension compression result. This result highly condenses the core concepts of the entire text at the word level.

[0108] Step S320: Construct an information association graph to merge the sentence dimension features. Take each key sentence in the sentence dimension features as a node and the association relationship between key sentences as an edge to construct a directed weighted graph. Use the shortest semantic path algorithm to retain the core nodes and the nodes on their shortest paths, delete isolated nodes and redundant branches, and generate the sentence dimension compression result.

[0109] Using the key statements extracted in step S220 as nodes and the labeled relationships between these statements as edges, a directed weighted graph is constructed. The edge weights in the graph can be assigned based on factors such as the strength of the relationship or the distance between sentences. After obtaining the complete information association graph, core nodes are first determined from all nodes based on sentence function weight thresholds. For example, definition sentences, question sentences, method sentences, data sentences, and conclusion sentences with function weights higher than the set threshold are marked as core nodes. Subsequently, the shortest semantic path algorithm is used to calculate and retain nodes on the shortest path between all core nodes. These nodes form the backbone chain connecting the core arguments, ensuring that the complete argumentative framework from problem statement to method description to data support and conclusion summary is preserved. At the same time, the algorithm deletes isolated nodes with no relationship in the graph, as well as redundant branch nodes expressing the same or similar semantics. For example, among multiple parallel sentences describing the same experimental phenomenon, only the most representative one is retained. After this series of processing, the originally scattered key statements are organized into a concise and coherent logical network, forming a statement dimension compression result.

[0110] Step S330: Deduplicatize and merge entities in the context dimension features, and filter the merged entities based on the information association graph to generate context dimension compression results.

[0111] First, the entities extracted from the context dimension are standardized. For example, entity names such as "lithium battery," "lithium-ion battery," and "lithium battery" that refer to the same technology are merged into a single entity. Simultaneously, the attribute information and contextual fragments carried by each entity are aggregated. After deduplication and merging, the entities are further filtered based on the information association graph constructed in step S320. Specifically, the importance score of each entity in the information association graph is calculated. This score comprehensively considers the entity's centrality in the graph (e.g., the number of connections between the entity and core statement nodes), the strength of its association with other entities, and the cumulative frequency of the entity's appearance throughout the text. Taking an engineering academic paper as an example, if the entity "attention mechanism" appears in multiple core statements and is frequently associated with key entities such as "Transformer" and "performance improvement," its importance score is high, and it is retained as a core entity. Secondary entities that appear only once and have weak associations are filtered out. The final result is a context dimension compression that removes redundancy and occupies a key position in the logical network.

[0112] Through steps S310 to S330, the three dimensions of features have respectively completed the transformation from scattered information to structured compression results. The word frequency dimension, through a position-weight dual-dimensional adaptive compression strategy, strengthens the preservation of high-value words in the core areas. The sentence dimension, through information association graphs and the shortest-phrase semantic path algorithm, organizes scattered key sentences into a logically coherent argumentative framework. The context dimension, through entity deduplication and graph filtering, extracts the most representative core entities and their relationships. The compression results of these three dimensions are independent yet mutually reinforcing, collectively constituting the high-quality material for the subsequent fusion and generation of the final compressed text.

[0113] Furthermore, step S310 employs a position-weight dual-dimensional adaptive compression strategy to merge the word frequency dimension features, including:

[0114] Step S311: Determine the position weight configuration based on the document type. The position weight configuration includes the weight of the front core area, the weight of the rear core area, the weight of the middle area, and the equal sampling rate.

[0115] Different document types exhibit different information distribution patterns. For instance, the core innovations and conclusions of engineering academic papers are typically concentrated in the abstract / introduction and the conclusion / future outlook sections, while the experimental methods section, though lengthy, contains numerous details. Product evaluation reports, on the other hand, often outline the evaluation background and product specifications at the beginning, provide scores and purchase recommendations at the end, and detail the testing process in the middle. Therefore, for engineering academic papers, the weights of the core areas at the beginning and end can be configured to be relatively high, while the weight of the middle area should be relatively low. Simultaneously, the proportional sampling rate for the middle area should be determined as a suitable percentage based on the information density of this document type. This configuration reflects the inherent differences in the information value of different areas within different document types.

[0116] Step S312: Divide the long text into front core region, back core region and middle region according to position.

[0117] Based on the total document length and a preset regional division ratio, for example, 8% of the text is designated as the front core region, 8% as the rear core region, and the remaining 84% as the middle region. Taking a 30,000-word engineering academic paper as an example, the front core region corresponds to the first half of the abstract and introduction, approximately 2,400 words, including the research background and problem definition; the rear core region corresponds to the conclusion and outlook section, approximately 2,400 words, including a summary of research results and future directions; the middle region covers the main content such as related work, methodological descriptions, and experimental procedures, approximately 25,200 words. This physical division provides a clear operational scope for subsequent differentiated compression.

[0118] Step S313: High-weight retention is performed in the front core region and the rear core region, with a retention rate of the first preset ratio value.

[0119] For these two areas carrying core information, a high retention rate is adopted to ensure that key content is not excessively compressed. Taking engineering academic papers as an example, a retention rate of 60% can be set for the front and back core areas, meaning that 60% of the output vocabulary in these two areas will be retained, including core expressions such as the definition of the research problem, an overview of the innovative points, and conclusions. This high-weight retention strategy reflects the priority protection of information in key locations of the document.

[0120] Step S314: Perform uniform sampling in the middle area, and perform uniform sampling according to the sampling rate determined by the document type.

[0121] The middle region typically contains a large amount of information and detailed descriptions. Using proportional sampling can effectively control the compression ratio while preserving the basic information outline of this region. Taking engineering academic papers as an example, if the sampling rate is determined by the document type to be 18%, then 18% of the words are uniformly extracted from the output vocabulary of the middle region and retained. For example, one word is selected every few words from the method description and experimental data. This ensures that the core technical details and key data points of this region can be preserved at a certain density, while significantly reducing redundant content.

[0122] Step S315: Calculate the comprehensive weight of each output word by combining the position weight, the improved semantic TF-IDF weight, and the sentence function weight. Sort the words in descending order of comprehensive weight and filter out the words to be retained.

[0123] The overall weight of each output word is determined by its positional weight within its region, its improved semantic TF-IDF weight, and the functional weight of the sentence it belongs to. For example, a word located in the front core region, with a high improved semantic TF-IDF weight and situated in a defining sentence, will have a significantly higher overall weight than a regular word located in the middle region. After calculating the overall weight of all output words, they are sorted from highest to lowest. Then, based on the preset compression target (e.g., overall compression ratio or the total number of words to be retained), words are selected and retained sequentially from the top of the sorted list, ultimately forming the word frequency dimension compression result. This process ensures that at the word level, both positional importance and the semantic importance of the word itself, as well as the functional role of the sentence, are considered, making the retained word set optimal across multiple dimensions.

[0124] Through steps S311 to S315 above, the position-weight dual-dimensional adaptive compression strategy achieves refined information filtering in the word frequency dimension. By configuring regional weights and sampling rates differently according to document type, different retention strategies are applied to different regions. The importance of each word is calculated and sorted based on multi-dimensional weights, ensuring that the final word frequency dimension compression result effectively removes redundant words while maintaining the integrity of core concepts.

[0125] Furthermore, in step S320, the construction of the information association graph merges the sentence-dimensional features, including:

[0126] Step S321: Using each key statement in the statement dimension features as a node and the relationships between key statements as edges, construct a directed weighted graph. Relationships include causal relationships, comparative relationships, parallel relationships, progressive relationships, complementary relationships, and referential relationships.

[0127] These key sentences, extracted from step S223, are sentences of high overall importance and already embody the core argument of the document. In step S222, each pair of logically related sentences was labeled with a specific relationship type, including causal, comparative, parallel, progressive, supplementary, and citation relationships. Based on this, each key sentence is mapped to a node in a graph, and the relationship between each pair of sentences is mapped to directed edges connecting the corresponding nodes. Each edge is assigned a weight based on the importance of the relationship or the semantic distance between the sentences. For example, in an engineering academic paper, a causal relationship exists between the sentence describing "existing methods have low computational efficiency" and the sentence describing "this paper proposes a sparse attention mechanism," and this edge is given a higher weight; while the weight of the parallel relationship edge between sentences describing two parallel experimental phenomena is relatively lower. Through this construction process, the originally linear text sequence is transformed into a network structure graph that intuitively reflects the logical connections between the core argument units.

[0128] Step S322: Determine the core nodes from the nodes based on sentence function weights, retain the core nodes and the nodes on their shortest paths using the shortest phrase semantic path algorithm, and delete isolated nodes and redundant branches.

[0129] First, a threshold is set based on the sentence function weight of each node. Nodes with function weights higher than this threshold are identified as core nodes. For example, sentences with high function weights, such as definition sentences, question sentences, method sentences, data sentences, and conclusion sentences, are selected as core nodes, while transition sentences with lower function weights are excluded. Then, the shortest semantic path algorithm is used to calculate the shortest path between all pairs of core nodes in the graph. The nodes on these paths (including the core nodes themselves) form the key chain connecting the entire argument's framework. Taking an academic paper as an example, the "research question" node might connect to the "experimental data" node through the "innovative method" node, and then to the "conclusion" node. This path fully presents the logical main line from problem statement to method solution, data support, and conclusion summary. While retaining these core nodes and path nodes, isolated nodes without any connecting edges are deleted because these nodes cannot integrate into the overall argument network. Redundant branch nodes are also removed; for example, among multiple nodes describing the same experimental phenomenon, only the most representative one is retained, and the rest are removed as redundant branches. After this processing, the remaining nodes and edges form one or more backbone paths that can coherently convey core information.

[0130] Step S323: Merge multiple nodes that express the same semantics to form a simplified information association network.

[0131] In key statements, multiple sentences may express the same or highly similar semantic content. For example, the same innovative point may be repeatedly emphasized in different parts of a paper, or the same concluding statement may be repeated in multiple places in an evaluation report. By calculating the semantic similarity between nodes, multiple nodes with similarity exceeding a preset threshold are merged into one node. This merged node aggregates the attribute information of the original nodes and inherits all their original associated edges, thus ensuring the unity of semantic units and the integrity of the relationships. The merging operation further eliminates semantic redundancy, making the information association network more compact and clear. The final simplified information association network is the result of statement-dimensional compression. It retains the core argument of the original text and achieves a high degree of logical condensation through path pruning and node merging.

[0132] Through steps S321 to S323, the merging process of statement dimension features transforms discrete key statements into a structured logical network. First, a directed weighted graph is constructed using relationships as edges, explicitly expressing the logical connections between statements. Then, core nodes are determined based on functional weights, and the shortest path algorithm is used to extract backbone paths, ensuring the complete preservation of the core argument chain from problem to conclusion. Finally, redundancy is further eliminated by merging synonymous nodes, achieving an optimal balance between information completeness and structural simplicity in the final statement dimension compression result, providing a high-quality sentence-level skeleton for the subsequent generation of logically coherent compressed text.

[0133] Furthermore, step S330 involves deduplicating and merging entities in the context dimension features, and filtering the merged entities based on the information association graph, including:

[0134] Step S331: For entities of the same category, deduplication is performed based on the entity name or identifier.

[0135] Entities extracted from contextual features may originate from different locations throughout the text. The same entity may have variations in name or expression in different contexts. For example, different expressions referring to the same technical object may include "lithium-ion battery," "lithium battery," or "lithium-ion battery," or the same person may appear in different paragraphs with their full name and abbreviation. Through entity name normalization, these variant names pointing to the same real-world object are uniformly mapped to a standard identifier. Simultaneously, their respective attribute information, contextual fragments, and frequency records in different locations are aggregated. Taking an engineering academic paper as an example, expressions such as "Transformer model," "Transformer architecture," and "Transformer" appearing multiple times in the text are merged into a single unified entity node representing "Transformer" after deduplication. Its attribute information aggregates descriptions from different locations, including its basic structure, year of proposal, and main characteristics, laying a clean and non-redundant entity foundation for subsequent processing.

[0136] Step S332: For each type of entity after deduplication, calculate its importance score in the information association graph. The importance score is determined based on the entity's centrality, association strength, and frequency of occurrence in the graph.

[0137] Information association graphs use key statements as nodes and logical relationships between statements as edges. Entities are incorporated into the graph through their associations with these statement nodes. For example, the entity "attention mechanism" may appear in multiple key statements and frequently co-occur or be logically associated with entities such as "Transformer" and "performance improvement." Centrality reflects the structural position of an entity in the graph; entities with many connections to core statement nodes and located at the intersection of multiple paths have higher centrality. Association strength measures the closeness of the relationship between an entity and other entities, which can be quantified by co-occurrence frequency or the weight of relational edges. Occurrence frequency directly reflects the number of times an entity is mentioned throughout the text. By weighting and combining these three factors, for example, in an academic paper, if the entity "attention mechanism" frequently appears in multiple core statements, has strong associations with multiple core entities such as "Transformer," "self-attention," and "computational efficiency," and appears dozens of times throughout the text, its importance score will be significantly higher than that of a minor entity that appears only once or twice.

[0138] Step S333: For each type of entity after sorting, retain the core entities and delete the low-importance entities based on the importance score threshold.

[0139] For each entity category, such as technical methods, performance metrics, and model architectures, they are first sorted from highest to lowest importance score. Then, a score threshold is set, and entities with scores above the threshold are retained as core entities, while those with scores below the threshold are removed. The threshold can be dynamically determined based on the distribution of the number of entities in each category, for example, retaining the top 20% of scores, or truncating based on absolute scores. Taking academic papers as an example, in the technical methods category, entities with the highest scores, such as "attention mechanism," "Transformer," and "sparse computing," are retained as core entities, while variant expressions of "residual connection," which appear only once and have weak correlation, may be deleted due to their low scores. The final set of core entities is the result of context-dimensional compression; they are the most crucial information units in the full-text context and most closely related to the core arguments.

[0140] Through steps S331 to S333, the merging process of contextual dimension features achieves a systematic refinement of scattered entities. First, scattered information of the same entity is aggregated by name deduplication to eliminate redundancy in expression. Then, based on the information association graph, the importance of each entity is comprehensively evaluated from three dimensions: centrality, association strength, and frequency of occurrence, so that the selection criteria fully reflect the actual status of the entity in the context of the whole text. Finally, core entities are retained through threshold screening, ensuring that the contextual dimension compression result maintains the integrity of key information while eliminating secondary and redundant entity information, providing accurate structured material for subsequent fusion to generate high-density compressed text.

[0141] Specifically, in step S400, the initially compressed text obtained after integration undergoes a semantic integrity check, and based on the check results, missing information is supplemented from the original long text to obtain compressed text data, including:

[0142] Step S401: Perform a causal integrity check: If the preliminary compressed text retains the result description but lacks the corresponding cause description, then extract the cause description from the original long text and add it in.

[0143] After initially compressing the text by integrating the compression results from three dimensions—word frequency, sentence structure, and context—it may retain a description of an experimental conclusion or performance improvement. For example, in the compressed text of an engineering academic paper, there might be a conclusion sentence like "Experimental results show that the proposed method significantly improves the model's inference speed," but it might lack a description of the reason behind this result, such as "This is because the introduction of a sparse attention mechanism reduces computational complexity." Causal integrity checks identify causal markers or semantic implications in the text, locate such result descriptions, and then retrieve their corresponding causal descriptions from the original, very long text. If a missing reason is found, sentences or fragments containing causal markers and semantically matching the result description are extracted from the original text and inserted near the result description in the compressed text, restoring the causal chain to its integrity. Through this check and completion, causal relationships in the compressed text are maintained, avoiding the confusion of readers who only know the result but not the reason.

[0144] Step S402: Perform a comparison integrity check: If the preliminary compressed text retains the comparison conclusion but lacks the definition of the object being compared, then extract the definition of the object being compared from the original long text and add it in.

[0145] The initial compressed text may retain comparative conclusions such as "the method proposed in this paper is superior to existing baseline models." However, if the specific definition or performance description of the compared object is missing—for example, if it is not specified which model or its performance metrics are referred to as "existing baseline models"—the supporting strength of the comparative conclusion will be greatly reduced. The comparative integrity check identifies comparative markers such as "superior to," "higher than," and "compared to," locates the comparative conclusion sentence, and checks whether there is an explicit definition or description of the compared object in the text. If a missing definition is found, the process traces back to the original, extremely long text to locate the first introduction of the compared object, extracts its definition sentence or key descriptive information, and inserts it into the appropriate position above the comparative conclusion in the compressed text. For example, extracting "The baseline model uses a traditional Transformer structure, and its average inference latency is 120 milliseconds when processing text with a length of 8,000 tokens" from the original text as the definition of the compared object provides a complete reference benchmark for the comparative conclusion.

[0146] Step S403: Perform a data integrity check: If the preliminary compressed text retains performance improvement data but lacks baseline values, extract the baseline values ​​from the original long text and add them in.

[0147] Initially compressed text often retains various quantitative statements, such as "reasoning speed increased by 40%" or "energy consumption decreased by 25%." However, without a benchmark value, the extent of the increase or decrease cannot be accurately understood. Data integrity checks identify statements in the text containing percentages, multiples, or other relative change data, determining whether these data are accompanied by a clear benchmark description. For example, in engineering academic papers, if the compressed text contains "energy consumption decreased by 25%" without specifying a benchmark, the system locates the data in the original text, extracts the benchmark description (e.g., "compared to traditional dynamic programming algorithms, energy consumption decreased by 25%" or "from 15 kWh per 100 kilometers to 11.2 kWh per 100 kilometers"), and adds the missing benchmark information near the data. This operation ensures that all quantitative statements have a clear comparable basis, making the data information in the compressed text fully interpretable.

[0148] Step S404: Perform a citation integrity check: If the preliminary compressed text retains figures, tables, or literature citations but lacks corresponding citation source descriptions, then extract the citation source descriptions from the original long text and add them in.

[0149] If the initial compressed text retains citations such as "Reference [VII] points out," but the corresponding figure descriptions, table content, or reference entries are deleted during the compression process, the citations will lose their target, leading to difficulties in text comprehension. Citation integrity checks identify citation markers such as figure citations and reference numbers in the text to check whether the compressed text contains explanatory content about the cited object. For example, in the compressed text of an engineering academic paper, if the phrase "The distribution of experimental results is shown in the attached figure" appears, but the title or explanatory text of the "attached figure" is not retained in the full text, the system will trace back to the original text, extract the figure title, caption, or related text description of the "attached figure," and add it near the citation statement to ensure that the citation has traceable textual support.

[0150] Step S405: Perform a definition integrity check: If the preliminary compressed text retains technical terms and the definition sentence is missing at the first occurrence of the term, then extract the definition sentence from the original long text and insert it into the first occurrence position.

[0151] The initial compressed text may retain some technical terms, such as "solid-state battery" and "electrolyte membrane" in new energy technology reports. However, if the defining sentence is missing at the first appearance of these terms, readers may not accurately understand their meaning. Definition completeness checks identify core domain-specific terms in the text and check whether these terms are accompanied by defining sentences or explanatory descriptions such as "refers to" or "defined as" when they first appear. For example, in engineering academic papers, if the term "sparse attention" only appears as its name without explanation, the system traces back to the original text, locates the key sentence containing the term's definition, such as "sparse attention is a mechanism that reduces complexity by limiting the computational scope of attention," and inserts this definition sentence after the term's first appearance in the compressed text. This operation ensures that all core technical terms have a clear definition in the compressed text.

[0152] Step S406: Output the data after integrity check and completion as compressed text data.

[0153] After the aforementioned five integrity checks and corresponding automatic completion operations, all identified missing information in the initially compressed text has been supplemented. The causal chain is complete, the comparison benchmark is clear, the data is traceable, the citations are sourced, and the terminology is clearly defined. The text has achieved logical self-consistency. At this point, this verified and improved text data is organized according to preset format requirements, such as maintaining paragraph structure, adding necessary transition words, and adjusting word order to improve readability. The final output is compressed text data that can be directly used as input for large language models or for human reading.

[0154] Through steps S401 to S406, the semantic integrity check and automatic completion mechanism provide the final quality assurance for the compressed text. Causal integrity checking maintains the logical chain of the argument; contrastive integrity checking ensures the validity of comparisons; data integrity checking makes quantitative information interpretable; citation integrity checking avoids missing references; and definition integrity checking ensures the comprehensibility of terminology. These five checks work together to solve the common problems of logical breaks and missing information in traditional compression methods, ensuring that the final compressed text, while significantly reducing its length, still maintains the factual integrity and logical consistency of the original text.

[0155] The following example uses a 30,000-word engineering academic paper in the field of artificial intelligence. This document includes complete sections such as abstract, introduction, related work, methods, experiments, conclusions, and references, with a target compression ratio of 80%. First, the entire text is segmented using word segmentation tools to ensure that technical terms such as "deep learning" and "attention mechanism" remain intact. In the three-dimensional feature extraction stage, the context dimension uses a large language model to identify the document type as an engineering journal article and generates extraction rules for five core dimensions: research questions, innovative methods, experimental data, performance comparisons, and conclusions / prospects. Each dimension is configured with weights and positional biases. The word frequency dimension uses a pre-trained language model to perform semantic vector clustering of technical terms, identifying 15 core concept clusters, including core technologies, related models, and evaluation metrics. The sentence dimension classifies the sentences in the entire text functionally, resulting in 42 definition sentences, 86 method sentences, 127 data sentences, 35 conclusion sentences, and 310 transition sentences. In the feature merging stage, based on the document type of engineering journal articles, the weights of the front core region (1.8), the back core region (1.5), and the middle region (18%) were configured. The text was divided into a front core region of 2400 characters, a back core region of 2400 characters, and a middle region of 25200 characters, with high-weight retention of 60% and proportional sampling of 18% respectively. The initial compression result was 7416 characters. Subsequently, an information association graph was constructed, establishing the core information path from the limitations of the existing model to the proposed improvement method, to the description of the coding structure, the citation of experimental data, and the support of conclusions. 86 key sentences on the shortest semantic path were retained. Semantic integrity checks revealed 3 missing causal relationships, 1 missing comparison benchmark, 1 missing data benchmark, 1 missing citation description, and 1 missing terminology definition. The corresponding content was supplemented from the original text, adding approximately 200 characters. Finally, based on the target compression ratio, low-weight sentences were further removed from the middle area, resulting in a compressed text of 6016 words with a compression ratio of 80%. The core arguments were fully preserved, the key data retention rate exceeded 90%, and the logical coherence and readability were both good.

[0156] As shown in Table 1, to verify the applicability and stability of this invention under different document types, 14 common document types were selected for testing, including online novels, research reports, humanities papers, engineering papers, multi-product comparative evaluations, single-product evaluation reports, interview recordings, international political analysis reports, legal contracts and agreements, medical case reports, financial statement analyses, technical patent documents, in-depth news reports, and meeting minutes. The test results show that the compression ratio of all document types remained stable between 77.9% and 82%, generally close to the design target of 80%. Regarding the retention rate of core information, the beginning and ending sections of online novels retained 90%, and the process section retained 70%; research reports retained 95% of data information and 68% of analytical information; humanities papers retained 100% of arguments and 75% of evidence; engineering papers retained 92% of methods and 88% of data; multi-product comparative evaluations retained 98% of comparison table information and 65% of detailed evaluation information; and single-product evaluation reports retained 100% of parameter information. The results show that the compressed text retains 85% of the test content, 92% of the core questions and answers in interview transcripts, 95% of the background information in political analysis, 88% of the predictive analysis, 98% of the core clauses and 72% of the detailed clauses in legal contracts, 100% of the diagnostic information in medical cases and 90% of the treatment plans, 96% of the key indicators in financial statement analysis and 70% of the analysis content, 95% of the technical solutions in technology patents and 88% of the data information, 94% of the core facts in news reports and 80% of the viewpoints, and 100% of the resolutions in meeting minutes and 75% of the discussion content. The core information retention rate for all document types is between 85% and 98%, significantly better than the 60% to 70% level of traditional methods. Regarding logical coherence scoring, all document types scored above 85 points, with legal contracts reaching 91 points, and engineering papers and political analyses reaching 90 points, indicating that the compressed text maintains good readability and argumentative integrity. The experimental data fully demonstrates that the method of this invention has excellent adaptability to different document types, and the rule generation mechanism driven by the large language model plays a crucial role in various text processing applications.

[0157] Table 1

[0158]

[0159] This invention further expands support to six common document types. Each type is configured with targeted position weights and extraction rules based on its content characteristics and information distribution patterns, thereby achieving highly adaptable compression of text from different fields.

[0160] Taking legal contracts as an example, the positional weighting is configured as follows: 2.1 for the front core area, 2.2 for the back core area, and 0.7 for the middle area, highlighting the importance of breach of contract clauses at the back. Core extraction dimensions include contract parties, rights and obligations, liability for breach of contract, dispute resolution, and conditions for effectiveness. The information chain follows the logic from parties to rights, obligations, breach of contract, and remedies. Special handling requires that the definition sentence be retained when legal terms first appear, and the liability for breach of contract clause must be 100% complete. This applies to scenarios such as commercial contracts, labor contracts, lease agreements, and confidentiality agreements. Similarly, in medical case reports, the weighting of the front core area is 1.9 to highlight symptom description, 1.6 for the back core area, and 0.9 for the middle area. Core dimensions cover patient information, symptoms and signs, diagnostic conclusions, treatment plans, medication records, and prognostic assessments. The information chain extends from symptoms through examination, diagnosis, treatment to prognosis. Key vital signs and abnormal examination indicators must be 100% retained, and medication dosage is indispensable. This applies to scenarios such as clinical cases, teaching cases, and discussions of difficult cases. In addition, the financial statement analysis is configured with a front core area weight of 2.0 to focus on core indicators, a back area weight of 1.9 to focus on risks and outlook, and a middle weight of 0.7. Core dimensions include operating revenue, net profit, assets and liabilities, cash flow, financial ratios, and risk factors. The information chain is from revenue through costs, profits, cash flow to risk assessment. Key items and year-on-year and quarter-on-quarter data of the three major financial statements are retained first. It is suitable for annual report analysis, quarterly report interpretation, investment research, and audit reports.

[0161] The core area of ​​a technical patent document has a weight of 2.1 at the beginning to emphasize the technical problem, 1.8 at the end to highlight the claims, and 0.7 in the middle. The core dimensions involve the technical field, background technology, technical problem, solution, technical effect, and claims. The information chain extends from the prior art problem through the inventive solution, implementation method, to the technical effect. The claims must be retained in their entirety, word for word; neither independent nor dependent claims can be omitted. This applies to invention patent specifications. The core area of ​​an in-depth news report has a weight of 1.7 at the beginning to focus on the core event, 1.6 at the end to emphasize impact and viewpoints, and 0.9 in the middle. The core dimensions include the core event, time, place, people, event background, impact analysis, expert opinions, and subsequent developments. The information chain extends from the cause, process, and impact of the event to viewpoints and trends. The 5W1H elements are prioritized, and direct quotations must be cited. This applies to investigative reports, feature articles, in-depth analyses, and interviews. The meeting minutes record the core area at the beginning with a weight of 1.4, covering basic meeting information; the latter part with a weight of 1.8, emphasizing resolutions and actions; and the middle part with a weight of 0.9. The core dimensions include meeting topics, attendees, discussion points, meeting resolutions, action plans, responsible persons, and deadlines. The information chain extends from the topics through discussion and resolutions to action items, responsible persons, and deadlines. Meeting resolutions and action plans must be 100% retained and include specific responsible persons and deadlines. Duplicate statements and polite phrases should be removed. This method is suitable for scenarios such as board meetings, project meetings, departmental meetings, and strategic discussions.

[0162] The fourteen document types mentioned above comprehensively cover mainstream fields such as business, scientific research, law, medicine, finance, and news. Through differentiated position weight configuration and targeted extraction rules, the method of this invention demonstrates broad application adaptability. Actual testing and verification show that the compression ratio of various document types is consistently around 80%, and the core information retention rate is between 85% and 98%, fully demonstrating the effectiveness and stability of this method in multi-type text processing.

[0163] Accordingly, a second aspect of the present invention provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the above-described method for ultra-long text three-dimensional compression for large language models.

[0164] Accordingly, a third aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described method for three-dimensional compression of ultra-long texts for large language models.

[0165] The embodiments of this invention aim to protect a three-dimensional compression method for ultra-long texts targeting large language models, which has the following effects:

[0166] 1. By extracting and deeply integrating features from three dimensions—word frequency, sentence structure, and context—a multi-layered, three-dimensional text information capture mechanism is constructed. The word frequency dimension employs semantic vector clustering technology to overcome the limitations of traditional word frequency statistics, aggregating synonyms into concept nodes to achieve accurate identification of core concepts. The sentence structure dimension accurately grasps the functional role and logical positioning of each sentence through functional classification and dependency modeling. The context dimension leverages an LLM-driven rule generation engine to dynamically generate targeted information extraction rules based on document type, enabling the compression process to adapt to the characteristics of different text types. This three-dimensional collaborative feature extraction approach fundamentally solves the problems of insufficient semantic understanding and one-sided information capture in traditional single-dimensional compression methods, ensuring complete coverage from local vocabulary to global semantics.

[0167] 2. A position-weight dual-dimensional adaptive compression strategy is adopted. Differentiated weights and sampling rates are configured for the initial core area, the later core area, and the middle area according to the document type. This strengthens the preservation of key positional information such as the opening background and the conclusion, while the middle argument is effectively condensed through intelligent sampling. At the same time, by constructing an information association graph, with key sentences as nodes and logical relationships such as cause and effect, comparison, and progression as edges, the shortest semantic path algorithm is used to preserve the core argument chain and delete isolated and redundant nodes. This ensures that the compressed text not only retains key information units but also more completely restores the logical context and argument structure of the original text, effectively solving the problems of information fragmentation and poor logical coherence in traditional compression methods.

[0168] 3. A comprehensive semantic integrity constraint mechanism is introduced before the compressed text is generated. Causal integrity checks ensure the complete correspondence between conclusions and causes; comparative integrity checks ensure the clarity and traceability of the definitions of both sides being compared; data integrity checks maintain the correlation between performance indicators and benchmark values; citation integrity checks ensure the traceability of figures and documents; and definition integrity checks ensure that technical terms have clear explanations when they first appear. This allows the compressed text to maintain an unbroken causal chain, complete data benchmarks, and unambiguous terminology definitions while significantly reducing its length. This significantly improves the usability and understandability of the compressed text and provides a high-quality input foundation for large language models to process ultra-long texts.

[0169] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0171] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0172] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A three-dimensional compression method for ultra-long texts oriented towards large language models, characterized in that, Includes the following steps: Receive extremely long text data, perform word segmentation on the extremely long text data, and generate an initial text word sequence; Based on the initial text word sequence, three-dimensional feature extraction processing is performed on the entire ultra-long text to obtain word frequency dimension features, sentence dimension features and context dimension features respectively; The word frequency dimension feature, the sentence dimension feature and the context dimension feature are respectively merged to obtain the word frequency dimension compression result, the sentence dimension compression result and the context dimension compression result; The word frequency dimension compression results, sentence dimension compression results, and context dimension compression results are integrated. The semantic integrity of the integrated preliminary compressed text is checked, and the missing information is filled in based on the check results to obtain compressed text data. The process of extracting three-dimensional features from the entire ultra-long text based on the initial text word sequence includes: The TF-IDF method, which is based on semantic vector clustering and incorporates a semantic center distance factor, is used to perform word frequency dimension processing on the entire ultra-long text to obtain word frequency dimension features; A composite scoring mechanism based on sentence function classification and dependency modeling is used to perform sentence dimension processing on the entire text of ultra-long texts to obtain sentence dimension features; An LLM-driven rule generation engine is used to perform contextual dimension processing on the entire text of ultra-long texts to obtain contextual dimension features. The LLM-driven rule generation engine first identifies the document type, then generates targeted information extraction rules based on the document type, and finally applies the information extraction rules to extract contextual information. The step of performing word frequency dimension processing on the entire ultra-long text using the TF-IDF method based on semantic vector clustering includes: The semantic vector of each word in the initial text word sequence is obtained using a pre-trained language model; K-means clustering is performed on the semantic vectors of all words, and words with similar semantics are aggregated into concept nodes. Each concept node selects one output word. Calculate the semantic TF-IDF weight for each output word, whereby the semantic TF-IDF weight is related to word frequency, inverse document frequency, and semantic center distance; The output words are sorted and filtered according to the semantic TF-IDF weights to obtain the word frequency dimension features.

2. The method for three-dimensional compression of ultra-long texts for large language models according to claim 1, characterized in that, The composite scoring mechanism based on sentence function classification and dependency modeling is used to process the sentence dimension of the entire text, including: Each sentence in the entire long text is classified by function, and the function weight of each type of sentence is different. Identify sentence pairs that have a relationship, and label the relationship type of each sentence pair. The relationship types include causal relationship, contrast relationship, parallel relationship, progressive relationship, complementary relationship and reference relationship. Based on the labeling results, establish a sentence dependency graph. Based on the functional weight of each sentence and the degree of each sentence in the sentence dependency graph, key sentences are extracted to obtain sentence dimension features.

3. The method for three-dimensional compression of ultra-long texts for large language models according to claim 1, characterized in that, The LLM-driven rule generation engine performs contextual processing on the entire text, including: The document type is determined by analyzing the structural features, vocabulary style, and paragraph organization of the entire text using LLM analysis. LLM is used to generate information extraction rules corresponding to the document type. The extraction rules include core dimensions, weight allocation, keyword library and positional bias. The aforementioned extraction rules are applied to extract contextual information from the entire text of the ultra-long text, resulting in contextual dimension features.

4. The method for three-dimensional compression of ultra-long texts for large language models according to claim 1, characterized in that, The feature merging process for the word frequency dimension features, sentence dimension features, and context dimension features includes: A position-weight dual-dimensional adaptive compression strategy is adopted to merge word frequency dimension features. The position weight configuration is determined according to the document type. The ultra-long text includes a front core region, a rear core region, and a middle region. The retention rate and sampling rate of each region are different. The comprehensive weight of each output word is calculated based on the position weight, semantic TF-IDF weight, and sentence function weight. The words are sorted in descending order of comprehensive weight and the retained words are selected to generate word frequency dimension compression results. The information association graph is constructed to merge the sentence dimension features. Each key sentence in the sentence dimension features is used as a node, and the relationship between key sentences is used as an edge to construct a directed weighted graph. The shortest semantic path algorithm is used to retain the core nodes and the nodes on their shortest path, and to delete isolated nodes and redundant branches to generate the sentence dimension compression result. The entities in the context dimension features are deduplicated and merged, and the merged entities are filtered based on the information association graph to generate the context dimension compression result.

5. The method for three-dimensional compression of ultra-long texts for large language models according to claim 4, characterized in that, The method of merging word frequency dimension features using a position-weight dual-dimensional adaptive compression strategy includes: The position weight configuration is determined based on the document type, and the position weight configuration includes the front core region weight, the rear core region weight, the middle region weight, and the equal sampling rate; The extremely long text is divided into a front core region, a rear core region, and a middle region according to its position. High-weight retention is applied to the front core area and the rear core area, with a retention rate of the first preset ratio value; Perform proportional sampling in the middle region, and perform uniform sampling according to the sampling rate determined by the document type; The overall weight of each output word is calculated by combining positional weight, semantic TF-IDF weight, and sentence function weight. The words are then sorted in descending order of their overall weight and selected for retention.

6. The method for three-dimensional compression of ultra-long texts for large language models according to claim 4, characterized in that, The construction of the information association graph merges the sentence-dimensional features, including: A directed weighted graph is constructed using each key statement in the statement dimension features as a node and the relationships between key statements as edges; the relationships include causal relationships, comparative relationships, parallel relationships, progressive relationships, supplementary relationships, and referential relationships. The core nodes are determined from the nodes based on sentence function weights. The core nodes and the nodes on their shortest paths are retained by the shortest phrase semantic path algorithm, while isolated nodes and redundant branches are deleted. Multiple nodes expressing the same semantics are merged to form a streamlined information association network.

7. The method for three-dimensional compression of ultra-long texts for large language models according to claim 4, characterized in that, The process of deduplicating and merging entities in the contextual dimension features, and then filtering the merged entities based on the information association graph, includes: For entities of the same category, deduplication is performed based on the entity name or identifier; For each type of entity after deduplication, its importance score is calculated in the information association graph. The importance score is determined based on the entity's centrality, association strength and frequency of occurrence in the graph. For each type of entity after sorting, core entities are retained based on importance score thresholds, while low-importance entities are deleted.

8. The method for three-dimensional compression of ultra-long texts for large language models according to any one of claims 1-7, characterized in that, The initial compressed text obtained after integration undergoes a semantic integrity check, and based on the check results, missing information is completed from the original long text to obtain compressed text data, including: Perform a causal integrity check: If the preliminary compressed text retains the result description but lacks the corresponding cause description, then extract the cause description from the original long text and add it in; Perform a comparison integrity check: If the preliminary compressed text retains the comparison conclusion but lacks the definition of the object being compared, then extract the definition of the object being compared from the original long text and add it in; Perform a data integrity check: If the preliminary compressed text retains performance improvement data but lacks a baseline value, then extract the baseline value from the original long text and add it in; Perform a citation integrity check: If the preliminary compressed text retains charts or literature citations but lacks corresponding citation source descriptions, then extract the citation source descriptions from the original long text and add them in; Perform a definition integrity check: If the preliminary compressed text contains technical terms and the definition sentence is missing at the first occurrence of the term, then extract the definition sentence from the original long text and insert it into the first occurrence position; The data, after integrity checks and completion, is output as the compressed text data.

Images