Rag system document dynamic slicing strategy optimization method based on multi-dimensional analysis

Abstract

The application discloses a RAG system document dynamic slicing strategy optimization method based on multi-dimensional analysis, relates to the technical field of information retrieval processing, and comprises the following contents: analyzing and normalizing a document, identifying title, paragraph, table segment, code segment and other structural units to form a structural unit sequence, extracting title levels, paragraph boundaries and coherence to obtain structural image data; generating an abstract text by overview reading to obtain a theme set and theme complexity, extracting keywords and constructing keyword co-occurrence information to form content image data; generating a strategy parameter frame according to the structural image data and the content image data, the strategy parameter frame containing slicing size, slicing method and slicing overlap degree; slicing according to the strategy parameter frame and checking boundaries by semantic similarity to generate slicing metadata and write the slicing metadata into a knowledge base or an index system. The method keeps structure and semantics consistent, reduces fragmentation and repetition, and improves retrieval consistency and answer stability.

Description

Technical Field

[0001] This invention relates to the field of information retrieval and processing technology, specifically to a method for optimizing the dynamic slicing strategy of RAG system documents based on multi-dimensional analysis. Background Technology

[0002] In scenarios such as enterprise knowledge base Q&A, customer service assistance, R&D document retrieval, compliance review, and bidding material preparation, retrieval enhancement to generate RAGs requires submitting heterogeneous documents such as PDFs, Word documents, web pages, manuals, work order records, and explanatory texts generated from databases to the knowledge base at any time. When a query comes in, the document is retrieved first and then generated. The current system mostly adopts a document slicing-vectorization-indexing model when entering the database: the document is segmented according to fixed length, punctuation / line break rules, heading paragraphs, or semantic similarity, and vectors and indexes are generated. Due to the large length span, a lot of structural noise (tables, lists, code blocks, citations and numbering), large differences in topic distribution, and frequent updates, fixed-length, simple rule slicing is prone to cutting off key contexts at cross-paragraph dependencies, splitting the same fact into multiple slices; structural slicing produces excessively large or small fragments when the heading level is not standardized or the paragraph boundaries are unstable; although semantic slicing is close to the semantic boundary, it has high computational cost and unstable boundaries in scenarios with topic drift, dense terminology, or long paragraphs.

[0003] Meanwhile, there is no uniform scale for slice size and overlap. When slices are too large, multiple topics within a slice reduce vector representation, making it impossible to find relevant information in similarity retrieval. When slices are too small, the excessive number of fragments can cause index bloat and make it easy to miss scattered information when retrieving Top-K data. Insufficient overlap amplifies semantic breaks, while excessive overlap introduces a large amount of repetition, increasing storage, embedding, and deduplication overhead, and occupying the generation context window. During the response phase, a large number of repetitive fragments need to be concatenated within a limited window. For frequently updated documents, when the stability constraints of slice boundaries cannot be guaranteed, small changes can lead to slice rearrangement and repeated vectorization, resulting in abnormally high incremental update costs and version mixing issues. These defects can lead to recall bias, insufficient citations, or cross-segment concatenation errors on the business side, resulting in hallucinatory answers, compliance errors, and decision-making mistakes. Furthermore, in high-concurrency scenarios, they can also increase retrieval and generation latency.

[0004] Therefore, the current technical problem is that, in the case of passive access and dynamic updates of heterogeneous documents, the existing slicing lacks the ability to adaptively determine parameters based on document content and structural features. It cannot make slicing boundaries similar to those of documents in terms of slicing granularity, slicing method and slicing overlap range under the same database entry process, causing the slicing boundaries to deviate from the natural semantic boundaries of query related information. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a method for optimizing the dynamic document slicing strategy of a RAG system based on multi-dimensional analysis. This method involves generating summary text through overview reading to obtain the topic set and topic complexity, extracting keywords, and constructing keyword co-occurrence information to form content profile data. Based on the structural and content profile data, a strategy parameter frame is generated, including slice size, slicing method, and slice overlap. Slicing is performed according to the strategy parameter frame, and boundaries are verified using semantic similarity. Slice metadata is then generated and written to a knowledge base or indexing system. This method maintains structural and semantic consistency, improves retrieval consistency and response stability, and solves the technical problems described in the background art.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] The RAG system document dynamic slicing strategy optimization method based on multi-dimensional analysis slices the document and writes it into the knowledge base or indexing system. This includes parsing and normalizing the document, forming a structural unit sequence according to structural units such as titles and paragraphs, and extracting structural profile data by title level, paragraph boundary, paragraph length and paragraph coherence.

[0010] Summarized text is generated based on structural unit sequences. Topic modeling is performed to obtain a topic set and topic complexity index. Keyword set is extracted and keyword co-occurrence information is constructed to obtain content profile data.

[0011] A set of slicing strategy parameters is generated based on structural profile data and content profile data. The set of slicing strategy parameters includes slice size, slicing method and slice overlap, and is encapsulated as a strategy parameter frame.

[0012] Based on the strategy parameter frame, a slice sequence is generated. Candidate boundaries are verified and adjusted based on semantic similarity. Slice metadata is generated and the slice sequence and slice metadata are written into a knowledge base or indexing system.

[0013] Furthermore, the document undergoes unified encoding, whitespace normalization, and punctuation standardization. Meaningless control characters are removed, and structural units are identified according to headings, paragraphs, list items, table segments, and code segments. Type and level tags are generated for each structural unit, and the start and end positions are recorded to form a sequence of structural units.

[0014] Furthermore, the structural profile data is extracted from the structural unit sequence and includes heading hierarchy relationships, paragraph boundary distribution, paragraph length distribution, sentence break density, sequential continuity, and citation and numbering integrity. It also performs consistency verification on cross-page line breaks and numbering jumps, and establishes a correspondence between the structural profile data and the structural unit index.

[0015] Furthermore, the document is read in overview form based on the structural unit sequence, candidate sentence segments are organized according to the heading level while maintaining the original order, representative sentence segments are selected to generate summary text, and the summary text and heading level path are recorded together. The summary text is then split into summary fragments according to the structural unit index and incorporated into the content profile data.

[0016] Furthermore, a set of topics is obtained by using continuous structural unit fragments as input for topic modeling, and a topic complexity index is generated. The topic complexity index includes the number of topics and the degree of dispersion of topic distribution. The topic set and topic distribution results are mapped back to the structural unit index as a component of the content profile data.

[0017] Furthermore, a keyword set is extracted from the structural unit sequence. The keyword set includes paragraph keywords, table column header and row header keywords, and code segment function name and variable name keywords. Keyword co-occurrence information is constructed by co-occurrence statistics of adjacent structural units and within the same structural unit, and the keyword co-occurrence information is stored in correspondence with the structural unit index.

[0018] Furthermore, the slicing method is determined based on the structural profile data and the content profile data. The slicing method is selected from fixed-size slices, semantic slices, and structural slices. The evaluation results of the slicing method, structural clarity, and paragraph-level coherence are written into the strategy parameter frame.

[0019] Furthermore, based on the abstract text and topic complexity index, combined with paragraph length distribution, a preset threshold level for slice size is generated, and a lag switching rule is set for threshold level switching. Slice overlap is determined based on keyword co-occurrence information and paragraph-level coherence, and slice size and slice overlap are written into the strategy parameter frame.

[0020] Furthermore, after generating the slice sequence based on the strategy parameter frame, boundary windows are constructed on both sides of the candidate boundary and semantic similarity is calculated. The vector used for semantic similarity is generated by the same embedding model as the retrieval stage.

[0021] Candidate boundaries below a preset threshold are adjusted to adjacent paragraph boundaries and the boundary index is updated synchronously, while ensuring that the structural units of table segments and code segments are not split by boundaries.

[0022] Furthermore, slice metadata is generated for each slice, which includes document identifier, slice number, title level path, structural unit location, keyword tags and topic tags;

[0023] Before writing to the knowledge base or indexing system, the strategy parameter frame is checked for consistency, and then the slice sequence and slice metadata are written to the knowledge base or indexing system in a version replacement manner.

[0024] (III) Beneficial Effects

[0025] This invention provides a method for optimizing the dynamic slicing strategy of RAG system documents based on multi-dimensional analysis, which has the following beneficial effects:

[0026] By decomposing documents into a sequence of structural units and generating structural profile data, slicing decisions can be reused based on available structural criteria, reducing retrieval noise caused by misslicing of table or code segments due to misaligned heading levels. By generating summaries, topic sets, and keyword co-occurrence information under the structural unit sequence and generating content profile data, the strategy is not dependent on a single word count rule, while maintaining the integrity of key information coverage even in documents with divergent topics or excessively dense information. By combining structural and content profile data to generate strategy parameter frames and encapsulating slice size, slicing method, and slice overlap, the strategy generation and execution entry points are consistent, avoiding recall instability caused by parameter anomalies when the same document is entered into the database multiple times.

[0027] By driving the generation of slice sequences through strategy parameter frames and performing semantic similarity checks and boundary adjustments on candidate boundaries, the slice boundaries can simultaneously meet the requirements of structural unit boundaries and semantic continuity, reducing the failure of generated answers caused by context breaks. Slice metadata is generated and bound to the knowledge base or indexing system's writing process. During retrieval, the position is tracked in real time based on title hierarchy path, structural unit location, keyword tags, and topic tags. When documents are updated, new and old slices are replaced based on consistency checks of strategy parameter frames, achieving closed-loop collaboration between structural profile data, content profile data, strategy parameter frames, and slice metadata. Attached Figure Description

[0028] Figure 1 This is a diagram showing the overall process and data object relationships of the present invention;

[0029] Figure 2 This is a schematic diagram of the structural image generation and structural skeleton in step one of the present invention;

[0030] Figure 3 This is a schematic diagram illustrating the overview reading and structural alignment of the content image in step two of this invention;

[0031] Figure 4 This is a schematic diagram of the slicing strategy inference and strategy parameter frame in step three of the present invention;

[0032] Figure 5 This is a sequence diagram for step four of the present invention: slice execution - boundary verification and fine-tuning - data entry and submission. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Please see Figures 1-5 This invention provides a method for optimizing the dynamic document slicing strategy of RAG system based on multi-dimensional analysis.

[0035] Step 1: Convert document structural features, structural clarity, and paragraph coherence into a calculable and traceable structural profile, allowing Steps 2 and 3 to reference boundaries and dependencies within the same structural coordinate system. When knowledge base access scenarios involve similar content such as PDF pagination, Word paragraphs, and webpage DOM, headers, footers, tables of contents, code indentation, and table separators often appear mixed with the main text. Segmenting by character count or punctuation can cause this noise to fragment the text, leading to further fragmentation.

[0036] First, process the input document After unifying the encoding, merging whitespace, and cleaning up control characters, the system identifies headings, paragraphs, list items, table segments, and code segments. For each identification result, a type identifier, source page number, and start and end offset are written. Finally, a sequence of structural units is formed according to the original text order. .

[0037] The PDF uses a layout block aggregation process, first aggregating text line boxes to obtain line groups, then combining indentation, font style, and numbering format to determine title candidates, and using cross-page repetition rules to remove headers and footers; Word uses joint style and numbering parsing, reading paragraph style names, indentation levels, and numbering levels and fixing them as attributes; the webpage uses DOM traversal, mapping heading tags, list tags, and table tags to structural units and removing the navigation bar. To ensure uninterrupted processing, an exception flag is generated when a structural unit cannot be determined or the text is empty. And preserve the original fragment so that subsequent steps can be performed according to... Trigger a fallback branch instead of silently losing the data.

[0038] When used, the structural unit sequence Provides a unified framework for subsequent boundary modeling and isolates formatting noise; original text location information allows any subsequent results to be traced back to the document location; anomaly markers. Make parsing failures visible and allow them to be explicitly handled in subsequent processes.

[0039] Structural slicing relies on the stability of heading levels and paragraph boundaries, but missing heading styles, line breaks across pages, and mixed formatting can cause hierarchical jumps and boundary jitter. Without first assessing stability, structural slicing can produce excessively large or excessively small fragments, interfering with retrieval. Therefore, seemingly heading / boundary elements are filtered to be usable headings / boundaries. This is done from the structural unit sequence. Extract title candidates and construct a title hierarchy graph. Simultaneously construct paragraph boundary sequences It records page transitions; then it introduces hierarchical consistency constraints and boundary steady-state constraints to ensure that hierarchical transitions and boundary intervals satisfy interpretability rules, thereby obtaining measurable structural clarity. .

[0040] Title confidence is calculated based on font size, bold size, numbering shape, and indentation level. The maximum spanning tree selects hierarchical edges to maximize overall confidence. Boundary steady-state analysis is based on boundary spacing distribution and the sources of page breaks, with backtracking corrections applied to false boundaries caused by page numbers and footnotes. Structural clarity. Defined as:

[0041]

[0042] Where: structural clarity : Value This is used to characterize the interpretability of the structural skeleton and is called in step three; the compression mapping function : Value The specific form is Its function is to compress unbounded input; independent variable : Range of values Its function is to carry the weighted entropy term;

[0043] Natural constant : The value range is a constant, and its function is exponential calculation; hierarchical normalized entropy : Value Its function is to characterize the degree of disorder in the distribution of title hierarchy. Entropy is calculated after converting frequency to probability and normalized using the maximum entropy; boundary-normalized entropy... : Value Its function is to characterize the degree of disorder in the distribution of paragraph boundary intervals. It calculates the entropy by converting the binning frequency into probability and normalizes it with the maximum entropy.

[0044] Weighting coefficient : Value Its function is to regulate the hierarchical entropy. Contribution; weighting coefficient : Value Its function is to adjust the boundary entropy pair Contribution; stability constant : Value Its function is to suppress numerical amplification when entropy approaches zero;

[0045] Structural clarity : Counting events based on title-level monotonicity and numbered breakage; the more events, the better. The smaller the value; normalize the event count by the number of titles and use... It means that among them Normalized event rate. Citation completeness. Defined as the ratio of resolvable references to the total number of references; when the total number of references is... season .

[0046] Paragraph coherence Defined as the average cosine similarity of adjacent paragraph vectors; when the number of paragraphs is insufficient, let... And write the exception flag.

[0047] When using, title hierarchy diagram Paragraph boundary sequence Forming an interpretable structural framework; structural clarity By compressing the steady-state properties into a unified parameter, it becomes easier to make a consistent judgment on the availability of structural slices in step three; after the cross-page pseudo-boundary is rolled back, the boundary sequence is closer to the continuity of the main text.

[0048] Business documents often contain cross-paragraph dependencies such as "see section," "see table," and "see figure." If the quoted sentence and the target content are cut into different segments, a broken link phenomenon will occur after retrieval, where there is only a reference without explanation or only explanation without reference. This broken link manifests in the generation stage as missing references, splicing errors, or inability to verify the reference position.

[0049] This demonstrates cross-segment dependency signals and semantic drift signals at the structural level. (In the structural unit sequence) Extract reference anchors and numbered anchors from paragraphs, list items, and table sections, and construct a reference relationship diagram based on anchor point matching. For each reference edge, locate the target structural unit within a limited window and calculate the reference integrity. To indicate whether dependencies are closed; generate lightweight semantic representations for adjacent paragraphs and calculate paragraph coherence. Mark the topic's drift position.

[0050] Anchor point extraction uses a combination of trigger words, number format, and layout position as constraints, and date / version numbers cannot be used as anchor points. Target positioning employs a hierarchical search, first matching within the same level of headings, then extending broadly to adjacent levels of headings, and using a referencing relationship diagram. Loop check removes self-indicating broken links; paragraph coherence Using term-weighted directional consistency metric:

[0051]

[0052] Where: paragraph coherence : Value Its function is to characterize the degree of semantic direction consistency between adjacent paragraphs and to participate in the slicing method and overlap inference in step three; index : Range of values Its function is to iterate through adjacent paragraph pairs; the number of paragraph pairs. : Range of values Its function is to limit the coverage of the coherence calculation; number of structural units The value range is positive integers, and its function is to represent the length of the structural unit sequence and provide support. The boundary values ​​of ;

[0053] Weight : Range of values Its function is to reduce the impact of template sentences and heading sentences on consistency metrics; paragraph vectors Dimensional range Its function is to depict the first The semantic direction of each paragraph; paragraph vector Dimensional range and Consistent, functioning as... Forming adjacent pairs; inner product operation The range of values ​​is determined by the vector norm, and its function is to measure the consistency of semantic direction.

[0054] When using it, refer to the relationship diagram. and citation completeness Dependency chains can be explicitly preserved; paragraph coherence It can provide location hints for topic drift, making it easier for subsequent boundaries to approach the drift point. Hierarchical search and loop checks help avoid misbinding and merge unrelated paragraphs into the same dependency chain.

[0055] Furthermore, incremental updates are a common phenomenon in knowledge bases. If the structural profile cannot be stably aligned to the original text position, even minor edits will cause the structural boundaries to rearrange and trigger a full recalculation, resulting in duplicate vectorization, version mixing, and difficulties in tracing. Step one requires merging intermediate products into a reusable interface so that subsequent steps can read and partially reconstruct them as needed.

[0056] Therefore, the structural-side product is transformed into a persistent, verifiable, and incrementally aligned input. This involves converting the structural unit sequence... Title hierarchy diagram Paragraph boundary sequence Reference relationship diagram Structural clarity Paragraph coherence Citation completeness With exception markers Encapsulated as a structural profile ; Calculate the content fingerprint for each structural unit A structural image index table is generated, enabling steps two and three to read the specified set of structural units according to the index table.

[0057] Among them, content fingerprint Segmented rolling hashing is used to divide the structural unit into different sub-segments based on punctuation and whitespace. Hash calculations and aggregations are performed on each sub-segment for alignment during local insertion; content fingerprinting. It can also be used as an alternative to segmented MinHash or SimHash for local changes in table segments and code segments; the persistence of the structural profile is serialization with checksum, which validates key fields during writing, and only reverts and rebuilds the affected structural units and recalculates their adjacent reference edges if a read fails.

[0058] When using it, structural portrait Include structural boundaries, dependencies, and steady-state parameters in the inputs that can be directly called upon in subsequent steps; content fingerprint. It supports cross-version alignment and partial reconstruction, avoiding link jitter caused by full reordering; the persistence with checksum can detect and control the rollback of structural profile corruption.

[0059] Step 2: Structural image output in Step 1 In the input document Perform simple reading and extraction of abstracts, topics, and keywords, with each extraction result corresponding to a structural unit sequence. The positional relationships are bound together to facilitate reasoning in the slicing strategy of step three within the same coordinate system.

[0060] In enterprise knowledge base integration scenarios, headers and footers, table of contents, version history, copyright notices, and blank pages often appear mixed with the main text, and the line break and pagination rules of different document formats are inconsistent. If the overview read directly extracts summaries, themes, and keywords from the full text, noisy text will be included in the statistics, and then navigation phrases, version numbers, and page numbers will be mistakenly taken as theme clues and keyword clues, ultimately causing the slicing strategy in step three to deviate from the semantics of the main text.

[0061] First, determine the boundaries of the structural units entering the overview, and solidify the relationship chains by tracing back from these boundaries to their original positions in the text. This enables subsequent reasoning to distinguish between textual information and formatting information. Step two: Read the structural profile. Title hierarchy diagram Paragraph boundary sequence Exception markers Reference Relationship Diagram Then according to the structural unit sequence Generate an overview candidate set in sequence; for those marked as anomalies Structural units marked as failing type determination or missing text are retained with their original text and an undeterminable type indicator to avoid silent discarding that could lead to content gaps. Header, footer, and table of contents structural units are removed based on page repetition and title level position. Table segments and code segments are retained as structured fragments based on their title and boundary position to facilitate subsequent keyword extraction of key values, column headers, and code identifiers.

[0062] The overview reading process employs a two-level scanning method: first, it scans the corresponding structural units within the coverage area of ​​each title node; then, it scans the transition areas of the title nodes and takes into account paragraph coherence. The local minimum position is used to determine the transition boundary, thereby separating continuous paragraphs of the main text from paragraphs that jump across chapters; at the same time, the reference relationship diagram is used. Perform a closure check and check the completeness of references. Lower-level structural units are added to the dependency completion queue, and their reference targets are added to the structural unit sequence. The corresponding structural units in the summary are added to the candidate set of the overview to ensure that no dependency pairs are missed when calculating the summary and topic.

[0063] When using it, after the overview reading is complete and structural constraints and noise isolation are implemented, enter the abstract, topic, and keywords; anomaly markers. The abnormal structural units will still exist and can still be referenced and traced in subsequent steps; the completion queue is used to merge the reference chain into the same overview window, providing context for the slice overlap inference in step three.

[0064] Abstracts are used to present the main points and information framework of a document. However, corporate documents often contain paired structures such as clauses and explanations, conclusions and supporting evidence, and processes and exceptions. If abstracts are extracted only based on sentence importance, it is easy to retain only the conclusions while losing the supporting evidence, or only the process steps while losing the exceptions. This gap can lead to misjudgments in subsequent reasoning regarding the focus of the topic and the density of information, thus affecting the granularity and overlap of the segments.

[0065] By making the summary cover the heading context and including paragraphs that must co-occur in the dependency chain in the summary candidate, the summary becomes a global prompt available in step three rather than an isolated fragment.

[0066] Step 2: Based on the title hierarchy diagram Assign structural weights to candidate paragraphs in the abstract, prioritizing those that appear earlier in the text and have better paragraph coherence under each heading node. Higher structural units are used as candidates for topic statements; simultaneously, based on the citation relationship diagram... With citation completeness Candidates are expanded in pairs to ensure that structural units containing trigger words such as "see see," "see table," and "see figure" appear simultaneously with their target structural units in the abstract candidates. Then, the length of the candidate set is controlled; if a candidate spans multiple heading nodes, it is sorted according to paragraph boundary sequence. The stable boundary is used to divide the candidate window, avoiding semantic jumps caused by cross-chapter splicing.

[0067] The abstract generation process offers two optional engineering paths: The first is an extractive path, which segments candidate structural units into sentences, generates sentence vectors using a pre-trained encoder, and balances coverage of different heading nodes with avoiding repetitive expressions using the maximum marginal relevance criterion to output the abstract text. The second is a generative path, which concatenates candidate structural units into input based on heading nodes, generates the abstract using a sequence-to-sequence model, and performs a citation closure check after decoding: if the abstract text contains citation trigger words, it checks the citation relationship graph. The process involves adding the title and key sentences of the cited target to the input fragment and generating the new fragment again until no more targetless citation trigger words appear in the abstract text.

[0068] When in use, the abstract, driven by structural weights, covers the title's context, saving on cross-chapter jumps; the introduction of closed-loop verification is beneficial for the integrity of paired relationships between headings and content; the extraction and generation of paths allow the system to choose an implementation method without changing the input-output relationship in scenarios with insufficient computing power and high requirements for coherence.

[0069] Furthermore, topic modeling is used to characterize the set of topics involved in a document and their distribution. However, in process and specification documents, topics gradually unfold as chapters progress. If all structural units are mixed and input into the topic model, the natural differences caused by the progression of chapters will be misjudged as topic discrepancies, leading to an overestimation of topic complexity. This, in turn, prompts step three to use overly fine slicing or unnecessary high overlap.

[0070] To ensure that topic modeling respects the structural progression, structural units should first be organized into semantically continuous segments, and then the topic distribution should be characterized at the segment level. Step two involves defining the topic distribution within each heading node based on the paragraph boundary sequence. Generate candidate segments and take paragraph coherence into account. The local descent position divides the candidate fragment into smaller, continuous fragments; then, topic modeling is performed on each continuous fragment to obtain a fragment-level topic distribution, which is then aggregated into a document-level topic set according to the heading level, and the consistency of the topic distribution across fragments is used to characterize the topic complexity; when the citation relationship graph When there are frequent cross-chapter references, the relevant segments are merged into the same summary unit according to the reference edge during the topic summary stage to avoid the same topic being mistakenly split into multiple topics due to cross-references.

[0071] For topic modeling, either a probabilistic topic model or a clustering topic model can be used. The probabilistic topic model uses continuous segments as the smallest unit, and after word segmentation and stop word filtering, it trains the topic distribution using bag-of-words vectors, employing the LDA path. The clustering topic model uses continuous segment vectors as input, selecting density clustering or hierarchical clustering to obtain topic clusters, and mapping the cluster centers to topic representations. Both paths record the topic results as a ternary relationship of topic tag, source segment, and corresponding title node, and write it back to the structural unit sequence. In the location index.

[0072] For example, maintenance personnel import a maintenance manual containing fault symptoms, troubleshooting steps, parameter tables, and recovery scripts into the system. The system has already generated a structural profile in step one. And retain the structural unit sequence In step two, the system first organizes the paragraphs describing the fault symptoms into continuous segments based on the title node and completes the theme modeling, then sorts them according to paragraph coherence. Within the investigation steps section, the step descriptions and anomaly branches are separated into different continuous segments and modeled separately; when the text mentions "refer to recovery script," the system uses the reference relationship diagram. By incorporating the corresponding segments of the script chapters into the same summary unit, a set of topics that can be traced back to the title node and structural unit position is obtained. Operation and maintenance personnel can select the topic tag to locate the corresponding title and paragraph position and view its continuous segment range.

[0073] When used, continuous segment organization allows topic modeling to respect the chapter progression pattern and reduce misjudgments due to topic complexity; reference edge merging allows cross-chapter dependencies to be included in the same topic summary unit; ternary relation backwriting allows topic results to be directly referenced in step three and used for the construction of criteria for slicing method selection.

[0074] Keywords are used to express information-intensive points and retrieval anchors, but keywords in corporate documents are often distributed in titles, table headers, numbered clauses, and code identifiers. If terms are extracted only from continuous text, key entities in tables and code segments will be missed. If co-occurrence relationships are not constructed, it is impossible to determine whether a keyword depends on the limiting conditions of adjacent paragraphs, which will result in a lack of basis in the slice overlap inference in step three.

[0075] By solidifying the source of keywords, co-occurrence with other keywords, and the boundaries that co-occurrence crosses into structural alignment relationships, keywords can not only be retrieved but also used to infer contextual dependencies.

[0076] Step 2 uses structural unit sequences Keyword extraction for each unit: Candidate words are extracted from titles and paragraphs using a combined strategy of word frequency and graph ranking; column headers, row headers, and cell named entities are extracted from table segments, and key-value pairs are formed based on column headers and cells; function names, variable names, and error codes are extracted from code segments and grouped by indentation level; then, the co-occurrence of keywords in adjacent structural units is statistically analyzed within each title node, constructing keyword co-occurrence information, and the co-occurrence boundary positions are correlated with paragraph boundary sequences. Aligning records enables subsequent identification of the sources of differences between co-occurrence across paragraphs and co-occurrence across headings.

[0077] Keyword extraction can employ either a statistical path or a graph path: the statistical path sorts terms by weight within structural units and adds a weight to the title; the graph path constructs a graph of terms within structural units and obtains the sorting result through iterative solving, and the TextRank path can be used. Both paths write the keyword set and co-occurrence information into the content profile, along with structural unit indexes and title node indexes; when anomalies are flagged... When the indicated text is undeterminable, keyword extraction is reverted to character-level segmentation, retaining only consecutively occurring entity strings. This reverted result is marked as an anomaly source, so that step three can apply a conservative weight to this paragraph when judging the overlap of segments and avoid spreading the co-occurrence of anomalies to adjacent title nodes.

[0078] For statistical co-occurrence, for each structural unit Construct a set of keywords When the structural unit type When used as a title or paragraph, Obtained from the keyword extraction process; when When it is a table segment, It must contain at least the column header string, the row header string, and the named entity string in the cell; when When it is a code segment, It must include at least the function name, variable name, and error code string.

[0079] Define the intersection-union ratio for adjacent structural elements:

[0080]

[0081] Keyword co-occurrence density is defined as the average of adjacent crossover ratios:

[0082]

[0083] Among them: Keyword set : Values ​​are a set of strings, used to unify the statistical criteria for co-occurrence; set intersection operator AND operator Its function is to characterize the shared keywords and total keywords of adjacent units; cardinality operator. : Values ​​are non-negative integers, used for counting; intersection-union ratio : Range of values Its function is to determine the intensity of single-step co-occurrence; keyword co-occurrence density. : Range of values Its function is to determine the overall co-occurrence intensity.

[0084] Extracting headings, table segments, and code segments during use ensures no entity omissions; co-occurrence information and paragraph boundary sequences are also included. Alignment can distinguish between cross-paragraph dependencies and cross-heading jump dependencies; anomaly fallback can allow undecidable paragraphs to still provide retrieval anchors and can be identified by subsequent steps to adopt a conservative strategy.

[0085] Step 3: Create a structural portrait The slicing strategy parameters are combined with the content profile to output slicing strategy parameters that can be directly executed in step four, and gating back and rollback branches are given to ensure that the slicing strategy remains feasible even when structural noise, topic divergence and cross-segment dependencies coexist.

[0086] The slicing strategy parameters are output in the form of a strategy parameter frame. The strategy parameter frame contains fields for document identifier, strategy version, slice size, slice method, slice overlap, boundary sequence, and forced node sequence, and a checksum is generated based on the field order. If the checksum fails, a retransmission request is initiated and the process reverts to the cost-first approach to avoid incomplete parameter sets being read in step four and to maintain repeatability. The strategy parameter frame is ordered by field. Perform concatenation serialization, using a single delimiter between fields, for list fields (such as...) Index values ​​are separated by commas. All strings are UTF-8 encoded with no extra whitespace characters. Checksum This is the hexadecimal string obtained by performing SHA-256 on the serialized result. Step four involves recalculating the order and encoding of the same field during verification. If everyone agrees, it will pass.

[0087] In multi-source documents where the table of contents and main text are mixed, content profiling tends to favor short sentences in the table of contents, leading to a misalignment between the inferred boundaries and the main text structure, and subsequently, breaks appearing at title transitions. First, reconnect the content clues to the structural unit sequence. Therefore, inferences are only made within structurally covered areas of the text. Read the structural profile. Title hierarchy diagram Paragraph boundary sequence Exception markers Reference Relationship Diagram The abstract sentence, topic tags, and keyword tags are bound to the source structure unit; then, the percentage of paragraphs covered by the main text is calculated on a per-heading node basis. When the percentage is insufficient, gating backfilling is triggered, and the explanatory paragraphs and cited target paragraphs of adjacent headings are added to the inference input; when an anomaly is flagged... When indicating an undecidable structural unit, only its keyword tags are retained and its topic tags are frozen.

[0088] Among them, gating backfilling is executed with priority given to same-level headings and reference edge constraints; the engineering implementation can choose to execute the conditions one by one through the optional rule engine, or write each heading node to cover at least one explanatory section and each reference edge endpoint to cover at least one structural unit as constraints, and the Boolean satisfiability solver SAT will provide a satisfactory solution.

[0089] To avoid the scope of backfilling expanding indefinitely, a fixed upper limit is set for the number of backfilling iterations for the same title node. When the upper limit is exceeded, the topic statistics of the title node are frozen and the execution is switched to a cost-first approach.

[0090] When using it, infer the input and the sequence of structural units. The positional relationships are fixed, and directory noise no longer dominates the summary and topic statistics; citation endpoints are included in the same inference window, reducing the risk of cross-segment dependencies being cut off; and non-deterministic structural units no longer drive the topic distribution.

[0091] The size of the slice determines the scope of information. If only length is relied upon, it is easy to mix multiple topics into the same slice where the topic diverges, or to cut out limiting conditions outside the slice in the interpretation of terms.

[0092] This approach combines abstract coverage length, topic dispersion, and paragraph length distribution, and introduces a structural support factor to reflect structural boundary availability, thus subjecting granularity inference to both content complexity and structural stability. Abstract coverage length and the number of topic tags are aggregated by title node, and then analyzed using paragraph boundary sequences. Obtain the paragraph length distribution; then use structural clarity. Paragraph coherence Citation completeness Constructing structural support factors When the structural support factor When the slice size is high and the topic dispersion is low, increase the slice size and prioritize the use of the title boundary. (This is followed by a seemingly unrelated sentence about structural support factors.) When the slice size is low or the topic has high dispersion, reduce the slice size and prioritize the use of steady-state paragraph boundaries. Structural support factor. Defined as:

[0093]

[0094] Where: structural support factor : Value This is used to measure the support of structural boundary availability for granular inference; compression mapping function. : Value , Used to compress unbounded inputs; weighting coefficients : Value Used to adjust structural clarity Contributions;

[0095] Weighting coefficient : Value Used to adjust citation integrity Contribution; weighting coefficient : Value Used to adjust paragraph coherence Contribution; Structural clarity : Value This is used to characterize the interpretability of the heading hierarchy and boundary distribution; citation completeness. : Value Used to characterize the degree of closure of a citation chain; paragraph coherence. : Value This is used to characterize the degree of semantic consistency between adjacent paragraphs;

[0096] hyperbolic arctangent function : Value , Used to amplify the discriminative power when the coherence is close to 1; natural logarithm function : Value Used to suppress marginal gain and improve numerical stability; natural constant : A constant used in exponentiation calculations;

[0097] The slice size uses a single measurement caliber and a pre-set set of three granularity levels, selecting one from the inference chain. If abstract generation or topic modeling fails, the inference returns to the dual-input inference of paragraph length distribution and structural support factor, and the slice size is set to the medium granularity. If the granularity inference of adjacent title nodes crosses levels, the switch only occurs when two consecutive title nodes point to the new level, avoiding granularity fluctuations near the boundary.

[0098] The suggested slice size level obtained by the title node sequence is denoted as: The actual gear used is recorded as Set up a continuous consistency counter. :like but ,otherwise .when and season Otherwise The rule ensures that a switch only occurs when at least two consecutive header nodes provide the same gear recommendation.

[0099] When used, granularity inference is constrained by both structural stability and content complexity, reducing drift caused by a single caliber; structural support factors form a causal connection between structural parameters and granularity selection; and backtracking branches ensure that executable granularity can still be provided even when content profiles are missing.

[0100] The slicing method determines the basis for boundary generation. Structural slices depend on heading and paragraph boundaries, semantic slices depend on semantic transition points, and fixed-size slices depend on length accumulation. If the method selection lacks reference endpoint constraints, truncation often occurs near tables and code segments. By combining structural steady-state signals, semantic drift signals, topic clustering states, and reference endpoints, a boundary candidate graph that can be directly solved is formed.

[0101] When the structural steady state is dominant, a structural candidate graph is generated with first-level candidates for headings and second-level candidates for paragraphs. When the semantic drift is dominant, a semantic candidate graph is generated with the boundaries of steady-state paragraphs as nodes and the endpoints of the reference relationship graph are written as forced nodes. When the anomaly marker set is in play, a fixed-size candidate graph is generated and structural preservation nodes are inserted to protect the header row, table column header and the starting line of the code segment.

[0102] The candidate graph is uniformly represented as a structural unit index interval and a forced node set. Boundary sequences are then selected based on constraints including covering all title nodes, closing reference endpoints, and minimizing the number of cuts. For engineering implementation, either a restricted shortest path solver or an integer linear programming solver can be used. When multiple solutions are obtained, deterministic selection is performed based on the order of title levels and the original order of paragraph boundary sequences to ensure that the same input yields the same boundary sequence.

[0103] Define candidate boundary set Each candidate boundary Corresponding structural unit index cut point (occurring at) and (Between). Define binary decision variables. Indicates whether to select candidate boundaries .

[0104] The constraints must include at least: Coverage constraint: There must be at least one slice within the coverage area of ​​the header node, equivalent to the boundary having at least one lower bound; Mandatory node constraint: For mandatory node sequences... For a node in the referenced node, it must be ensured that it is not cut off by the boundary, by prohibiting the boundary from falling within the index interval between the two nodes; Monotonicity and minimum interval constraints: the selected boundary index is incremental, and there is at least one structure between the boundaries.

[0105] The objective function should at least include a weighted sum of reducing the number of cuts and reducing default risk:

[0106]

[0107] in The range of values ​​for the cost of segmentation is as follows. ; The boundary risk weight is constructed from the density of anomaly markers, table segments, code segments, and reference endpoints near the boundary, and its value range is [range missing]. .

[0108] When not using integer programming, candidate boundaries can be treated as graph nodes, feasible jumps between adjacent candidate boundaries can be treated as graph edges, edge weights can be defined as the aforementioned cost items, and the shortest path can be used to solve for the boundary sequence; this implementation is an equivalent example to integer programming when the objective is consistent.

[0109] When used, multi-source clues are unified into candidate graphs and forced nodes, and boundary generation and dependency closure are completed in the same solution process; structure-preserving nodes prevent tables and code segments from being cut off; when signals conflict, the constraint conditions provide a unique boundary sequence.

[0110] Slice overlap determines the shared context range of adjacent slices. Insufficient overlap will cut off cross-segment constraints, while excessive overlap will introduce repetition and occupy the context window. By utilizing keyword co-occurrence density and paragraph coherence drift signals, and incorporating structural instability into the same mapping relationship, the scope of repetition can be controlled while preserving dependencies.

[0111] Statistical keyword co-occurrence density The cross-segment dependency strength is obtained by combining the citation relationship graph, and then the keyword co-occurrence density is calculated. Paragraph coherence With structural clarity Mapped to slice overlap This will increase the keyword co-occurrence density. High, absolute value of paragraph coherence Low or structural clarity Low slice overlap is increased; keyword co-occurrence density is increased. The intersection-union ratio (CIU) of keyword sets within two adjacent structural units is used for calculation, and a moving average is applied within the title node to reflect persistent co-occurrence; when the citation completeness is... When a low-slice citation crosses a title node, an overlapping window is added around the citation endpoint to ensure that the source and target citation sentences fall within the shared area of ​​adjacent slices. Slice overlap. Defined as:

[0112]

[0113] Slice overlap : Value This is used to characterize the sharing ratio of adjacent slices; compression mapping function. : Value , Used to compress unbounded inputs; weighting coefficients : Value Used to adjust keyword co-occurrence density Contributions;

[0114] Weighting coefficient : Value This is used to adjust the contribution of the drift term; weighting coefficients. : Value This is used to adjust the contribution of structural instability terms; keyword co-occurrence density. : Value Used to characterize the degree of co-occurrence of keywords within adjacent structural units; paragraph coherence. : Value It is used to characterize the degree of semantic consistency between adjacent paragraphs. The smaller the value, the more pronounced the drift; structural clarity : Value Used to characterize the degree of structural stability. The smaller the value, the more pronounced the instability; natural logarithm function : Value Used to suppress marginal gain and improve numerical stability; natural constant : The range of values ​​is a constant, used for exponential calculations;

[0115] Specifically, the slice overlap is converted into the length of overlapping text and measured with the same caliber as the slice size, while a strategy explanation marker is output. Its value is limited to one of structure priority, semantic priority, and cost priority; when the structural support factor is in the high range and the structural clarity is in the high range, the output is structure priority; when the keyword co-occurrence density is in the high range and the paragraph coherence shows a continuous decline, the output is semantic priority; when the abnormal tags are concentrated or the computing budget is limited, the output is cost priority.

[0116] When used, overlap inference makes the shared range change with the dependency strength, reducing the breakage of cross-segment constraints; endpoint appending windows write cross-segment references into the shared area; policy interpretation flags solidify the inference reason into the output field, making it easy for step four to be executed with the same caliber.

[0117] Step 4: Scale the slices output in Step 3. Slicing method Slice overlap The boundary sequence is converted into a repeatable slice sequence, and boundary verification and fine-tuning, metadata encapsulation and writing are completed during the slice landing process, so that the knowledge base index can stably support the call to generate RAG for retrieval enhancement.

[0118] In engineering integration, slice execution often occurs in different processes or services than document upload, parsing, and topic calculation. If step four directly reads unverified strategy parameters, it can easily lead to missing fields, misaligned fields, or version rollbacks, which in turn can cause discrepancies between the boundary sequence and the structural unit sequence. Inconsistency. First, the executable strategy is transformed into a one-time verifiable input, thus avoiding untraceable divergences during the slice execution phase.

[0119] Step four: First, receive the strategy parameter frame and verify the document identifier field against the structure profile. The document identifiers must be consistent, and then the strategy version field must be checked against the structure profile. The generation timestamp correspondence is established, and then the field order and checksum of the strategy parameter frame are verified. If the verification fails, step four sends a retransmission request upstream and keeps the current document in a pending state until a valid strategy parameter frame is received. If the verification passes, step four generates a slice task description object, which carries the slice size. Slicing method Slice overlap Boundary sequence, forced node sequence, structural unit sequence Index range and content fingerprint And put the task description object into the task queue.

[0120] Content fingerprint Standardized text For input, normalization includes encoding standardization, control character removal, whitespace normalization, and line ending standardization; the normalized text will be... The content fingerprint is obtained by concatenating the characters in order and then performing SHA-256. .

[0121] In step four, an idempotent task key is used to avoid repeated execution. The idempotent task key is composed of a document identifier and a content fingerprint. The task queue is generated by concatenation; it uses a first-in-first-out scheduling method and writes four-state flags (pending, in execution, completed, and failed rollback) when a task is enqueued. The failed rollback state will trigger a one-time rollback strategy, which is fixed to the slicing method. Set as fixed-size slices, slice overlap Set to median, change boundary sequence to The accumulated boundaries ensure that the document can still enter the database entry process and will not remain in an unavailable state.

[0122] When in use, the policy parameter frame undergoes consistency verification before entering slice execution, ensuring that the boundary sequence is consistent with... The index range can be directly aligned; the idempotent task key ensures that duplicate reporting and retransmission will not generate duplicate slices; the failure fallback state ensures that slice execution can still maintain a complete path even when the upstream is abnormal.

[0123] Tables and code segments in a document typically carry key field names, column headers, error codes, or function names. If the boundaries fall in the middle of a table segment or the middle of a code segment's indentation, it can create gaps in the content and affect subsequent retrieval and location. By strictly adhering to the boundary sequence during slice execution and imposing preservation rules on structural unit types, slices can satisfy both the repeatability of the boundary sequence and the readability of the structural units.

[0124] Step 4: Read the slice method from the task description object ,when When slicing the structure, step four maps the boundary sequence to a title hierarchy diagram. The coverage area, and within each heading coverage area, a sequence of paragraph boundaries. As a secondary boundary; when When performing semantic slicing, step four extracts the structural unit index intervals according to the boundary sequence, and appends the structural units pointed to by the forced node sequence to both ends of each interval, so that the reference endpoints and the header line fall into the same slice; when When slicing to a fixed size, step four follows... For structural unit sequence The length is accumulated and truncated, and a structure-preserving convergence is performed on any interval containing a table segment or a code segment, which moves the boundary to the nearest paragraph boundary before the start line of the table segment or the start line of the code segment.

[0125] Structure Preservation Closure: If the boundary enters the interior of a table or code segment, the boundary moves forward to the nearest paragraph boundary before the starting structural unit of that table or code segment. If no nearest paragraph boundary exists, it moves to the starting boundary of the heading node; otherwise, it moves to the starting boundary of the document.

[0126] In step four, the overlapping segments are analyzed using slice overlap. The calculated overlapping text length is used for truncation, prioritizing the paragraph boundary sequences on both sides of the boundary. Select consecutive paragraphs as overlapping areas to avoid truncating sentences in the middle; when referring to relationship diagrams When a reference endpoint is located near a boundary, step four adds the paragraph containing the reference endpoint and its target paragraph to the overlap region candidate, expands the paragraph coverage of the overlap region without changing the boundary sequence, and then writes the expanded overlap region into the overlap field of the slice object.

[0127] When used, slice execution preserves the structure at table segments and code segments, reducing fragment gaps; overlapping splicing fragments are obtained from paragraph boundary alignment, reducing intra-sentence truncation; after reference endpoints are included in the overlapping field, it is easier to form a readable context during subsequent retrieval and splicing.

[0128] Boundary sequences originate from the inference stage, and situations may still arise where boundaries are too close or cross dependency chains at the paragraph progression point. Without verification, sliced ​​objects may exhibit local fragments that are difficult to understand independently after being stored in the database. The checks on whether boundaries fall into stable positions and whether there are semantic breaks on either side of the boundaries are transformed into executable verifications, thereby enabling boundary fine-tuning without altering the core strategy.

[0129] Step 4: Extract the left and right windows of each boundary location, align the windows according to the paragraph boundary sequence, and calculate the paragraph vector. and When the window similarity is low and the reference relationship graph is near the boundary. When the endpoint is near the referenced point, step four prioritizes dependency closure fine-tuning, moving the boundary to the nearest paragraph boundary after the referenced endpoint, ensuring that the source and target paragraphs are not separated; when the window similarity is low and the referenced endpoint is near the boundary, step four performs steady-state backtracking fine-tuning, moving the boundary to paragraph coherence. The nearest paragraph boundary near the local minimum location makes the boundary close to the natural drift point; when anomaly markers... When indicating undecidable structural units near the boundary, step four freezes the boundary and expands the context coverage only through overlapping fields to avoid index instability caused by repeatedly moving the boundary in undecidable regions. Window similarity is calculated in cosine form.

[0130]

[0131] Where: paragraph vector Dimensional range Its function is to represent the semantic direction of the window to the left of the boundary; paragraph vector Dimensional range and Consistency, serving to characterize the semantic direction of the window to the right of the boundary; inner product operation The range of values ​​is determined by the vector norm, which measures the consistency of semantic direction; vector norm : Range of values Its function is to scale the paragraph vectors; vector norm : Range of values Its function is to perform scale normalization on the paragraph vector;

[0132] Step four involves performing boundary sequence monotonicity and minimum interval checks after boundary fine-tuning. Monotonicity requires that each side of the boundary index adds one structural unit, and minimum interval requires that adjacent boundaries add at least one structural unit. Boundaries that do not meet the minimum interval requirement after fine-tuning will be rolled back to the boundary before fine-tuning and compensated by expanding the overlapping field.

[0133] Closed fine-tuning: When the windows on both sides of the boundary are similar and reference relationship diagrams When a reference edge exists that crosses the boundary, the boundary is prohibited from falling within the index region between the source and target structural units. The boundary is moved to the nearest paragraph boundary after the target structural unit. If the minimum interval is not met after the move, the overlapping field is rolled back and expanded to cover both the source and target references.

[0134] Steady-state backoff fine-tuning: If no reference edges are involved, then search for paragraph coherence near the boundaries. The boundary is moved to the paragraph boundary corresponding to the local minimum position. If the move results in the adjacent slice length being less than the minimum structural unit number, the overlapping field is rolled back and expanded. In use, boundary checks link similarity, reference closure, and steady-state rollback into a single chain constraint, making fine-tuning actions interpretable and rollbackable; freezing undeterminable regions avoids boundary jitter; monotonicity and minimum interval checks ensure that slice objects can be stably read by subsequent indexes.

[0135] Furthermore, if sliced ​​objects lack structural location, topic, and keyword tags, subsequent retrieval can only rely on vector similarity, which is prone to recall drift in multi-topic documents; at the same time, incremental updates without content fingerprints... The alignment mechanism can cause old slice remnants to be mixed with new slices.

[0136] The slice object is encapsulated as a content + position + tag + version, and written to the index using a commit protocol to ensure consistency and rollback capability when entering data into the database.

[0137] Step four generates a metadata set for each slice object. The metadata set includes document identifier, slice number, boundary sequence position, path of the title node to which it belongs, keyword tags, topic tags, and strategy explanation tags. Content fingerprint The strategy version field and generation timestamp are then used; subsequently, embedding vector calculations are performed on the slice object text and written to the vector index, while the metadata set is written to the key-value index and an inverted index is established using the document identifier as the primary key; when the content fingerprint... If there is a discrepancy with previous versions, step four should first be performed by pressing the content fingerprint. Locate the affected structural unit index range, delete only the old slice index corresponding to that range, and then write the new slice index to avoid a full rebuild.

[0138] Vectorized writing employs a two-phase commit: the first phase writes to the staging area and returns a staging handle; the second phase commits to the formal area after verifying that the number of slices matches the number of boundaries, that metadata fields are complete, and that the policy version field matches. If the second phase verification fails, the staging handle is discarded and the old version index is retained. The engineering implementation can use a vector database that supports transactions, or save staging slice files in object storage and drive index updates with a commit list.

[0139] Phase 1: Write the slice object's text, vectors, and metadata to the temporary storage area and return the temporary storage handle. The temporary storage area uses document identifiers and content fingerprints. Use namespaces to isolate different versions.

[0140] Phase Two: Perform consistency checks on the staging area. Check conditions include at least the following: the number of slices matches the number of boundaries; each slice's metadata fields are complete; the policy version field of the frame is consistent; and the content fingerprint is valid. Consistent. If the verification passes, the handle will be temporarily stored. The corresponding data atomicity is switched to be visible in the official area; if the verification fails, it is discarded. The old version in the official area will remain unchanged.

[0141] In use, metadata encapsulation enables slice objects to have location traceability and tag-based filtering; two-phase commit allows write failures to be rolled back without destroying old indexes; content fingerprinting... Supports incremental replacement and avoids version mixing.

[0142] As a supplement: TextRank: Using the terms after text segmentation into structural units as nodes, constructing edges with co-occurrence windows, iteratively calculating term weights, and selecting the top few terms as keywords; damping coefficient. Range of values The upper limit of the number of iterations is within a certain range. The range of convergence threshold values MMR (for document summarization): Cosine similarity is used to assess the relevance between candidate sentence vectors and document summarization target vectors, while redundancy is suppressed using the similarity between candidate sentences; a balance coefficient is also used. Range of values .

[0143] LDA: Input is a bag-of-words representation of continuous segments, output is a topic-word distribution and a segment-topic distribution; the candidate set of topic numbers can be set as a finite set and selected by perplexity or consistency index; when the text is too short to be stably trained, it falls back to the clustering topic model. Clustering topic model: Input is a continuous segment vector, and topic clusters are obtained by hierarchical clustering or density clustering; the neighborhood radius and minimum number of samples of density clustering are set as finite candidates and selected by cluster stability; when the number of clusters is 1, it is considered to be in the topic set.

[0144] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0145] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0147] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for optimizing the dynamic document slicing strategy of a RAG system based on multi-dimensional analysis, which slices documents and writes them into a knowledge base or indexing system, characterized by: include, The document is parsed and standardized, and a sequence of structural units is formed according to structural units such as headings and paragraphs. The structural profile data is obtained by extracting heading levels, paragraph boundaries, paragraph lengths and paragraph coherence. Summarized text is generated based on structural unit sequences. Topic modeling is performed to obtain a topic set and topic complexity index. Keyword set is extracted and keyword co-occurrence information is constructed to obtain content profile data. A set of slicing strategy parameters is generated based on structural profile data and content profile data. The set of slicing strategy parameters includes slice size, slicing method and slice overlap, and is encapsulated as a strategy parameter frame. Based on the strategy parameter frame, a slice sequence is generated. Candidate boundaries are verified and adjusted based on semantic similarity. Slice metadata is generated and the slice sequence and slice metadata are written into a knowledge base or indexing system.

2. The method for optimizing the dynamic document slicing strategy of a RAG system according to claim 1, characterized in that: The document undergoes unified encoding, whitespace normalization, and punctuation standardization. Meaningless control characters are removed, and structural units are identified according to headings, paragraphs, list items, table segments, and code segments. Type and level tags are generated for each structural unit, and the start and end positions are recorded to form a sequence of structural units.

3. The method for optimizing the dynamic document slicing strategy of a RAG system according to claim 2, characterized in that: The structural profile data is extracted from the structural unit sequence and includes heading hierarchy relationships, paragraph boundary distribution, paragraph length distribution, sentence break density, sequential continuity, and citation and numbering integrity. It also performs consistency verification on cross-page line breaks and numbering jumps, and establishes a correspondence between the structural profile data and the structural unit index.

4. The method for optimizing the dynamic document slicing strategy of a RAG system according to claim 3, characterized in that: The document is read in overview form based on the structural unit sequence. Candidate sentence segments are organized according to the heading level while maintaining the original order. Representative sentence segments are selected to generate summary text. The summary text and heading level path are recorded together. The summary text is split into summary fragments according to the structural unit index and incorporated into the content profile data.

5. The method for optimizing the dynamic document slicing strategy of a RAG system according to claim 3, characterized in that: A set of topics is obtained by using continuous structural unit fragments as input for topic modeling, and a topic complexity index is generated. The topic complexity index includes the number of topics and the degree of dispersion of topic distribution. The topic set and topic distribution results are mapped back to the structural unit index as a component of the content profile data.

6. The method for optimizing the dynamic document slicing strategy of a RAG system according to claim 1, characterized in that: Extract a set of keywords from the structural unit sequence. The set of keywords includes paragraph keywords, column header and row header keywords of table segments, and function name and variable name keywords of code segments. Construct keyword co-occurrence information based on co-occurrence statistics of adjacent structural units and within the same structural unit, and store the keyword co-occurrence information in correspondence with the structural unit index.

7. The method for optimizing the dynamic document slicing strategy of a RAG system according to claim 1, characterized in that: The slicing method is determined based on the structural profile data and the content profile data. The slicing method is selected from fixed-size slices, semantic slices, and structural slices. The evaluation results of the slicing method, structural clarity, and paragraph-level coherence are written into the strategy parameter frame.

8. The method for optimizing the dynamic document slicing strategy of a RAG system according to claim 6, characterized in that: Based on the abstract text and topic complexity index, combined with paragraph length distribution, a preset threshold level for slice size is generated, and a lag switching rule is set for threshold level switching. Furthermore, the slice overlap is determined based on keyword co-occurrence information and paragraph-level coherence, and the slice size and slice overlap are written into the strategy parameter frame.

9. The method for optimizing the dynamic document slicing strategy of a RAG system according to claim 1, characterized in that: After generating the slice sequence based on the strategy parameter frame, a boundary window is constructed on both sides of the candidate boundary and semantic similarity is calculated. The vector used for semantic similarity is generated by the same embedding model as the retrieval stage. Candidate boundaries below a preset threshold are adjusted to adjacent paragraph boundaries and the boundary index is updated synchronously, while ensuring that the structural units of table segments and code segments are not split by boundaries.

10. The method for optimizing the dynamic document slicing strategy of a RAG system according to claim 9, characterized in that: Generate slice metadata for each slice, which includes document identifier, slice number, title level path, structural unit location, keyword tags and topic tags; Before writing to the knowledge base or indexing system, the strategy parameter frame is checked for consistency, and then the slice sequence and slice metadata are written to the knowledge base or indexing system in a version replacement manner.

Images