A rapid assessment method of RAG system based on pipeline architecture

Abstract

This invention relates to the field of semantic retrieval technology, specifically a rapid evaluation method for RAG systems based on a pipelined architecture. The specific implementation process includes: receiving the original regulatory document stream and deconstructing it into a grammatical evolution queue; deploying intercept probes to collect logical anchor data; importing the pipelined architecture and introducing semantic perturbation analysis to simulate logical offsets caused by upstream changes and constructing defensive feature values ​​for each node; quantifying the impact of upstream defects on downstream judgments, generating a risk transmission weight spectrum and characterizing cascading patterns; identifying sensitive source nodes causing cascading offsets, locating system performance bottlenecks, and generating a pipelined evaluation report. This invention, by introducing semantic perturbation analysis and risk transmission spectrum, achieves rapid evaluation of multi-node cascading error propagation in RAG systems, enabling the location of performance bottlenecks and providing targeted optimization paths, thereby improving the evaluation efficiency and tuning capabilities of RAG systems.

Description

Technical Field

[0001] This invention relates to the field of semantic retrieval technology, specifically to a rapid evaluation method for RAG systems based on a pipelined architecture. Background Technology

[0002] In the field of information retrieval and data processing, Retrieval Augmented Generation (RAG) technology is a mainstream information processing architecture that combines information retrieval modules with large language model generation modules. A typical RAG system usually operates as an information processing pipeline, containing multiple data processing components, such as document preprocessing and chunking to structure the original document, vectorization encoding to convert text into high-dimensional vectors, retrieval recall based on semantic similarity to retrieve candidate documents, reordering of candidate results for secondary refinement, and context-based generation of large language model answers. To measure the output quality of such complex systems, current industry practices generally use automated evaluation frameworks to benchmark the systems. Most of these existing technologies calculate macro-level scalar scores such as fidelity or relevance for the final text generated by the system, or calculate isolated performance metrics for the retrieval and generation components separately.

[0003] Because the RAG system is a complex pipeline comprising multiple cascaded upstream and downstream components such as partitioning, retrieval, rearrangement, and generation, the retrieval phase integrates sparse retrieval based on BM25 and dense retrieval based on vectors; the generation phase inputs the user question and retrieval results into a large language model to obtain a response. However, the above process suffers from severe efficiency bottlenecks: on the one hand, as the size of the knowledge base file increases, the retrieval time of the BM25 algorithm increases dramatically; on the other hand, the generation phase relies on the large language model for reasoning, requiring at least two calls to the large language model for each query, resulting in huge computational resource overhead. There is also significant idle waiting between stages, leading to low hardware resource utilization and making a complete evaluation typically too time-consuming, severely restricting the iterative efficiency of algorithm development. In summary, the existing RAG system requires a large amount of evaluation test data. To ensure the algorithm is fully validated before deployment, developers need to collect a large amount of test data for the RAG knowledge base and prepare a large number of query cases, resulting in a heavy evaluation workload. Since the evaluation process for each query requires at least two calls to the large language model, resource consumption is high, and hardware resource overhead is extremely high. The evaluation stages are executed sequentially, resulting in low overall system throughput and excessively long evaluation time, which greatly reduces the efficiency of algorithm development and iteration.

[0004] To address this, a rapid evaluation method for RAG systems based on a pipelined architecture is proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a rapid evaluation method for RAG systems based on a pipelined architecture, enabling rapid evaluation of RAG systems.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A rapid evaluation method for RAG systems based on a pipelined architecture includes: The system receives the original regulatory document stream and deconstructs it into a word order evolution queue within the RAG system based on contextual logical dependencies. Interception probes are deployed at each node of the word order evolution queue for retrieval, reordering, and answer generation to capture logical anchor data containing retrieval item time sequence labels, reordering ranking weights, and generated text fragments. The logical anchor data is imported into a parallel pipeline architecture, and semantic perturbation analysis is introduced for different nodes. The semantic perturbation analysis is used to simulate the logical offset caused by upstream changes, and the hit deviation of the retrieval recall node, the order fluctuation of the reordering node, and the consistency score of the answer generation node are calculated to obtain the defense feature value of each independent node against logical deviation. Logical testing is performed based on the defense feature values ​​to quantify the risk transmission weight spectrum of the impact of upstream node defects on downstream node judgment, and to obtain the cascading pattern that changes with the depth of the queue as the word order evolves. Based on the risk transmission weight spectrum, sensitive sources that cause the cascade pattern to shift are identified, which are determined to be the performance bottlenecks of the RAG system, and a pipeline evaluation report containing bottleneck node coordinates, risk diffusion quantification curves, and targeted suggestions is generated.

[0007] Preferably, the specific implementation process of receiving the original regulatory document stream and deconstructing it into a phrasal evolution queue within the RAG system according to contextual logical dependencies includes: The received raw regulatory document stream is subjected to structured preprocessing. Cross-sentence logical dependencies are extracted based on paragraph boundary identifiers and semantic connectors, and a directed dependency graph is constructed. Based on the topological sorting of nodes in the directed dependency graph, the raw document stream is divided into a sequence of text segments. The text segment sequence is assigned a context position index label and arranged according to the semantic evolution direction to generate a word order evolution queue carrying position index labels.

[0008] Preferably, the specific implementation process of deploying interception probes at each node of the retrieval, reordering, and answer generation in the word order evolution queue to capture logical anchor data containing retrieval item time sequence labels, reordering ranking weights, and generated text fragments includes: Interception probes are injected into the retrieval and recall nodes to capture the vector similarity scores and corresponding temporal tags of each candidate retrieval item, forming the temporal tags of the retrieval items; interception probes are injected into the reordering nodes to capture the position weight vectors of each candidate item output by the reordering model, forming the reordering position weights; interception probes are injected into the answer generation nodes to capture the generated text fragments and corresponding source citation identifiers token by token during the large language model decoding process, forming the generated text fragments; the temporal tags of the retrieval items, the reordering position weights, and the generated text fragments are aligned by timestamps and aggregated into logical anchor data, which is then output to the data entry point of the parallel pipeline architecture.

[0009] Preferably, the specific implementation process of importing the logical anchor data into a parallel pipeline architecture and introducing semantic perturbation analysis for different nodes includes: The logical anchor data is routed to the corresponding independent analysis channels in the parallel pipeline architecture according to the retrieval recall layer, re-ranking layer, and answer generation layer. In the retrieval recall channel, random perturbations are applied to the semantic vectors of the retrieval entries to generate perturbed versions of the recall candidate set. In the re-ranking channel, gradient permutation perturbations are applied to the position weight vectors to simulate parameter drift in the re-ranking model. In the answer generation channel, local semantic substitution perturbations are applied to the input context window to simulate the interference of upstream input changes on the generation process.

[0010] Preferably, the specific implementation process of simulating the logical offset caused by upstream changes through semantic perturbation analysis, calculating the hit deviation of the retrieval recall node, the order fluctuation of the reordering node, and the consistency score of the answer generation node, and obtaining the defense feature value of each independent node against logical deviation includes: For retrieval and recall nodes, the Jaccard distance between the recall set before and after perturbation is used as the hit deviation to measure the node's sensitivity to semantic perturbation. For reordering nodes, the absolute value of the difference between the Kendall rank correlation coefficients between the order sequences before and after perturbation is used as the order fluctuation to measure the reordering stability. For answer generation nodes, a comparative evaluation based on semantic embedding cosine similarity is used to calculate the consistency score of the generated text in the semantic space before and after perturbation. The hit deviation, order fluctuation, and consistency score are normalized and then concatenated to form the defensive feature value of each node.

[0011] Preferably, the specific implementation process of performing logical tests based on the defense feature values ​​to quantify the risk transmission weight spectrum of the impact of upstream node defects on downstream node judgments, and obtaining the cascading pattern that changes with the depth of the phrasing evolution queue, includes: Using the defense characteristics of each node as independent variables, a directed acyclic graph structure for causal transmission between nodes is constructed. The path coefficients of the error propagation path from the retrieval recall node defects to the reordering node and the answer generation node are estimated using a causal inference framework, yielding the risk transmission weights of each directed edge. These risk transmission weights are grouped and statistically analyzed according to the depth index of the word order evolution queue, and a function curve of the transmission weights changing with queue depth is fitted to identify the monotonically increasing, decreasing, and oscillating hierarchical patterns of the transmission weights with depth. Finally, the risk transmission weights of each path are summarized to generate a risk transmission weight spectrum between nodes.

[0012] Preferably, the specific implementation process of identifying the sensitive source causing the shift in the cascading pattern based on the risk transmission weight spectrum, determining it as the performance bottleneck of the RAG system, and generating a pipeline evaluation report containing bottleneck node coordinates, risk diffusion quantification curves, and targeted suggestions includes: Sensitivity ranking is performed on the risk transmission weight spectrum. Source nodes whose cumulative outgoing edge weights of each directed path exceed a preset threshold are marked as sensitive sources and located as performance bottleneck nodes in the RAG system pipeline. The coordinate information of the bottleneck nodes in the sequence evolution queue is extracted and written into the evaluation report as the bottleneck node coordinates. Based on the changes in the weights of each path in the risk transmission weight spectrum with the depth of the queue, a risk diffusion quantification curve is plotted to characterize the magnitude and direction of error propagation along the pipeline. Combining the type of the bottleneck node and the corresponding defense feature value, the optimization strategy knowledge base is matched to generate targeted suggestions including component replacement suggestions, hyperparameter adjustment directions, and data augmentation schemes. A complete pipeline evaluation report is then summarized and output.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention achieves fine-grained dynamic observation of the entire pipeline process by deploying intercept probes and constructing logical anchor data at key nodes of the RAG system (retrieval recall, reordering, and answer generation). It can fully characterize the intermediate processing states and their evolutionary relationships, thereby improving the interpretability and traceability of the system evaluation.

[0014] 2. This invention introduces a semantic perturbation-based analysis mechanism, which applies controlled perturbations to different nodes and constructs defensive feature values. This can effectively simulate the impact of upstream changes on downstream, realize the quantitative characterization of multi-component cascade error propagation, improve the shortcomings of not being able to analyze upstream and downstream dependencies and error amplification effects, thereby improving the accuracy and diagnostic depth of the evaluation results.

[0015] 3. By constructing a risk transmission weight spectrum and identifying sensitive source nodes, this invention enables automatic location of performance bottlenecks in RAG systems and quantitative analysis of risk diffusion paths. It can not only reveal cascading patterns but also generate targeted optimization suggestions, improving system tuning efficiency and decision-making pertinence, and overcoming the difficulty in accurately locating the root causes of problems. Attached Figure Description

[0016] Figure 1 This is a flowchart of a rapid evaluation method for RAG systems based on a pipelined architecture proposed in this invention. Figure 2 This is the parallel timing diagram of the RAG pipeline proposed in this invention; Figure 3 This is a schematic diagram of the dynamic allocation scheme for GPU hardware resources proposed in this invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It must be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to constitute any limitation on the scope of protection of this invention. Therefore, all equivalent changes or modifications conceived by those skilled in the art based on the content disclosed in this invention without inventive effort should fall within the scope of protection claimed by this invention.

[0018] Reference Figures 1 to 3 This invention provides a rapid evaluation method for RAG systems based on a pipelined architecture, the technical solution of which is as follows:

[0019] Example 1: Reference Figure 1 This embodiment proposes a rapid evaluation method for RAG systems based on a pipelined architecture, including: The system receives the original regulatory document stream and deconstructs it into a word order evolution queue within the RAG system based on contextual logical dependencies. Interception probes are deployed at each node of the word order evolution queue for retrieval, reordering, and answer generation to capture logical anchor data containing retrieval item time sequence labels, reordering ranking weights, and generated text fragments. The logical anchor data is imported into a parallel pipeline architecture, and semantic perturbation analysis is introduced for different nodes. The semantic perturbation analysis is used to simulate the logical offset caused by upstream changes, and the hit deviation of the retrieval recall node, the order fluctuation of the reordering node, and the consistency score of the answer generation node are calculated to obtain the defense feature value of each independent node against logical deviation. Logical testing is performed based on the defense feature values ​​to quantify the risk transmission weight spectrum of the impact of upstream node defects on downstream node judgment, and to obtain the cascading pattern that changes with the depth of the queue as the word order evolves. Based on the risk transmission weight spectrum, sensitive sources that cause the cascade pattern to shift are identified, which are determined to be the performance bottlenecks of the RAG system, and a pipeline evaluation report containing bottleneck node coordinates, risk diffusion quantification curves, and targeted suggestions is generated.

[0020] Furthermore, the specific implementation process of receiving the original regulatory document stream and deconstructing it into a phrasal evolution queue within the RAG system based on contextual logical dependencies includes: The received raw regulatory document stream is subjected to structured preprocessing. Cross-sentence logical dependencies are extracted based on paragraph boundary identifiers and semantic connectors, and a directed dependency graph is constructed. Based on the topological sorting of nodes in the directed dependency graph, the raw document stream is divided into a sequence of text segments. The text segment sequence is assigned a context position index label and arranged according to the semantic evolution direction to generate a word order evolution queue carrying position index labels.

[0021] Specifically, upon receiving the original regulatory document stream, the system first performs structured preprocessing on the document stream. In this embodiment, the document stream is specifically a financial regulatory document stream. The structured preprocessing takes the original formatting information and text tags of the document as input, and specifically identifies paragraph boundary identifiers carried in the document, including but not limited to chapter number markers, clause number markers (such as "Article X", "XXX" format hierarchical numbering), paragraph indentation features, and the frequency threshold of consecutive line breaks (usually, two or more consecutive line breaks are used as candidate signals for paragraph boundaries). At the same time, the system scans the document text sentence by sentence, identifying semantic connectors with semantic connection functions, including connectors indicating conditional relationships (such as "if", "any", "in the case of"), connectors indicating causal relationships (such as "therefore", "leading to"), connectors indicating parallel and supplementary relationships (such as "in addition", "at the same time", "another"), and connectors indicating adversative and limiting relationships (such as "but", "unless", "must not"). Through the presence and direction of the above connectors, the logical dependency polarity and dependency direction between adjacent or cross-segment sentences are inferred.

[0022] Based on the paragraph boundary identifier detection results and cross-sentence semantic connector analysis results, the system extracts all cross-sentence logical dependencies in the document and constructs a directed dependency graph using these dependencies as directed edges and each semantic clause or normative statement unit as a node. In the directed dependency graph, each directed edge points from an upstream logical node (condition or triggering rule) to a downstream logical node (obligation or execution rule), and the weight of the edge is determined by the type strength of the dependency word. Taking a certain clause of a banking supervision and management regulation as an example: if the document contains the clause "Financial institutions shall submit a rectification report within 15 days from the date of receiving the regulatory instruction; if they fail to submit it within the time limit, the regulatory department has the right to impose administrative penalties on them," the system can identify that there is a conditional dependency relationship between the obligation clause "submit a rectification report within 15 days" and the consequence clause "if they fail to submit it within the time limit, administrative penalties will be imposed," and constructs a directed edge from the former to the latter in the directed dependency graph, rather than separating the two clauses to independent nodes that are unrelated to each other.

[0023] After constructing the directed dependency graph, the system segments the original document stream into a sequence of text fragments based on the topological sorting of the nodes in the graph. The topological sorting employs a breadth-first traversal strategy, starting from the root node with an in-degree of 0 and expanding layer by layer according to the dependency direction, ensuring that the node set at each level includes all its predecessor nodes. During the topological sorting process, if a directed cycle (i.e., circular dependency, typically found between mutually referencing clauses in financial regulatory texts) is detected in the directed dependency graph, the system automatically performs virtual chain breaking: the directed edge with the weakest semantic dependency strength in the cycle is selected and deleted, and the break point is recorded in the metadata for subsequent reconstruction analysis during the evaluation phase. Based on the topological sorting order, the system merges nodes continuously distributed at the same topological level that meet the semantic coherence threshold into a single text fragment, and divides nodes spanning different topological levels into independent text fragments, thereby segmenting the original document stream into a set of ordered text fragment sequences. The semantic coherence threshold is determined as follows: At least 100 pairs of adjacent sentences are randomly selected from the target document corpus to form a positive sample set (determined as semantically coherent), and 100 pairs of sentences spanning paragraphs are selected to form a negative sample set (determined as semantically incoherent). The cosine similarity of each pair of sentences under the target embedding model is calculated. To maximize the classification accuracy of positive and negative samples, a grid search is performed within the range of 0.60 to 0.85 with a step size of 0.05, and the threshold corresponding to the highest classification accuracy is selected. In this embodiment, the target document corpus consists of regulatory guidelines and operating procedures publicly released by the State Financial Regulatory Commission. The semantic embedding model used is text2vec-base-chinese (embedding dimension 768). If the embedding model is changed, the above calibration process should be repeated.

[0024] After acquiring the text segment sequence, the system assigns context position index labels to the text segment sequence. The context position index labels are encoded using a triple structure, recording the physical start and end character offsets of the text segment in the original document, the level number of the text segment in the directed dependency graph topological sorting, and the horizontal sequence number of the text segment within the same level. These three pieces of information together constitute the globally unique position identifier of the segment. Subsequently, the system arranges the text segments labeled with index labels according to the semantic evolution direction: firstly, they are arranged in ascending order by topological level number to ensure that logically upstream segments always precede logically downstream segments; within the same level, they are arranged in ascending order by horizontal sequence number to maintain the original document order of parallel terms at the same level. After the above arrangement operation, each text segment forms a strictly ordered queue structure according to its logical semantic evolution path, i.e., generating a word order evolution queue carrying position index labels. The word order evolution queue is output in the form of structured data objects. Each queue element contains a text content field, a triple position index field, and a directed dependency graph parent node reference field. The total length of the queue is consistent with the number of text segments.

[0025] This embodiment introduces a directed dependency graph topology sorting mechanism to drive text segmentation and position indexing, thereby achieving structured capture of the inherent semantic evolution path of financial regulatory documents. This ensures that each text fragment carries a complete logical semantic unit, thus eliminating the accumulation of cross-node errors caused by mismatch in segmentation granularity.

[0026] Furthermore, the specific implementation process of deploying interception probes at each node of the retrieval, reordering, and answer generation in the word order evolution queue to capture logical anchor data containing retrieval item time sequence labels, reordering ranking weights, and generated text fragments includes: Interception probes are injected into the retrieval and recall nodes to capture the vector similarity scores and corresponding temporal tags of each candidate retrieval item, forming the temporal tags of the retrieval items; interception probes are injected into the reordering nodes to capture the position weight vectors of each candidate item output by the reordering model, forming the reordering position weights; interception probes are injected into the answer generation nodes to capture the generated text fragments and corresponding source citation identifiers token by token during the large language model decoding process, forming the generated text fragments; the temporal tags of the retrieval items, the reordering position weights, and the generated text fragments are aligned by timestamps and aggregated into logical anchor data, which is then output to the data entry point of the parallel pipeline architecture.

[0027] Specifically, during the intercept probe injection phase of the retrieval and recall node, the system embeds the probe logic between the similarity calculation module and the result return interface of the vector retrieval engine in the form of a hook function, achieving lossless interception of the normal retrieval process. After the retrieval engine completes the vectorization encoding of each text segment in the word order evolution queue and performs nearest neighbor retrieval, the intercept probe reads the complete information of the candidate retrieval entries from the retrieval engine's internal result buffer. This includes the document identifier, document content summary, cosine similarity score between its semantic vector and the query vector, and the system millisecond-level timestamp returned by the retrieval engine for each candidate entry. The cosine similarity score is stored with 32-bit floating-point precision, ranging from [-1, 1]. A higher score indicates a stronger semantic relevance between the candidate entry and the query text. The candidate entries are sorted in descending order of their cosine similarity scores and encapsulated as retrieval entry time-series labels. The structure is an ordered list, with each list element containing a candidate entry identifier field, a similarity score field, and a time-series timestamp field.

[0028] During the intercept probe injection phase of the reordering node, the system embeds the probe logic into the inference output interface of the reordering model using the same hook function mechanism. Data capture is performed after the reordering model completes the fine-grained ranking and scoring of the retrieved candidate set and before returning the reordering results downstream. The intercept probe reads the relevance score vector of each candidate item calculated by the model from the inference output tensor of the reordering model. Each component in the relevance score vector corresponds to a candidate item, and its value represents the quantitative estimate of the matching degree between the candidate item and the query text by the reordering model. The value is usually within the range of 0 to 1 after Softmax normalization. The probe module rearranges the candidate items in descending order of relevance score, and simultaneously records the change in order of each candidate item before and after reordering (i.e., the signed integer difference between the original position before reordering and the new position after reordering), as well as the system's millisecond-level timestamp for the completion of the reordering inference process. This information is encapsulated into reordering position weights. The reordering weights are stored in the form of a list of structured objects. Each list element contains a candidate entry identifier field, a pre-reordering position field, a post-reordering position field, a reordering relevance score field, and a time-series timestamp field.

[0029] During the interception probe injection phase at the answer generation node, the system embeds probe logic into the token-by-token output interface of the large language model decoding engine to achieve step-by-step tracking of the generation process. The interception probe triggers a data collection operation every time the large language model generates an output token, recording the text content of the currently generated token, the token's position in the generation sequence, the input context fragment actually referenced by the model for generating the token (filtered by an attention weight threshold, selecting context windows with attention weights exceeding 0.15), its corresponding source citation identifier (i.e., the position index label of the element in the word order evolution queue to which the context fragment belongs), and the system's millisecond-level timestamp indicating that the token generation is complete. The source citation identifier is consistent with the triple position index in the word order evolution queue, ensuring that each token in the generated text can be traced back to a specific regulatory document fragment source. The probe module concatenates the above information collected token-by-token in the order of the generation sequence to form a fragmented data structure of the generated text.

[0030] After the probe data collection at the three nodes is completed, the system performs a timestamp alignment operation on the multiple data streams. The timestamp alignment uses the unique identifier of the query request as the primary key, identifying the time sequence labels, reordering weights, and generated text fragments belonging to the same query request as grouped data. Based on the system millisecond-level timestamps recorded in the data collected by each node, the collected records in the three data streams are arranged in ascending time sequence to verify the causal time sequence constraint that the retrieval recall probe collection time is earlier than the reordering probe collection time, and the reordering probe collection time is earlier than the answer generation probe collection time. If the timestamp order of any two data streams violates the above constraint (usually caused by system clock drift), the system automatically replaces the physical timestamps with the logical processing order of each node for alignment correction and marks the correction in the data metadata. After alignment, the system aggregates the retrieval entry time sequence labels, reordering weights, and generated text fragments into a unified logical anchor data object. This logical anchor data object contains three sub-fields, corresponding to the three types of probe data, and includes metadata fields such as the query request identifier, alignment correction mark, and aggregation completion timestamp. The logical anchor data is then standardized and serialized, and output to the data entry point of the parallel pipeline architecture in the form of a message queue, so that the subsequent semantic perturbation analysis module can read it according to the retrieval recall layer, reordering layer and answer generation layer respectively.

[0031] This embodiment deploys interception probes to inject a fine-grained data capture mechanism into each key node of the RAG pipeline in a non-intrusive manner. It collects intermediate state quantities of each node's processing process and completes the alignment and convergence of multiple data streams based on timestamps. This transforms the system evaluation mode from final state observation to full-process tracking, achieving dynamic monitoring of the entire pipeline process without affecting normal system services. This provides a complete and reliable data foundation for subsequent cascaded error propagation analysis.

[0032] Furthermore, the specific implementation process of importing the logical anchor data into the parallel pipeline architecture and introducing semantic perturbation analysis for different nodes includes: The logical anchor data is routed to the corresponding independent analysis channels in the parallel pipeline architecture according to the retrieval recall layer, re-ranking layer, and answer generation layer. In the retrieval recall channel, random perturbations are applied to the semantic vectors of the retrieval entries to generate perturbed versions of the recall candidate set. In the re-ranking channel, gradient permutation perturbations are applied to the position weight vectors to simulate parameter drift in the re-ranking model. In the answer generation channel, local semantic substitution perturbations are applied to the input context window to simulate the interference of upstream input changes on the generation process.

[0033] Reference Figure 2 Specifically, after the logical anchor data is aggregated and serialized by the probe module and output to the data entry point of the parallel pipeline architecture, the system, according to the scheme described in this claim, first performs hierarchical routing on the logical anchor data, and then performs differentiated semantic perturbation on the data of the corresponding nodes in each independent analysis channel to generate perturbation version intermediate state data for subsequent defense feature value calculation.

[0034] In the hierarchical routing phase of the logical anchor data, the routing scheduling module of the parallel pipeline architecture reads the sub-field type identifier in each logical anchor data object. It distributes the sub-field data carrying the time sequence label of the retrieval entry to the retrieval recall independent analysis channel, the sub-field data carrying the reordering weight to the re-sorting independent analysis channel, and the sub-field data carrying the generated text fragments to the answer generation independent analysis channel. These three data streams are simultaneously written to the input buffer queue of their respective channels in an asynchronous concurrent manner. At the hardware level, these three independent analysis channels correspond to three mutually isolated computing thread pools. Each thread pool exclusively uses its allocated memory area and GPU computing resources, and the channels do not share any runtime state, thus ensuring the independence of perturbation experiments and the repeatability of results within each channel. The parallel pipeline architecture employs a pipelined parallel scheduling strategy, enabling the processing tasks of the three nodes—retrieval and recall, reordering, and answer generation—to overlap in time: while the system is processing the answer generation task for the Nth batch of queries, the reordering node is already processing the reordering task for the N+1th batch of queries in parallel, and the retrieval and recall node is simultaneously processing the retrieval task for the N+2th batch of queries. The computing resources of the three processing stages are fully reused, and the overall throughput is improved compared to the original workflow where each stage waits serially.

[0035] During the semantic perturbation application phase of the retrieval and recall independent analysis channel, the channel processing module reads the time-series label of each retrieval entry from the input buffer queue and extracts the semantic vector representation corresponding to each candidate retrieval entry. This semantic vector is generated by the retrieval and recall node during nearest neighbor retrieval; it is typically a dense floating-point vector with 768 or 1024 dimensions, with each vector component being a 32-bit floating-point number. To further reduce computation time during the retrieval phase, the retrieval and recall node employs the BM25S pre-computation strategy. During system initialization, it pre-compiles word frequency statistics and inverted index construction for all documents in the knowledge base and serializes and caches the results to local storage. This ensures that each query during runtime only requires a memory table lookup operation, reducing the retrieval time of a single query from seconds to milliseconds. The BM25S pre-computation index is incrementally rebuilt when the knowledge base content is updated, only partially updating the inverted list corresponding to newly added or changed documents, without requiring a full recalculation. This maintains high-speed retrieval response while ensuring index real-time performance.

[0036] The channel processing module applies a random perturbation to the semantic vector. This random perturbation is implemented as isotropic Gaussian noise, i.e., a random sample value with a mean of 0 and a standard deviation equal to the controlled perturbation amplitude parameter is independently superimposed on each component of the semantic vector. The controlled perturbation amplitude parameter is determined proportionally to the L2 norm of the original semantic vector. In this embodiment, it is set to 5% of the L2 norm of the original vector. This proportion is determined based on the following: when the amplitude is less than 2%, the perturbation has too little impact on the recall result and it is difficult to distinguish the robustness differences between nodes; when the amplitude is greater than 10%, the perturbation intensity exceeds the normal semantic drift range and loses its significance in simulating the real upstream error. A 5% perturbation amplitude strikes a balance between the two. After applying the above random perturbation to the semantic vector of each candidate search entry, the channel processing module re-executes nearest neighbor retrieval with the perturbed semantic vector to generate a perturbed version of the recall candidate set. The perturbed version of the recall candidate set and the original recall candidate set are jointly stored in the output buffer of the channel for subsequent hit deviation calculation.

[0037] During the semantic perturbation application phase of the independent analysis channel reordering, the channel processing module reads each reordering ranking weight data from the input buffer queue and extracts the relevance score vector output by the reordering model for each candidate item. The relevance score vector is a one-dimensional floating-point vector with a length equal to the number of candidate items. After Softmax normalization, its components are located in the interval between 0 and 1, and its numerical distribution reflects the reordering model's ranking preference for candidate items under the current model parameter state. The channel processing module applies gradient permutation perturbation to the relevance score vector to simulate parameter drift in the re-ranking model. The specific implementation process of gradient permutation perturbation is as follows: First, the first-order difference gradient sequence of each component in the relevance score vector is calculated, i.e., an ordered sequence of differences between adjacent components. Then, a local random permutation is performed on the gradient sequence according to a preset perturbation intensity parameter. The permutation operation operates within a continuous sub-interval whose length does not exceed 20% of the total number of candidate entries. In this embodiment, when there are 20 candidate entries, the length of the permutation sub-interval is set to 4. Finally, the original score vector is reconstructed by integrating the permuted gradient sequence to obtain a perturbed version of the relevance score vector. The candidate entries are then rearranged according to the perturbed score vector to generate a perturbed version of the re-ranking result. Compared to directly applying additive noise to the score vector, gradient permutation perturbation can more realistically simulate the local ranking disorder caused by training data distribution shifts or model quantization compression in the re-ranking model, rather than a shift in the overall score.

[0038] During the semantic perturbation application stage of the independent analysis channel for answer generation, the channel processing module reads each generated text fragment from the input buffer queue and extracts the content of the input context window actually referenced by the large language model and the corresponding source citation identifier. The input context window is a string concatenated from the text content of the first few candidate entries output by the reordering node. In this embodiment, the top 5 candidate entries after reordering are concatenated to form the context window, with an average length of approximately 1200 Chinese characters. The channel processing module applies local semantic substitution perturbation to the input context window to simulate the interference of upstream input changes on the generation process. The specific implementation process of the local semantic substitution perturbation is as follows: First, word segmentation processing is performed on the context window to identify professional terminology units belonging to the financial regulatory field, including but not limited to regulatory indicator names, statutory obligation period values, institution type names, and administrative penalty clause numbers; then, random sampling is performed from the above units using a preset substitution rate parameter (set to 15% of the total number of professional terminology units in the context window in this embodiment), and synonyms or near-synonyms are applied to the sampled units. Word substitution involves retrieving replacement terms from a pre-built thesaurus in the financial regulatory domain. The semantic similarity (cosine similarity) between the replacement and original terms is required to be between 0.65 and 0.85, meaning the replacement and original terms are semantically similar but not entirely equivalent. This simulates local semantic drift caused by version differences or document updates in upstream search results. The replaced context window serves as the input to the perturbation version generation node, driving the large language model to re-generate the answer. The generated text fragments of the perturbation version are recorded token by token and saved along with the original generated text fragments to the output buffer of this channel. To improve the inference throughput of the answer generation channel, the system uses the vLLM inference framework to perform batch inference on the large speech model. The original and perturbation version context windows corresponding to multiple queries are merged into a single inference batch. This continuous batch processing mechanism enables efficient reuse of GPU computing resources, thereby reducing the average inference latency per query. Simultaneously, by dynamically allocating GPU memory, a suitable ratio of memory space is allocated to the generator and evaluator, balancing their throughput and eliminating inter-pipeline waiting caused by throughput differences, thereby further improving the overall system throughput. Figure 3 As shown, this embodiment proposes a fusion of dynamic segmentation and model parallelism, which further explores the available space for the generator in the memory occupied by the evaluator, thereby enabling the generator to obtain more memory compared to methods that separate dynamic segmentation or model parallelism.

[0039] The perturbation experiments of the three independent analysis channels are executed synchronously and concurrently in a parallel pipeline architecture. The output results of the three channels are summarized to form a paired dataset containing the original state and the perturbation state. The paired dataset will serve as the direct input for the subsequent calculation of the defense feature values ​​of each node, supporting the quantitative evaluation of hit deviation, order fluctuation and consistency score.

[0040] This embodiment uses hierarchical routing and node-differentiated semantic perturbation analysis to design controlled perturbation application strategies adapted to the data structures of three types of nodes with drastically different properties. In a parallel pipeline architecture, it synchronously drives multiple perturbation experiments, realizing the active quantitative detection of the inherent robustness boundary of each node, improving the efficiency of pipeline evaluation and the accuracy of error tracing of each node.

[0041] Furthermore, by simulating the logical offset caused by upstream changes through semantic perturbation analysis, calculating the hit deviation of the retrieval recall node, the order fluctuation of the reordering node, and the consistency score of the answer generation node, the specific implementation process of obtaining the defense feature value of each independent node against logical bias includes: For retrieval and recall nodes, the Jaccard distance between the recall set before and after perturbation is used as the hit deviation to measure the node's sensitivity to semantic perturbation. For reordering nodes, the absolute value of the difference between the Kendall rank correlation coefficients between the order sequences before and after perturbation is used as the order fluctuation to measure the reordering stability. For answer generation nodes, a comparative evaluation based on semantic embedding cosine similarity is used to calculate the consistency score of the generated text in the semantic space before and after perturbation. The hit deviation, order fluctuation, and consistency score are normalized and then concatenated to form the defensive feature value of each node.

[0042] Specifically, the hit deviation is measured by the Jaccard distance between the recall set before perturbation and the recall set after perturbation. The Jaccard distance is defined as 1 minus the ratio of the number of elements in the intersection to the number of elements in the union of the two sets. The Jaccard distance is 0 when the two sets are completely identical and 1 when the two sets are completely disjoint. Therefore, the hit deviation range is a closed interval from 0 to 1. The larger the value, the greater the difference between the recall result and the original result after the retrieval node is subjected to semantic perturbation, that is, the higher the sensitivity of the node to semantic perturbation. Specifically, the system reads the original recall candidate set and the perturbation version recall candidate set corresponding to each word order evolution queue element from the output buffer of the retrieval recall independent analysis channel. The original recall candidate set is denoted as set A, which contains the document identifiers of the first few candidate entries returned by the retrieval recall node in the unperturbed state. The perturbed version recall candidate set is denoted as set B, which contains the document identifiers of the first few candidate entries returned by the retrieval recall node after re-performing nearest neighbor search after applying Gaussian noise perturbation to the semantic vector. In this embodiment of the invention, the number of candidate entries recalled for each query is uniformly set to 20, that is, both set A and set B are finite sets containing 20 document identifiers.

[0043] The order fluctuation is measured by the absolute value of the difference between the Kendall rank correlation coefficients of the order sequences before and after the perturbation. It reflects the stability of the reordered node by quantifying the magnitude of the change in rank consistency between the two order sequences. The Kendall rank correlation coefficient measures the relative order consistency of pairs of elements in two ordered sequences, with a value range of [-1, 1]. The closer the coefficient is to 1, the more consistent the arrangement of the two sequences. The order fluctuation is measured by the decrease in Kendall rank consistency of the perturbed order sequence relative to the original order sequence, defined as 1 minus the Kendall rank correlation coefficient between the perturbed order sequence and the original reordered order sequence, i.e., order fluctuation = 1 - Kendall(perturbed order sequence, original order sequence). When the perturbed order sequence is completely consistent with the original order sequence, the Kendall coefficient is 1 and the order fluctuation is 0; when they are completely reversed, the Kendall coefficient is -1 and the order fluctuation is 2. The larger the value, the lower the stability of the order judgment of the reordered node after being perturbed by gradient permutation.

[0044] The consistency score is calculated using a comparative evaluation method based on semantic embedding cosine similarity. This method quantifies the directional similarity of the generated text before and after perturbation in the semantic vector space to reflect the semantic stability of the answer generation node. Specifically, the system reads the original generated text and the perturbed version generated text corresponding to each query from the output buffer of the independent analysis channel for answer generation. Semantic embedding encoding is then performed on both texts. This semantic embedding encoding uses the same pre-trained semantic embedding model as the retrieval and recall node, mapping the text to a dense semantic vector of fixed dimensions. In this embodiment, the embedding dimension is 768 dimensions. After semantic embedding encoding, the system calculates the cosine similarity between the semantic embedding vector of the original generated text and the semantic embedding vector of the perturbed version generated text. This cosine similarity is the consistency score of the answer generation node corresponding to the query, with a value range of [-1, 1]. The closer the value is to 1, the more consistent the semantic content of the generated text before and after perturbation, indicating higher stability of the answer generation node against upstream input perturbations.

[0045] Because the dimensions and numerical distribution ranges of the three indicators—hit deviation, ordinal fluctuation, and consistency score—differ, direct concatenation would lead to a systematic imbalance in the contribution weights of the three indicators in the subsequent model. Therefore, the system performs normalization processing on each of the three original indicators before concatenation. The specific normalization process is as follows: For the hit deviation sequence, the system performs minimum-maximum normalization on the original hit deviation values ​​corresponding to the queue elements, mapping the minimum value of the sequence to 0 and the maximum value to 1, resulting in a normalized hit deviation sequence; for the ordinal fluctuation sequence, the same minimum-maximum normalization processing is performed to obtain a normalized ordinal fluctuation sequence; for the consistency score sequence, the original cosine similarity is first negative to achieve semantic direction unification, and then minimum-maximum normalization processing is performed to obtain a normalized consistency score sequence.

[0046] After normalization, the system concatenates the three normalized indices corresponding to each syllogism evolution queue element into a three-dimensional vector in a fixed order. This three-dimensional vector represents the defense feature value of each node corresponding to that queue element. Each component of the defense feature value is encoded in a way that the positive value corresponds to the node's vulnerability; that is, the larger the value of each component, the weaker the corresponding node's resistance to semantic perturbations. The vector is stored in the form of a three-dimensional floating-point vector, with the three components corresponding to the normalized hit deviation, normalized ordinal fluctuation, and normalized consistency score, respectively. The defense feature value vectors of all queue elements are aggregated to form a 680-row, 3-column defense feature value matrix, which serves as the standard input for the subsequent risk transmission weight spectrum calculation stage, supporting the quantitative modeling and analysis of the cascading error propagation law between nodes in the RAG pipeline.

[0047] This embodiment introduces three metrics that are mathematically highly compatible with the processing logic of each node, namely Jaccard distance, the absolute value of the difference in Kendall rank correlation coefficients, and semantic embedding cosine similarity, based on the data structure characteristics of the three types of nodes. After normalization and concatenation, the resulting three-dimensional defense feature vector comprehensively represents the defense characteristics of the three types of nodes in the RAG pipeline against semantic perturbations on a unified numerical scale. This provides high-quality feature input with dimension alignment and scale consistency for subsequent causal modeling of risk transmission weight spectrum.

[0048] Furthermore, based on the aforementioned defense feature values, logical testing is performed to quantify the risk transmission weight spectrum of the impact of upstream node defects on downstream node judgments. The specific implementation process of obtaining the cascading pattern that changes with the depth of the phrasing evolution queue includes: Using the defense characteristics of each node as independent variables, a directed acyclic graph structure for causal transmission between nodes is constructed. The path coefficients of the error propagation path from the retrieval recall node defects to the reordering node and the answer generation node are estimated using a causal inference framework, yielding the risk transmission weights of each directed edge. These risk transmission weights are grouped and statistically analyzed according to the depth index of the word order evolution queue, and a function curve of the transmission weights changing with queue depth is fitted to identify the monotonically increasing, decreasing, and oscillating hierarchical patterns of the transmission weights with depth. Finally, the risk transmission weights of each path are summarized to generate a risk transmission weight spectrum between nodes.

[0049] Specifically, using the defense feature values ​​of each node as independent variables, a directed acyclic graph (DAG) structure for causal transmission between nodes is constructed under the node topological order constraints of the RAG pipeline. The DAG uses the defense feature value components (i.e., normalized hit deviation, normalized order fluctuation, and normalized consistency score) corresponding to the retrieval recall node, reordering node, and answer generation node as three vertices. The topological constraints of the directed edges are determined based on the information flow direction of the RAG pipeline: the retrieval recall node is the first stage of the pipeline, and its output directly enters the reordering node; the reordering node is the second stage, and its output enters the answer generation node; the answer generation node is the third stage. Accordingly, the DAG contains three directed edges: the first path from the retrieval recall node to the reordering node; the second path from the retrieval recall node to the answer generation node (a direct transmission path across levels, simulating the cross-level impact of retrieval recall error on generation quality); and the third path from the reordering node to the answer generation node. The directed acyclic graph (DAG) satisfies the acyclic constraint, meaning there are no directed edges pointing from downstream nodes to upstream nodes, maintaining consistency with the unidirectional information flow characteristic of the RAG pipeline in terms of causal structure. In the aforementioned implementation example, the system successfully constructed a DAG based on the above topological constraints. The graph contains 3 vertices and 3 directed edges, and the loop detection algorithm verifies that there are no directed cycles, satisfying the formal definition requirements of a DAG.

[0050] The risk transmission weights corresponding to each directed edge in the aforementioned directed acyclic graph are estimated using a causal inference framework. This framework employs structural equation modeling as the core algorithm for path coefficient estimation, using the defense eigenvalue matrix corresponding to 680 elements of the word order evolution queue as observation samples to perform maximum likelihood estimation on the path coefficients of each directed edge in the directed acyclic graph. Specifically, for the first path from the retrieval recall node to the reordering node, the system uses the normalized hit deviation sequence as the independent variable and the normalized rank fluctuation sequence as the dependent variable, and estimates the conditional causal effect between the two while controlling for other variables. The resulting path coefficient is the risk transmission weight of the directed edge, representing the expected increment of the rank fluctuation of the reordering node when the retrieval recall node's deviation increases by 1 unit. For the second path from the retrieval recall node to the answer generation node, the system uses the normalized hit deviation sequence as the independent variable and the normalized consistency score sequence as the dependent variable, and estimates the direct cross-level transmission effect of the retrieval recall error on the stability of answer generation while controlling for the rank fluctuation of the reordering node. For the third path from the reordering node to the answer generation node, the system uses the normalized rank fluctuation sequence as the independent variable and the normalized consistency score sequence as the dependent variable, and estimates the direct transmission effect of the reordering error on the stability of answer generation while controlling for the normalized hit deviation of the retrieval recall node. The above three risk transmission weights are attached to the corresponding edges of the directed acyclic graph as scalar attributes of each directed edge, for use in subsequent grouping statistics and cascading pattern identification.

[0051] The risk transmission weights are grouped and statistically analyzed according to the depth index of the phrasal evolution queue to identify the cascading pattern of transmission weight changes with queue depth. Specifically, the depth index of the phrasal evolution queue is determined by the level number of each text fragment in the aforementioned directed dependency graph topological sorting. The larger the level number, the deeper downstream the text fragment is in the logical chain of the financial regulatory document. The system uses the level number as the grouping key to divide all 680 queue elements into several groups according to their topological level. For each topological level group, the system calculates the mean of the risk transmission weights of the three paths corresponding to the 680 queue elements in that group, obtaining a scatter sequence of transmission weights for each path as a function of topological level depth. Each path forms a transmission weight-depth correspondence sequence of length 14. Subsequently, the system performs function curve fitting on the transmission weight-depth corresponding sequence for each path. The fitting strategy combines piecewise linear regression and cubic spline interpolation: First, piecewise linear regression is performed to divide the 14 depth nodes into 3 intervals to identify which depth intervals show a monotonically increasing trend, which intervals show a monotonically decreasing trend, and which intervals show an oscillating trend in the transmission weight; then, cubic spline interpolation is used to fit the global curve to smoothly connect the piecewise linear trends of the 3 intervals, thus obtaining a function curve in which the transmission weight continuously changes with the queue depth. The identification of cascading patterns is based on the following criteria: If more than 80% of the adjacent intervals in the weight-depth correspondence sequence show increasing weights, it is determined to be a monotonically increasing cascading pattern, indicating that the error propagates along the pipeline and amplifies with increasing nesting depth of clause logic; if more than 80% of the adjacent intervals show decreasing weights, it is determined to be a monotonically decreasing cascading pattern, indicating that the RAG pipeline has a natural inhibitory effect on error propagation in deeply nested clauses; if the weights of adjacent intervals alternate between increasing and decreasing, and each direction accounts for more than 30%, it is determined to be an oscillating cascading pattern, indicating that the intensity of error propagation fluctuates periodically with the clause logic hierarchy.

[0052] After fitting the transmission weight-depth function curves and identifying cascading patterns for each of the three paths, the risk transmission weights, transmission directions, cascading pattern types, and transmission weight-depth function curves for each path are summarized into a risk transmission weight spectrum between nodes. This risk transmission weight spectrum stores the scalar risk transmission weights of each directed edge in the directed acyclic graph in the form of an adjacency weight matrix. The row indices of the matrix correspond to the source nodes, the column indices correspond to the target nodes, and the matrix elements are the path coefficient estimates for the corresponding directed edges. Simultaneously, the risk transmission weight spectrum also stores the fitting parameters and cascading pattern type labels for the transmission weight-depth function curves corresponding to each path, as well as the p-value and confidence interval for path significance testing. This risk transmission weight spectrum is output as a structured data object for direct use in subsequent sensitive source identification and performance bottleneck localization stages.

[0053] This embodiment utilizes the risk transmission weight spectrum and introduces a directed acyclic graph and structural equation model under the topological constraints of the RAG pipeline. It incorporates the defense characteristic values ​​of three types of nodes into a unified causal modeling framework, realizing the synchronous quantitative estimation of the risk transmission weights of the three error propagation paths. Furthermore, it reveals the dynamic law of the transmission weights changing with the depth of the queue as the word order evolves through group statistics and function curve fitting, effectively improving the causal diagnosis capability of multiple nodes and laying a quantitative foundation for the accurate location of subsequent performance bottlenecks.

[0054] Furthermore, the specific implementation process of identifying sensitive sources causing shifts in the cascading pattern based on the risk transmission weight spectrum, determining them as performance bottlenecks of the RAG system, and generating a pipeline evaluation report containing bottleneck node coordinates, risk diffusion quantification curves, and targeted recommendations includes: Sensitivity ranking is performed on the risk transmission weight spectrum. Source nodes whose cumulative outgoing edge weights of each directed path exceed a preset threshold are marked as sensitive sources and located as performance bottleneck nodes in the RAG system pipeline. The coordinate information of the bottleneck nodes in the sequence evolution queue is extracted and written into the evaluation report as the bottleneck node coordinates. Based on the changes in the weights of each path in the risk transmission weight spectrum with the depth of the queue, a risk diffusion quantification curve is plotted to characterize the magnitude and direction of error propagation along the pipeline. Combining the type of the bottleneck node and the corresponding defense feature value, the optimization strategy knowledge base is matched to generate targeted suggestions including component replacement suggestions, hyperparameter adjustment directions, and data augmentation schemes. A complete pipeline evaluation report is then summarized and output.

[0055] Specifically, the risk transmission weight spectrum is sorted by sensitivity. The cumulative outgoing edge weights of each directed path are used as the sensitivity metric. Source nodes whose cumulative outgoing edge weights exceed a preset threshold are marked as sensitive sources and identified as performance bottleneck nodes in the RAG system pipeline. Specifically, the system reads the path coefficients of all outgoing edges for each node when it is a source node from the adjacency weight matrix of the risk transmission weight spectrum. All outgoing edge weights of the same source node are summed to obtain the cumulative outgoing edge weight of that node. The preset threshold is set to the 75th percentile of the cumulative outgoing edge weights of all nodes; in this embodiment, this threshold is approximately 0.68. The preset threshold is not fixed at the 75th percentile. The system supports dynamic adjustment by the user based on the tolerance requirements of the actual business scenario. For example, in a high-risk financial compliance scenario with zero tolerance for error transmission, the threshold can be lowered to the 50th percentile to capture more potential bottleneck nodes; in a resource-constrained rapid screening scenario, the threshold can be raised to the 90th percentile to focus on the most prominent bottlenecks.

[0056] The coordinate information of the bottleneck node in the word order evolution queue is extracted and written into the evaluation report as the bottleneck node coordinates. The bottleneck node coordinates are not a single scalar, but a multi-dimensional coordinate tuple containing the following three types of positional information: First, the node type identifier, i.e., the logical role of the bottleneck node in the RAG pipeline (retrieval recall, reordering, or answer generation), which in the aforementioned implementation case is a retrieval recall node; second, the word order evolution queue depth coordinates, i.e., the binary coordinates composed of the topological level number and level index corresponding to the word order evolution queue element that results in the highest defense feature value for this node (i.e., the most sensitive to semantic perturbations). The system retrieves several queue elements with the largest normalized hit deviation components from the defense feature value matrix, extracts the level number and level index from their triple position index, and then... The bottleneck depth coordinates are as follows: In a preferred embodiment, the top 10 queue elements with the highest normalized hit deviation are concentrated in the deep clause intervals of topology levels 9 to 12, corresponding to text fragments numbered 1 to 4 at each level. The bottleneck depth coordinates are recorded as rectangular coordinate areas of level numbers 9 to 12 and level numbers 1 to 4. The third type is the original document physical coordinates, which are the original document physical start and end character offsets recorded in the aforementioned triplet position index. In this implementation case, this corresponds to a specific clause text interval within the range of Chapter 4, Section 3 to Chapter 4, Section 7 of a compliance manual, with a character offset interval of approximately character 18400 to character 24700. The combination of the above three types of coordinate information forms complete bottleneck node coordinates, enabling readers of the assessment report to accurately locate the distribution range of the performance bottleneck from three dimensions: node type, queue logical position, and original document physical position, rather than just obtaining an abstract node type label.

[0057] The risk diffusion quantification curve is plotted based on the change sequence of the path weights in the risk transmission weight spectrum with the depth of the queue, depicting the magnitude and direction of error propagation along the pipeline. Specifically, with the 14 topological levels (1 to 14) of the word order evolution queue as the horizontal axis and the mean of the risk transmission weights of each path within the corresponding depth group as the vertical axis, an independent risk diffusion quantification curve is plotted for each path recorded in the risk transmission weight spectrum. In the aforementioned implementation case, a total of 3 curves are plotted, corresponding to the path from retrieval recall to reordering, the path from retrieval recall to answer generation (skipping levels), and the path from reordering to answer generation, respectively. The system also marks 95% confidence intervals on each curve. The confidence interval is based on the standard error of the transmission weights of each path within the corresponding depth group. A wider confidence band is given to depth levels with larger variance in transmission weights, indicating that the error transmission behavior in that depth interval has higher uncertainty. The three risk diffusion quantification curves are embedded in the assessment report in vector graphic format, with additional textual explanations of the path meaning of each curve, the type of cascading pattern, and the numerical labeling of the transmission weight at key depth nodes. This allows the assessment report reader to accurately grasp the dynamic evolution of the error propagation along the pipeline.

[0058] Based on the type of bottleneck node and its corresponding defense characteristic value, an optimization strategy knowledge base is matched to generate targeted suggestions including component replacement recommendations, hyperparameter adjustment directions, and data augmentation schemes. The entries in the optimization strategy knowledge base are sourced from RAG system evaluation and optimization research published in academic conferences and journals such as ACL, EMNLP, and SIGIR, as well as publicly released RAG engineering practice reports from organizations such as NVIDIA and HuggingFace. Each strategy entry must provide at least one quantitative effect verification data point (such as the mean and standard deviation of the hit rate improvement), and the verification data must come from corpus experiments with similar structural features to the financial regulatory text of this invention. The optimization strategy knowledge base is organized in a tree-like index structure. The first-level index represents bottleneck node types (retrieval recall nodes, reordering nodes, and answer generation nodes). The second-level index represents the main problem features revealed by the defensive feature value vector (such as combinations of high hit deviation and low rank fluctuation, or high rank fluctuation and low consistency score). The third-level index represents specific corpus features of financial regulatory documents (such as the proportion of deeply nested clauses, the density of numerical clauses, and the density of cross-regulatory citations). Leaf nodes store specific optimization strategy entries corresponding to the above three-level index combinations. Each strategy entry includes a component replacement suggestion field, a hyperparameter adjustment direction field, and a data augmentation scheme field. Based on the bottleneck node type (retrieval recall node), defensive feature value vector features, and corpus features of the current evaluation object, the system performs multi-level index matching in the optimization strategy knowledge base to retrieve the set of strategy entries that are closest to the above three-level feature combinations. In a preferred embodiment, the system matches a total of three targeted suggestions. The first suggestion is a component replacement: replace the general-domain pre-trained semantic embedding model used in the current retrieval and recall nodes with a dedicated embedding model that performs domain-adaptive fine-tuning on the financial regulatory corpus. Historical validation data shows that this replacement operation can improve the retrieval and recall hit rate of the financial regulatory corpus by approximately 17 to 23 percentage points, and correspondingly reduce the hit deviation by approximately 0.18 to 0.24 units. The second suggestion is a hyperparameter adjustment: increase the number of candidate items recalled in the retrieval and recall nodes from the current 20 to 35, and simultaneously adjust the truncation threshold for entering the re-ranking node from the top 20 candidate items to the top 35. This expands the recall pool to reduce the hit deviation caused by the exclusion of boundary candidate items under semantic vector perturbation in deeply nested clauses. Historical validation data shows that this parameter adjustment can reduce the proportion of highly sensitive clauses (items with a hit deviation exceeding 0.60) from approximately 18% to approximately 11%.The third suggestion is a data augmentation scheme: For the deep clause interval (topology levels 9 to 12) marked by the bottleneck node coordinates, related clauses with cross-document logical references to clauses in this interval are selectively extracted from the existing financial regulatory database. An augmented sample set, approximately three times the size of the current corpus entries for this interval, is constructed. Targeted contrastive learning is then performed on the semantic embedding model of the retrieval recall node to fine-tune it, enhancing the model's ability to distinguish the semantic representation of deeply nested clauses. Historical validation data shows that after implementing the above targeted data augmentation for deep clauses, the average hit deviation of clauses within this interval can be reduced by approximately 0.21 units. These three suggestions, along with optimization priority ranking (arranged in descending order based on the expected reduction in hit deviation for each suggestion), implementation cost assessment (combined scoring based on the required additional training data volume, inference latency increment, and parameter adjustment operation complexity), and expected overall effect range, are all included in the suggestion field of the evaluation report.

[0059] After completing the above four sub-steps, the system summarizes all results and outputs a complete pipeline evaluation report. The pipeline evaluation report is output in a structured document format and includes the following fields: report metadata fields (including evaluation timestamp, evaluation object system identifier, word order evolution queue size, and defense feature value matrix size), bottleneck node coordinate fields (including node type identifier, queue depth coordinates, and original document physical coordinates), risk transmission weight spectrum fields (including adjacency weight matrix, path cascade pattern type markers, and path significance statistics), risk diffusion quantification curve fields (including three vector graphic embeddings and key node numerical annotations), and targeted suggestion fields (including the content, priority, cost assessment, and expected effect range of three suggestions).

[0060] This embodiment achieves end-to-end automated output from risk transmission weight spectrum to complete assessment report by executing four sub-steps in a fully automated pipeline: sensitivity ranking, multi-dimensional coordinate extraction, quantification curve plotting, and knowledge base matching. It also improves the rapid assessment capability of the RAG system by setting the generation accuracy of optimization suggestions down to the specific component type, hyperparameter name, and data augmentation operation level.

[0061] Example 2: This embodiment fully deploys the aforementioned rapid evaluation method for RAG systems based on pipeline architecture into a review RAG system built on semantic retrieval technology by a large insurance group.

[0062] Furthermore, the system's rapid evaluation process begins with receiving the original regulatory document stream containing insurance claims regulatory clauses and internal operating procedures. Based on contextual logical dependencies, this stream is deconstructed into a word order evolution queue within the RAG system. Specifically, the system performs structured preprocessing on approximately 2,500 document streams, extracting cross-sentence logical dependencies and constructing a directed dependency graph based on paragraph boundary identifiers such as chapter numbers and semantic connectors such as "if," "because," and "otherwise." Based on the topological sorting of nodes in the directed dependency graph, a breadth-first traversal strategy is used to segment the original document stream into a sequence of semantically coherent text segments. The system assigns contextual position index labels containing physical offsets, topological level numbers, and level indices to approximately 1,200 segmented text segments, and strictly arranges them according to the direction of semantic evolution, ultimately generating a word order evolution queue carrying position index labels.

[0063] Furthermore, the system deploys intercept probes at each node of the word order evolution queue—retrieval recall, reordering, and answer generation—to capture logical anchor data containing retrieval item time-series labels, reordering position weights, and generated text fragments. At the retrieval recall node, the system injects intercept probes via hook functions to capture the vector similarity scores and millisecond-level timestamps of the top 20 candidate retrieval items returned for the slice query, forming retrieval item time-series labels. At the reordering node, the intercept probes capture the normalized position weight vectors and position changes of each candidate item output by the reordering model, forming reordering position weights. At the answer generation node, the intercept probes capture generated text fragments token by token during the large language model decoding process and record their corresponding source citation identifiers and timestamps. The system aligns the above three data streams by timestamp and performs clock drift correction before converging them into unified logical anchor data, which is then output to the data entry point of the parallel pipeline architecture.

[0064] Furthermore, the system imports logical anchor data into a parallel pipeline architecture and introduces semantic perturbation analysis for different nodes. The system routes the logical anchor data to independent analysis channels corresponding to the retrieval recall layer, re-ranking layer, and answer generation layer. In the retrieval recall channel, the system independently superimposes Gaussian noise with a mean of zero and a standard deviation of 4% of the original vector norm onto the semantic vectors of the retrieval items, generating a perturbed version of the recall candidate set. In the re-ranking channel, the system applies gradient permutation perturbation to the position weight vector, with an effect range of five consecutive candidate item sub-intervals, simulating parameter drift in the re-ranking model. In the answer generation channel, the system identifies insurance terminology in the input context window and applies local synonym substitution perturbation to the input context window at a replacement rate of 12%, simulating the interference of upstream input changes on the generation process. After simulating the logical offset caused by upstream changes through the semantic perturbation analysis, the system calculates the defense feature value of each independent node against logical bias. For the retrieval recall node, the system calculates the Jaccard distance between the recall sets before and after perturbation as the hit deviation to measure sensitivity. For reordering nodes, the system calculates the absolute value of the difference between the Kendall rank correlation coefficients between the order sequences before and after the perturbation as the order fluctuation to measure stability. For answer generation nodes, the system calculates the embedding cosine similarity of the generated text in the semantic space before and after the perturbation as the consistency score. After processing the hit deviation, order fluctuation, and consistency score using min-max normalization, the system concatenates them in a fixed order to form the three-dimensional defense feature value of each node.

[0065] Furthermore, the system performs logical tests based on the defense feature values ​​to quantify the risk transmission weight spectrum of the impact of upstream node defects on downstream node judgments, obtaining the cascading pattern that changes with the depth of the phrasal evolution queue. Specifically, the system constructs a directed acyclic graph structure for causal transmission between nodes, conforming to the constraint of unidirectional information flow, using the defense feature value components of each node as independent variables. Through the causal inference framework of structural equation modeling, the system performs maximum likelihood estimation of the path coefficients of the error propagation path from the retrieval and recall node defects to the reordering node and the answer generation node, obtaining the risk transmission weight of each directed edge. The system groups and statistically analyzes the above risk transmission weights according to the depth index of the phrasal evolution queue, fitting a function curve showing that the transmission weight changes continuously with the queue depth. The data shows that for claim clauses with deep mutual references, the transmission weight from reordering to answer generation exhibits a significant monotonically increasing cascading pattern, while the transmission weight from retrieval and recall to reordering shows an oscillating pattern in the middle of the depth segment before gradually decaying. Based on this, the system summarizes the risk transmission weights of each path, generating a complete risk transmission weight spectrum between nodes.

[0066] Furthermore, the system identifies sensitive sources causing deviations in the cascading pattern based on the risk transmission weight spectrum, determines them as performance bottlenecks of the RAG system, and generates a pipeline evaluation report containing key information. The system performs sensitivity ranking on the risk transmission weight spectrum, using the cumulative outgoing edge weights of each directed path as the quantification basis. In this case, the cumulative outgoing edge weight of the retrieval recall node reaches 0.90, exceeding the set 75th percentile threshold, and is marked as a sensitive source and precisely located as the primary performance bottleneck node of the insurance claims RAG system. The system extracts the depth coordinates and original document physical region coordinates of the bottleneck node with the highest normalized hit deviation in the word order evolution queue, and writes them as the bottleneck node coordinates into the evaluation report. Based on the changes in the weights of each path in the risk transmission weight spectrum with the depth of the queue, the system plots a risk diffusion quantification curve including confidence intervals, characterizing the magnitude and direction of error propagation along the pipeline. By combining bottleneck node types and their highly sensitive defense features, the system matches and optimizes the strategy knowledge base, generating targeted recommendations including replacing the general domain pre-trained model with an insurance-domain adaptive fine-tuning model, adjusting and relaxing the candidate recall cutoff threshold to the top 35 items, and implementing a targeted comparative learning data augmentation scheme for deeply nested claims clauses. All the above analysis data and conclusions are summarized and output as a complete pipeline evaluation report.

[0067] This embodiment transforms the static and fragmented evaluation mode of the text generation system into a dynamic tracking process by deploying non-intrusive end-to-end interception probes and introducing hierarchical semantic perturbation analysis tailored to different node characteristics. By establishing defensive feature values ​​and causal directed acyclic graphs, it effectively addresses the deficiency of being unable to quantify the amplification patterns of error propagation between multi-level components. Based on a quantitative risk transmission weight spectrum, it can automatically perceive and accurately identify core performance bottleneck nodes that cause overall performance degradation. Furthermore, by analyzing the cascading patterns at each depth level, it automatically outputs highly executable optimization strategies covering component replacement, hyperparameter tuning, and data augmentation, improving the evaluation accuracy and tuning efficiency of the multi-component cascaded retrieval architecture. Regarding overall evaluation speed, compared to the original workflow of sequential execution at each stage, the use of BM25S pre-computation reduces retrieval time to the millisecond level, and vLLM batch inference improves answer generation and evaluation throughput, thereby reducing idle waiting time at each stage.

[0068] It should be clarified that the embodiments described above are merely exemplary and are intended to aid in understanding the present invention, not to limit it. Those skilled in the art can make various changes and modifications after grasping the core ideas of the present invention. Therefore, the scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A rapid evaluation method for RAG systems based on a pipelined architecture, characterized in that, include: Receive the raw regulatory document stream and deconstruct it into a grammatical evolution queue within the RAG system according to the contextual logical dependency relationship; Interception probes are deployed at each node of the retrieval, reordering, and answer generation in the word order evolution queue to capture logical anchor data containing retrieval entry time tags, reordering order weights, and generated text fragments. The logical anchor data is imported into a parallel pipeline architecture, and semantic perturbation analysis is introduced for different nodes. The semantic perturbation analysis is used to simulate the logical offset caused by upstream changes, and the hit deviation of the retrieval recall node, the order fluctuation of the reordering node, and the consistency score of the answer generation node are calculated to obtain the defense feature value of each independent node against logical deviation. Logical testing is performed based on the defense feature values ​​to quantify the risk transmission weight spectrum of the impact of upstream node defects on downstream node judgment, and to obtain the cascading pattern that changes with the depth of the queue as the word order evolves. Based on the risk transmission weight spectrum, sensitive sources that cause the cascade pattern to shift are identified, which are determined to be the performance bottlenecks of the RAG system, and a pipeline evaluation report containing bottleneck node coordinates, risk diffusion quantification curves, and targeted suggestions is generated.

2. The rapid evaluation method for a RAG system based on a pipelined architecture according to claim 1, characterized in that, The specific implementation process of receiving the raw regulatory document stream and deconstructing it into a phrasal evolution queue within the RAG system based on contextual logical dependencies includes: The received raw regulatory document stream is subjected to structured preprocessing. Cross-sentence logical dependencies are extracted based on paragraph boundary identifiers and semantic connectors, and a directed dependency graph is constructed. Based on the topological sorting of nodes in the directed dependency graph, the raw document stream is divided into a sequence of text segments. The text segment sequence is assigned a context position index label and arranged according to the semantic evolution direction to generate a word order evolution queue carrying position index labels.

3. The rapid evaluation method for a RAG system based on a pipelined architecture according to claim 1, characterized in that, The specific implementation process of deploying interception probes at each node of the retrieval, reordering, and answer generation in the word order evolution queue to capture logical anchor data containing retrieval item time sequence labels, reordering ranking weights, and generated text fragments includes: Interception probes are injected into the retrieval and recall nodes to capture the vector similarity scores and corresponding temporal tags of each candidate retrieval item, forming the temporal tags of the retrieval items; interception probes are injected into the reordering nodes to capture the position weight vectors of each candidate item output by the reordering model, forming the reordering position weights; interception probes are injected into the answer generation nodes to capture the generated text fragments and corresponding source citation identifiers token by token during the large language model decoding process, forming the generated text fragments; the temporal tags of the retrieval items, the reordering position weights, and the generated text fragments are aligned by timestamps and aggregated into logical anchor data, which is then output to the data entry point of the parallel pipeline architecture.

4. The rapid evaluation method for a RAG system based on a pipelined architecture according to claim 1, characterized in that, The specific implementation process of importing the logical anchor data into a parallel pipeline architecture and introducing semantic perturbation analysis for different nodes includes: The logical anchor data is routed to the corresponding independent analysis channels in the parallel pipeline architecture according to the retrieval recall layer, re-ranking layer, and answer generation layer. In the retrieval recall channel, random perturbations are applied to the semantic vectors of the retrieval entries to generate perturbed versions of the recall candidate set. In the re-ranking channel, gradient permutation perturbations are applied to the position weight vectors to simulate parameter drift in the re-ranking model. In the answer generation channel, local semantic substitution perturbations are applied to the input context window to simulate the interference of upstream input changes on the generation process.

5. The rapid evaluation method for a RAG system based on a pipelined architecture according to claim 4, characterized in that, The specific implementation process of simulating the logical offset caused by upstream changes through semantic perturbation analysis, calculating the hit deviation of the retrieval recall node, the order fluctuation of the reordering node, and the consistency score of the answer generation node, and obtaining the defense feature value of each independent node against logical bias includes: For retrieval and recall nodes, the Jaccard distance between the recall set before and after perturbation is used as the hit deviation to measure the node's sensitivity to semantic perturbation. For reordering nodes, the absolute value of the difference between the Kendall rank correlation coefficients between the order sequences before and after perturbation is used as the order fluctuation to measure the reordering stability. For answer generation nodes, a comparative evaluation based on semantic embedding cosine similarity is used to calculate the consistency score of the generated text in the semantic space before and after perturbation. The hit deviation, order fluctuation, and consistency score are normalized and then concatenated to form the defensive feature value of each node.

6. The rapid evaluation method for a RAG system based on a pipelined architecture according to claim 1, characterized in that, Based on the aforementioned defense feature values, logical testing is performed to quantify the risk transmission weight spectrum of the impact of upstream node defects on downstream node judgments. The specific implementation process of obtaining the cascading pattern that changes with the depth of the phrasing evolution queue includes: Using the defense characteristics of each node as independent variables, a directed acyclic graph structure for causal transmission between nodes is constructed. The path coefficients of the error propagation path from the retrieval recall node defects to the reordering node and the answer generation node are estimated using a causal inference framework, yielding the risk transmission weights of each directed edge. These risk transmission weights are grouped and statistically analyzed according to the depth index of the word order evolution queue, and a function curve of the transmission weights changing with queue depth is fitted to identify the monotonically increasing, decreasing, and oscillating hierarchical patterns of the transmission weights with depth. Finally, the risk transmission weights of each path are summarized to generate a risk transmission weight spectrum between nodes.

7. The rapid evaluation method for a RAG system based on a pipelined architecture according to claim 6, characterized in that, The specific implementation process of identifying sensitive sources that cause shifts in the cascading pattern based on the risk transmission weight spectrum, determining them as performance bottlenecks of the RAG system, and generating a pipeline evaluation report containing bottleneck node coordinates, risk diffusion quantification curves, and targeted recommendations includes: Sensitivity ranking is performed on the risk transmission weight spectrum. Source nodes whose cumulative outgoing edge weights of each directed path exceed a preset threshold are marked as sensitive sources and located as performance bottleneck nodes in the RAG system pipeline. The coordinate information of the bottleneck nodes in the sequence evolution queue is extracted and written into the evaluation report as the bottleneck node coordinates. Based on the changes in the weights of each path in the risk transmission weight spectrum with the depth of the queue, a risk diffusion quantification curve is plotted to characterize the magnitude and direction of error propagation along the pipeline. Combining the type of the bottleneck node and the corresponding defense feature value, the optimization strategy knowledge base is matched to generate targeted suggestions including component replacement suggestions, hyperparameter adjustment directions, and data augmentation schemes. A complete pipeline evaluation report is then summarized and output.

Images