An audit knowledge retrieval path optimization method based on reinforcement learning

Abstract

This invention discloses a method for optimizing audit knowledge retrieval paths based on reinforcement learning, relating to the fields of information retrieval and artificial intelligence. The method includes: constructing a hierarchical path space for audit knowledge; generating multiple candidate paths based on the query; using the information gain of the answer distribution before and after retrieval as an intrinsic reward, modeling path selection as a Markov decision process, and training the optimal strategy using the GRPO algorithm; during inference, combining path and document score ranking, and outputting an interpretable answer. This invention can significantly improve the accuracy and efficiency of audit knowledge retrieval, reduce the number of retrieval steps, enhance the interpretability of results, possess adaptive learning capabilities, and is convenient for engineering deployment.

Description

Technical Field

[0001] This invention relates to the fields of information retrieval and artificial intelligence, and in particular to a method for optimizing audit knowledge retrieval paths based on reinforcement learning. Background Technology

[0002] Knowledge retrieval in auditing is a core component of information acquisition and risk assessment in auditing work. Auditors need to quickly and accurately locate knowledge evidence relevant to the audit task from massive amounts of legal provisions, standards, guidelines, and case documents. However, with the increasing complexity of auditing operations and the continuous growth of knowledge bases, traditional retrieval methods have revealed significant technical bottlenecks in terms of path optimization, retrieval efficiency, and result interpretability, making it difficult to meet the demands of modern auditing for intelligent, efficient, and traceable knowledge acquisition.

[0003] In existing technologies, mainstream audit knowledge retrieval methods can be mainly divided into two categories. One category is based on inverted indexes and semantic similarity models, using keyword matching and vector retrieval for document recall. Although this type of method is simple to implement, the retrieval path is fixed and lacks a dynamic optimization mechanism, making it impossible to adaptively adjust the retrieval strategy based on feedback information during the retrieval process. For the multi-hop reasoning requirements in complex audit scenarios (such as establishing a logical connection of evidence chain from "audit domain → audit subject → risk point → specific rules"), relying solely on keyword matching and semantic reordering is insufficient. In addition, this type of method relies on manually annotated metadata and pre-set dictionaries, making it poorly adaptable to dynamic updates of the audit knowledge base and unable to autonomously learn and optimize the retrieval path. Another type of method introduces knowledge graphs and iterative optimization mechanisms, using entity recognition and relational reasoning for multi-hop retrieval. However, it still has obvious limitations: it lacks a clear reward function design, the process of optimizing historical clues relies heavily on black-box reasoning of large language models, and it lacks quantifiable path evaluation standards; the retrieval efficiency is low, often requiring multiple iterations to obtain the answer, and the response speed is insufficient in scenarios with high audit real-time requirements; at the same time, this type of method does not consider the hierarchical knowledge structure characteristics of the audit field, and cannot effectively utilize the inherent hierarchical relationships of audit regulations and standards to guide the construction of retrieval paths. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention provides an audit knowledge retrieval path optimization method based on reinforcement learning.

[0005] Therefore, the technical problem solved by this invention is the core problem of insufficient retrieval path optimization, low retrieval efficiency, and lack of interpretability in existing audit knowledge retrieval systems.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for optimizing audit knowledge retrieval paths based on reinforcement learning, comprising: The documents in the audit knowledge base are hierarchically clustered and semantically annotated to form a four-level category structure from audit domain to specific rules, and a corresponding complete path index is generated for each document; Based on the audit query input by the user, and based on a four-layer category structure, multiple candidate search paths are generated through similarity calculation and collaborative filtering. Based on the generated candidate retrieval paths, the information gain brought by each retrieval step is quantified by comparing the uncertainty changes in the model's answer distribution before and after retrieval, serving as an intrinsic reward signal for reinforcement learning. The retrieval path selection is modeled as a Markov decision process. The intrinsic reward signal of reinforcement learning is used to train the policy network through the GRPO algorithm to learn the optimal path selection policy. During the reasoning phase, multiple paths are selected using the trained policy network. After retrieving the corresponding documents, the path scores and document scores are combined and sorted to generate an interpretable answer.

[0007] As a preferred scheme for optimizing audit knowledge retrieval paths based on reinforcement learning, wherein: The process involves hierarchical clustering and semantic annotation of documents in the audit knowledge base, forming a four-layer category structure from audit domains to specific rules, and generating a corresponding complete path index for each document, including: The documents in the audit knowledge base are modeled hierarchically, and a four-layer semantic category structure is constructed, which includes at least the audit domain layer, audit subject layer, risk point layer and detailed rule layer. In this process, each document is encoded into a dense vector using a pre-trained language model; A hierarchical clustering algorithm is used to perform multi-level clustering of document vectors, refining the clusters layer by layer from the top to the bottom, so that each document belongs to a unique complete path from the root node to the leaf node.

[0008] As a preferred scheme for optimizing audit knowledge retrieval paths based on reinforcement learning, wherein: The process of hierarchically clustering and semantically annotating documents in the audit knowledge base to form a four-level category structure from audit domain to specific rules, and generating a corresponding complete path index for each document, also includes: The common themes of documents within each category are analyzed using a large language model, and corresponding semantic tags are generated as category names; Based on the clustering results, a hierarchical path index is built for each document. The path index is composed of the category numbers of each level in sequence, and a mapping relationship between the path and the document is established, mapping each document in the audit knowledge base to a specific hierarchical semantic path.

[0009] As a preferred scheme for optimizing audit knowledge retrieval paths based on reinforcement learning, wherein: The process involves generating multiple candidate search paths based on the user-input audit query, using a four-layer category structure, through similarity calculation and collaborative filtering. These paths include: When generating multiple candidate search paths, at least one or more of the following strategies are used in combination: Based on the semantic similarity between the query and various categories, the most similar categories are selected layer by layer from the top to the bottom to expand and form a complete path; Based on the co-occurrence relationship between historical queries and paths, similar historical queries are matched for the current query, and corresponding paths are recommended. If a knowledge graph exists, entities are identified from the query, multi-hop reasoning is performed along the relation edges in the graph, and the reasoning results are mapped to hierarchical category paths.

[0010] As a preferred scheme for optimizing audit knowledge retrieval paths based on reinforcement learning, wherein: The process of generating multiple candidate search paths based on the user-input audit query, using a four-layer category structure, through similarity calculation and collaborative filtering, also includes: The discrete category sequence of each generated candidate path is input into a path encoder based on the Transformer architecture. The categories of each layer are converted into embedding vectors through a learnable embedding matrix, and the continuous vector representation of the path is encoded.

[0011] As a preferred scheme for optimizing audit knowledge retrieval paths based on reinforcement learning, wherein: The generated candidate retrieval paths, by comparing the changes in uncertainty of the model's answer distribution before and after retrieval, quantify the information gain brought by each retrieval step, serving as the intrinsic reward signal for reinforcement learning, including: The information gain brought about by each step of the retrieval process is used as the intrinsic reward signal of reinforcement learning, where information gain is defined as the reduction in uncertainty of the large language model regarding the answer distribution before and after retrieval. The information gain is estimated using the following method: Before retrieval, a large language model generates multiple candidate answers for the current query, and a bidirectional textual entailment model is used to perform semantic clustering on the generated candidate answers. Based on the clustering results, the answer distribution entropy before retrieval is estimated. After obtaining evidence through retrieval, the evidence is concatenated with the query to form an enhanced query. Candidate answers are then generated again by the large language model and semantic clustering is performed to estimate the entropy of the retrieved answer distribution. The difference in entropy before and after retrieval is calculated as the information gain reward for the retrieval step; The information gain reward is weighted and combined with the external task reward and the step penalty to form a comprehensive reward function.

[0012] As a preferred scheme for optimizing audit knowledge retrieval paths based on reinforcement learning, wherein: The process of modeling retrieval path selection as a Markov decision process, utilizing the intrinsic reward signal of reinforcement learning, and training a policy network using the GRPO algorithm to learn the optimal path selection policy includes: The audit knowledge retrieval path selection problem is modeled as a Markov decision process, defining states, actions, state transitions, reward functions, and discount factors. The state includes the current query, the history of retrieved paths, the set of evidence obtained, and the entropy of the current answer distribution. The action is to select a path from the candidate path set for retrieval, and the state transition is deterministic.

[0013] As a preferred scheme for optimizing audit knowledge retrieval paths based on reinforcement learning, wherein: The method of modeling retrieval path selection as a Markov decision process, utilizing the intrinsic reward signal of reinforcement learning, training a policy network through the GRPO algorithm, and learning the optimal path selection policy also includes: The policy network is trained using a group relative policy optimization algorithm. For each query, multiple retrieval trajectories are sampled from the current policy, the cumulative reward of each trajectory is calculated, and the advantage function is calculated based on the relative ranking of the cumulative rewards of each trajectory within the same group. The pruned policy objective function is constructed using the aforementioned advantage function and importance sampling ratio, and a KL divergence regularization term is added to constrain the policy update magnitude. During training, an optimal retrieval path selection strategy that maximizes the expected cumulative discount reward is learned through entropy regularization, ε-greedy sampling, and high-quality trajectory mixing sampling in the experience replay buffer.

[0014] As a preferred scheme for optimizing audit knowledge retrieval paths based on reinforcement learning, wherein: During the inference phase, the trained policy network selects multiple paths, retrieves corresponding documents, and sorts them based on a combination of path scores and document scores to generate an interpretable answer, including: During the inference phase, the trained policy network is used to generate multiple candidate paths for each query, and each path corresponds to a set of retrieved documents; Calculate the path score for each path and the document score for each document under the path. The path score is determined by a combination of the semantic similarity between the query and the path, the information gain of the path, and the path diversity. The document score is determined by the product of the path score and the semantic similarity between the query and the document.

[0015] As a preferred scheme for optimizing audit knowledge retrieval paths based on reinforcement learning, wherein: In the inference phase, the trained policy network is used to select multiple paths, retrieve corresponding documents, and then sort them based on a combination of path scores and document scores to generate an interpretable answer. This also includes: Aggregate documents from all paths, remove duplicates, sort by document score, and output a list of the top-ranked documents. The query and the output document list are concatenated to construct an enhanced suggestion, and the large language model is called to generate the final answer; The natural language reasoning model is used to verify the implication relationship between the generated answer and the retrieved document. If the implication probability exceeds a threshold, the answer is confirmed to be credible. An interpretability report is output.

[0016] The beneficial effects of this invention are as follows: By constructing a hierarchical audit knowledge path space and combining it with semantic information gain rewards, this invention can accurately locate relevant audit provisions and cases, improving retrieval accuracy by more than 5% compared to traditional methods, effectively reducing the information screening burden on auditors; by adopting a reinforcement learning optimization strategy based on uncertainty reduction, it can achieve the target answer with fewer retrieval steps (reducing them by 50%-66.7%), significantly shortening audit operation time and improving audit efficiency; the generated hierarchical category paths not only serve as the basis for retrieval but also as the logical support for the audit evidence chain, clearly demonstrating the reasoning process of "why these provisions are retrieved," enhancing the credibility and traceability of audit conclusions; no manual annotation of retrieval process data is required, and the reward is calculated through semantic clustering output by the model itself using the internal evaluator, enabling continuous optimization in audit practice and adapting to updates in audit regulations and new risk scenarios; the end-to-end optimization framework unifies path construction and retrieval optimization under the reinforcement learning framework, reducing the complexity of system implementation and facilitating deployment and application in actual audit business systems. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an overall flowchart of an audit knowledge retrieval path optimization method based on reinforcement learning provided by the present invention.

[0019] Figure 2This is a comprehensive analysis of the experimental results of the reinforcement learning-based audit knowledge retrieval path optimization method in a simulation example provided by the present invention. Detailed Implementation

[0020] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0021] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a method for optimizing audit knowledge retrieval paths based on reinforcement learning, including: S1: Perform hierarchical clustering and semantic annotation on the documents in the audit knowledge base to form a four-level category structure from the audit domain to specific rules, and generate a corresponding complete path index for each document; S2: Based on the audit query input by the user, and based on the four-layer category structure, multiple candidate search paths are generated through similarity calculation and collaborative filtering; S3: Based on the generated candidate retrieval paths, the information gain brought by each retrieval step is quantified by comparing the uncertainty changes of the model's answer distribution before and after retrieval, serving as the intrinsic reward signal for reinforcement learning; S4: Model the retrieval path selection as a Markov decision process, utilize the intrinsic reward signal of reinforcement learning, train the policy network through the GRPO algorithm, and learn the optimal path selection policy. S5: During the inference phase, multiple paths are selected using the trained policy network. After retrieving the corresponding documents, the path scores and document scores are combined and sorted to generate an interpretable answer.

[0022] It should be noted that, through steps S1-S5, this invention constructs an end-to-end audit knowledge retrieval path optimization framework. The audit knowledge base is organized into a hierarchical semantic path space. An intrinsic reward mechanism is designed using information gain, and the retrieval strategy is trained through a group relative strategy optimization algorithm. This enables the system to automatically learn the optimal path sequence from audit queries to relevant evidence without requiring manual annotation of retrieval process data. Not only can it return accurate audit provisions, cases, and answers with fewer retrieval steps and shorter response time, but it can also output complete hierarchical reasoning paths, significantly improving the accuracy, efficiency, and interpretability of retrieval results, providing reliable technical support for intelligent knowledge acquisition in audit practice.

[0023] Example 2, refer to Figure 1 As one embodiment of the present invention, based on the previous embodiment, a method for optimizing audit knowledge retrieval paths based on reinforcement learning is provided, including: In this embodiment, step S1 above involves hierarchical clustering and semantic annotation of documents in the audit knowledge base to form a four-layer category structure from audit domain to specific rules, and generating a corresponding complete path index for each document, including: A hierarchical model was constructed for the audit knowledge base, resulting in a four-layer semantic category structure.

[0024] The audit is divided into four levels: Level 1 (audit domain level), which includes top-level categories such as financial audit, compliance audit, risk audit, and internal control audit; Level 2 (audit subject level), which includes areas such as revenue recognition, cost accounting, and tax compliance; Level 3 (risk point level), which addresses the potential risk types of specific audit targets; and Level 4 (details level), which corresponds to specific audit standards, regulations, or case documents.

[0025] The hierarchical modeling process includes: Each document in the audit knowledge base is encoded using a pre-trained language model, generating a 768-dimensional dense vector representation. Let the document set be... The encoding function is Then the document The vector representation is .

[0026] It should be noted that the pre-trained language model is recommended to use BERT-base (110M parameters) or RoBERTa-large (355M parameters) as the document encoder, and T5-base (220M parameters) or T5-large (770M parameters) as the policy network backbone.

[0027] Hierarchical K-means clustering algorithm is used to perform multi-level clustering of document vectors. The number of clusters in the first layer is... The number of clusters is set to 8-12, corresponding to the main audit areas; the second layer continues clustering within each first-level cluster, with the number of clusters... Set to 5-10; refine the third and fourth layers accordingly, with the number of clusters being respectively... and Cosine similarity is used as the distance metric during clustering.

[0028] Semantic tags are generated for each cluster. Large language models (such as GPT-4 and Claude) are used to analyze the common themes of documents within a cluster and generate concise category names. For example, if a second-level cluster contains a large number of documents about "revenue recognition timing" and "revenue recognition conditions", it is labeled as the "revenue recognition" category.

[0029] Build a hierarchical path index. Each document A complete path from the root node to a leaf node is represented as ,in Indicates the first Layer category number. Establish the mapping relationship between paths and documents. .

[0030] It should be noted that this embodiment preferably uses 4 clustering layers, which can be adjusted to 3-5 layers depending on the size of the knowledge base; the number of clusters in each layer is as follows: Layer 1 K1=8-12, Layer 2 K2=5-10, Layer 3 K3=3-8, Layer 4 K4=2-5. The clustering algorithm is preferably hierarchical K-means, with a maximum number of iterations of 300 and a convergence threshold of 10. -4 .

[0031] In this embodiment, step S2 above, based on the audit query input by the user and a four-layer category structure, generates multiple candidate search paths through similarity calculation and collaborative filtering, including: For a given audit query Multiple candidate paths need to be generated. The following strategy is used for candidate path generation: Strategy 1: Greedy selection based on query-category similarity: For each level Calculate query With all categories in this layer Semantic similarity (parent node is the selected path): The top-K categories with the highest similarity are selected as candidates. This embodiment preferably sets... By selecting layer by layer, generate A complete candidate path.

[0032] Strategy 2: Collaborative filtering based on historical queries: Maintaining a query-path co-occurrence matrix ,in This represents the number of historical queries. This represents the total number of paths. For new queries... Find the most similar For each historical query, the corresponding high-frequency path is recommended.

[0033] Strategy 3: Relational reasoning based on knowledge graphs. If the audit knowledge base has constructed an entity-relationship graph, then starting from the entities identified in the query, multi-hop reasoning is performed along the relationship edges in the graph, mapping the reasoning path to hierarchical category paths.

[0034] To enable paths to be processed by neural network models, discrete category sequences are converted into continuous vector representations. This embodiment employs a path encoder. Implemented based on the Transformer architecture: in For the first The embedding vectors for each layer category are obtained through a learnable embedding matrix. This is the output of the path encoder. This serves as a holistic representation of the path, used for subsequent retrieval and sorting.

[0035] It should be noted that, in this embodiment, the preferred number of Transformer layers is 4-6, the hidden dimension is 512-768, and the number of attention heads is 8-12. The embedding dimension of each category is set to 256-512, and the total path representation dimension is 768-1024.

[0036] In this embodiment, in step S3 above, based on the generated candidate retrieval paths, the information gain brought by each retrieval step is quantified by comparing the changes in uncertainty of the model's answer distribution before and after retrieval. This information gain serves as the intrinsic reward signal for reinforcement learning and includes: This paper introduces the concept of information gain from information theory into reinforcement learning reward design. Information gain is defined as the change in uncertainty of the large language model's belief distribution on the answer before and after the retrieval action. Let the distribution of the large language model's belief on the answer before retrieval be... Evidence retrieved The posterior distribution after that is Information gain Defined as: in For distribution The Shannon entropy is calculated using the following formula: Information gain has three important properties: 1) Non-negativity, that is... The information gain is determined by several factors: 1) ensuring that the retrieval process does not increase uncertainty; 2) additivity, where the total information gain of a multi-step retrieval is equal to the sum of the information gains of each step; and 3) monotonicity, where more relevant evidence leads to a greater information gain. These properties ensure the legitimacy of information gain as a reward signal.

[0037] Furthermore, in practical applications, the actual distribution of answers... Since the information gain is unknown, an intrinsic estimator needs to be designed to estimate it. This method employs a self-supervised estimation strategy based on model output, and the specific steps are as follows: Step 1: For the query Without retrieving any evidence, let the large language model generate... One candidate answer , usually set A temperature sampling strategy is adopted, and the temperature parameters... This is to obtain a diverse distribution of answers.

[0038] Step 2: Perform semantic clustering on the generated answers using a bidirectional textual entailment model. For any two answers... and Calculate the implied probability and If the implied probabilities in both directions are greater than a set threshold (preferably 0.7 in this embodiment), then the two answers are considered semantically equivalent and classified into the same cluster.

[0039] Step 3: Estimate the distribution of answers before retrieval based on clustering results. Assume that after clustering... semantic equivalence classes Then the cluster The probability estimate is: Step 4: Retrieve Evidence Then, the evidence is pieced together into the query to form an enhanced query. Repeat steps 1-3 to obtain the distribution of the retrieved answers. .

[0040] Step 5: Calculate the information gain reward: This reward value reflects the degree to which the retrieval behavior contributes to reducing answer uncertainty, serving as an intrinsic reward signal for reinforcement learning.

[0041] The complete reward function consists of three parts: intrinsic information gain reward, extrinsic task reward, and regularization penalty term; expressed as: in: For intrinsic information gain reward, weight ; As an external task reward, a positive reward is given when the retrieved document correctly answers the query; it is usually set to... Otherwise Weight ; This is a step-counting penalty term, designed to encourage the model to achieve its goal with fewer retrieval steps. The penalty coefficient is... Weight .

[0042] By adjusting the weight parameters, a balance can be achieved between retrieval accuracy and efficiency.

[0043] It should be noted that in this embodiment, the preferred number of sampled answers N=20-50, a larger N improves the estimation accuracy but increases the computational cost; the sampling temperature T=0.8-1.2, a higher temperature increases diversity; and the implication threshold θ=0.7-0.8, a higher threshold makes the clustering more rigorous.

[0044] In this embodiment, step S4 above models the retrieval path selection as a Markov decision process. Utilizing the intrinsic reward signal of reinforcement learning, the policy network is trained using the GRPO algorithm to learn the optimal path selection strategy, including: The optimization problem of audit knowledge retrieval path is modeled as a Markov decision process (MDP), defined as a quintuple. : state space :state Includes the current query Historical retrieval paths Collection of evidence already obtained and the entropy of the current answer distribution The state is represented as: .

[0045] Action space :action Each time step selects a candidate path for retrieval. At each time step, the agent selects from the currently available candidate path set. Choose a path Retrieve relevant documents along this path. The size of the action space depends on the candidate path generation strategy, and is typically [missing information]. .

[0046] State transition probability State transition function Describe the action to be performed The state transition after the initial state change. In this problem, the state transition is deterministic: path selection... Then, the system retrieves the documents corresponding to the path and updates the evidence set and answer distribution.

[0047] reward function As mentioned earlier, the reward function combines intrinsic information gain, extrinsic task reward, and step penalty.

[0048] Discount factor The discount factor is used to balance immediate rewards and long-term returns, and is typically set to... .

[0049] The goal of an intelligent agent is to learn the optimal policy. Maximize the expected value of cumulative discount rewards: The retrieval policy is trained using the Group Relative Policy Optimization (GRPO) algorithm. GRPO is a policy gradient method that does not require a value function. It calculates the advantage function by comparing the relative performance of different trajectories within the same batch, thus avoiding the bias in value function estimation in the traditional Actor-Critic method.

[0050] The core idea of ​​the GRPO algorithm is: for each query ,sampling Different search trajectories (generally ), calculate the cumulative reward for each trajectory Then, the relative ranking within the group is used as the advantage estimate. The specific algorithm flow is as follows: Input: Initial policy Query set Number of samples Learning rate KL divergence constraint Output: Optimized strategy Step 1: For each query Use the current strategy sampling Search trajectory: Step 2: Calculate the cumulative reward for each trajectory: Step 3: Calculate the dominance function within the group. Sort the trajectories by cumulative reward and use the dominance function with ranking normalization: in The average reward for the group. Standard deviation It is a numerically stable term.

[0051] Step 4: Calculate the policy gradient. For the trajectory Each state-action pair Calculate the importance sampling ratio: Step 5: Calculate the objective function of the pruning strategy: Among them, the cutting parameters This prevents excessively large policy updates.

[0052] Step 6: Add a KL divergence regularization term to prevent the new policy from deviating too far from the old policy: in , where is the KL penalty coefficient.

[0053] Step 7: Update policy parameters using the Adam optimizer: Learning rate Initially set to The cosine annealing scheduling strategy is used to gradually decay the ignition.

[0054] Step 8: Repeat steps 1-7 until the policy converges or the maximum number of iterations is reached (usually 10-20 rounds).

[0055] To balance exploration and exploitation during training, this method employs the following strategy: Strategy 1: Entropy Regularization. Add a policy entropy term to the objective function to encourage the policy to maintain a certain degree of randomness. Entropy coefficient Initially set to It gradually decreases as training progresses. .

[0056] Strategy 2: ε-greedy sampling. In the early stages of training, sampling is performed using probability... Randomly select actions, with probability Sampling is performed according to the strategy distribution. from linear decay to .

[0057] Strategy 3: Experience replay buffer. Maintain a buffer with a capacity of... An experience pool is used to store high-quality retrieval trajectories (with accumulated rewards exceeding a threshold). Samples are taken from this experience pool during each training iteration. The data is mixed with the newly sampled data to improve sample utilization efficiency.

[0058] It should be noted that the preferred reinforcement learning training parameters in this embodiment are: Learning rate: Initial learning rate η = 3 × 10 -5 Cosine annealing scheduling is used, with a minimum learning rate of 1×10⁻⁶. -6 .

[0059] Discount factor: γ = 0.95-0.99, higher values ​​place greater emphasis on long-term returns.

[0060] PPO trimming parameter: ϵ=0.2.

[0061] KL penalty coefficient: β = 0.01-0.05.

[0062] Entropy regularization coefficient: Initially α=0.01, decays to 0.001.

[0063] Batch size: 32-64 queries per batch, with 4-8 tracks sampled per query.

[0064] Number of training rounds: 10-20 rounds (adjusted based on convergence).

[0065] In this embodiment, in step S5 above, during the inference phase, multiple paths are selected using the trained policy network, and after retrieving the corresponding documents, the path scores and document scores are combined and sorted to generate an interpretable answer, including: During the inference phase, multiple candidate paths are generated for each query, with each path corresponding to a set of retrieved documents. To comprehensively utilize the information from multiple paths, this embodiment employs a path-aware ranking strategy, including: Step 1: For the query ,generate Candidate paths (generally ).

[0066] Step 2: For each path Retrieve the corresponding document collection And calculate the path score. : in To query the semantic similarity with the path, For the information gain of this path, Path diversity is scored. Weights are set to... , , .

[0067] Step 3: For each document Calculate document score: Step 4: Aggregate documents from all paths, remove duplicates, sort by score, and generate the final Top-N document list (usually...). ).

[0068] Based on the retrieved documents, a large language model is used to generate the final answer, which is then validated. Step 1: Query By concatenating with Top-N documents, an enhanced hint can be constructed: Prompt template: Query: {query} Reference documents: 1. {doc1} 2. {doc2} ... Please answer the query based on the above reference documents and explain the auditing standards or regulations on which you are applying.

[0069] Step 2: Use the large language model to generate the answer. .

[0070] Step 3: Use a Natural Language Inference (NLI) model to verify the implication relationship between the answer and the document. For each document... Calculate the implied probability If at least one document has an implication probability greater than 0.8, the answer is considered reliable.

[0071] Step 4: Generate an interpretability report, including: search path (hierarchical category sequence), key document summary, and the article number on which the answer is based.

[0072] It should be noted that the preferred inference stage parameters in this embodiment are: Number of candidate paths: K = 5-10.

[0073] Maximum number of search steps: T max =5-8.

[0074] Stopping conditions: the answer entropy H < 0.7 or the answer confidence > 0.8 or the maximum number of steps is reached.

[0075] Top-N document count: N = 10-20.

[0076] Example 3, referring to Figure 2 As an embodiment of the present invention, an audit knowledge retrieval path optimization method based on reinforcement learning is provided. To verify the beneficial effects of the present invention, a simulation experiment is conducted for scientific demonstration.

[0077] Scenario 1: Revenue Recognition Retrieval in a Financial Audit Scenario Scenario description: Auditors need to look up "audit procedures and judgment criteria for revenue recognition across periods".

[0078] Step 1: Query Preprocessing The system receives queries The audit procedures and judgment criteria for "revenue recognition across periods" were encoded into a 768-dimensional vector using the BERT model. Simultaneously, query expansion was performed to identify synonyms: "across periods" → "across fiscal years," "across accounting periods," and "revenue recognition" → "revenue realization," "operating revenue."

[0079] Step 2: Generation of Hierarchical Paths The system generates 5 candidate paths: Path 1: Financial Audit → Revenue Recognition → Inter-period Risk → Standard ASC606 Path 2: Financial Audit → Revenue Recognition → Timing Determination → Case 2023-156 Path 3: Financial Audit → Accounting Estimates → Revenue Measurement → Standard CAS14 Path 4: Compliance Audit → Tax Compliance → Revenue Recognition → Tax Law Regulations Path 5: Risk Audit → Fraud Risk → Revenue Manipulation → Identification Procedures Step 3: Initial Retrieval and Information Gain Calculation The system first generates 30 candidate answers in the absence of evidence, then clusters them into 5 semantically equivalence classes using a bidirectional entailment model, and calculates the initial entropy. Then, following path 1, relevant documents for criterion ASC606 are retrieved, the answer is regenerated and clustered into 3 clusters, and the entropy after retrieval is calculated. Information gain A positive reward will be given.

[0080] Step 4: Selection of Reinforcement Learning Strategies Based on the current state (path 1 has been retrieved, entropy is 1.10), the policy network calculates the action probability for each candidate path: path 2 (0.45), path 3 (0.25), path 4 (0.15), and path 5 (0.15). The system selects path 2 to continue the retrieval, obtains the case document, and further reduces the entropy to... Information gain .

[0081] Step 5: Answer Generation and Verification After two steps of retrieval, the system determines that the uncertainty has been sufficiently reduced (entropy). The search was stopped. Based on the retrieved standards and case documents, the large language model generated the answer: "Revenue recognition across accounting periods should follow the five-step model of ASC 606. Key judgment criteria include: 1) Whether the contract spans accounting periods; 2) The timing of the fulfillment of performance obligations; 3) The method of allocating transaction prices. Audit procedures include examining contract terms, testing revenue cutoff, and analyzing the causes of revenue fluctuations." The NLI model verified that the implied probability between the answer and the document was 0.92, confirming the answer's reliability.

[0082] Step 6: Interpretable Output The system outputs the search paths as follows: "Financial Audit → Revenue Recognition → Intertemporal Risk → Standard ASC606" and "Financial Audit → Revenue Recognition → Timing Judgment → Case 2023-156", along with key document summaries and clause numbers, enabling auditors to trace the source of the answers.

[0083] Performance metrics: 2 retrieval steps, 0.8 seconds response time, accuracy verified, interpretability score 4.5 / 5.0.

[0084] Scenario 2: Tax Regulation Search in a Compliance Audit Scenario Scenario description: Auditors need to find out the "latest policies and applicable conditions for additional deduction of R&D expenses".

[0085] Step 1: Query Preprocessing and Path Generation The system identifies the key entities "R&D expenses" and "additional deduction" in the query and generates 6 candidate paths, including path 1: compliance audit → tax compliance → R&D expenses → policy 2024-35, and path 2: financial audit → cost accounting → R&D expenditure → accounting treatment.

[0086] Step 2: Multi-path parallel retrieval The system employs a beam search strategy, exploring the top-3 paths simultaneously. Path 1 retrieves the latest tax policy documents with an information gain of 0.58; Path 2 retrieves accounting standards with an information gain of 0.32; and Path 3 retrieves historical cases with an information gain of 0.25. Based on information gain, the results from Path 1 are prioritized.

[0087] Step 3: Knowledge Graph-Assisted Reasoning The system detected that the entity "R&D expenses" has associations with entities such as "high-tech enterprises" and "technology-based SMEs" in the knowledge graph. The search is expanded along the edges of these graph relationships, supplemented with relevant policy documents, to further improve the completeness of the answer.

[0088] Step 4: Answer Generation and Multi-Document Fusion The system generates answers based on eight documents retrieved through three paths, using a combination of extractive and generative methods. First, key clauses are extracted from the documents, and then a large language model integrates them into a coherent answer, including policy basis, applicable conditions, calculation methods, and precautions.

[0089] Performance metrics: 3 retrieval steps, 1.2 seconds response time, 8 relevant documents recalled, 100% Top-5 accuracy, and interpretability score of 4.8 / 5.0.

[0090] Scenario 3: Complex Multi-hop Reasoning in a Risk Audit Scenario Scenario Description: Auditors need to query "methods for identifying risks of improper transfer of benefits in related-party transactions and audit response measures", which is a complex query that requires multi-hop reasoning.

[0091] Step 1: Complex Query Decomposition The system breaks down complex queries into three sub-questions: 1) What is a related-party transaction? 2) What are the common forms of benefit transfer? 3) What are the audit response measures? Candidate paths for each sub-query are generated.

[0092] Step 2: Iterative Search and Path Optimization Round 1: Retrieve the definition of "related party transaction" to obtain basic concept documents, information gain 0.35.

[0093] Round 2: Based on the acquired knowledge of related-party transactions, search for "forms of benefit transfer" to obtain case documents, with an information gain of 0.48.

[0094] Round 3: Combining the information from the first two rounds, retrieve "audit response measures" to obtain audit procedure documents, with an information gain of 0.52.

[0095] Round 4: The system determined that "pricing fairness" related content needed to be supplemented, and continued the search, with an information gain of 0.28.

[0096] Step 3: Adaptive Adjustment of Reinforcement Learning Strategies After the third round of retrieval, the system calculates the cumulative information gain. The search threshold of 1.2 has been exceeded, but the answer confidence score is only 0.72 (below the 0.8 threshold). The policy network decides to continue searching, demonstrating the adaptive ability of reinforcement learning. After the fourth round of searching, the answer confidence score increases to 0.89, meeting the stopping condition.

[0097] Step 4: Demonstrating the interpretability of the hierarchical path The system outputs the complete inference path tree: Root node: Risk audit Level 1: Related Party Transaction Risks → Definition and Identification → Standard CAS36 Layer 2: Related Party Transaction Risks → Transfer of Benefits → Case Study Library 2022-089 Level 3: Related Party Transaction Risks → Audit Procedures → Practice Guidelines Chapter 8 Layer 4: Related Party Transaction Risks → Pricing Fairness → Compilation of Valuation Methods This hierarchical presentation clearly demonstrates the system's reasoning logic, enabling auditors to understand the purpose and basis of each step of the search.

[0098] Performance metrics: 4 retrieval steps, 2.1 seconds response time, 12 relevant documents retrieved, answer completeness score 4.6 / 5.0, interpretability score 4.9 / 5.0. Compared to traditional methods requiring 8-10 retrieval steps, this method improves efficiency by over 50%.

[0099] Scenario 4: System Training and Continuous Optimization Scenario description: Demonstrate how the system can continuously learn and optimize from auditing practices through reinforcement learning.

[0100] Step 1: Initialization and Pre-training The system uses a labeled dataset containing 5000 audit query-answer pairs for supervised pre-training. The policy network is initialized based on the T5-large model, and the path encoder is initialized using BERT-base. Cross-entropy loss is used during the pre-training phase, and the learning rate is... Train for 5 epochs.

[0101] Step 2: Fine-tuning reinforcement learning The GRPO algorithm was used for 10 rounds of iterative optimization. In each round, 500 new queries were sampled from the audit knowledge base, with 4 trajectories sampled for each query. The average cumulative reward was 3.2 in round 1, improving to 5.8 in round 5, and converging to 6.4 in round 10. The retrieval accuracy improved from an initial 75.3% to 87.1%, and the average number of retrieval steps decreased from 6.8 steps to 2.8 steps.

[0102] Step 3: Online Learning and Knowledge Base Updates After system deployment, feedback from auditors is continuously collected. When auditors correct or supplement the search results, the system records this interaction as a new training sample. Incremental learning is triggered every 100 new samples accumulated, using a relatively small learning rate. Fine-tune the model to avoid catastrophic amnesia.

[0103] Step 4: Dynamic Expansion of the Knowledge Graph When audit regulations are updated (e.g., new standards are released), the system automatically encodes the new document and inserts it into the hierarchical path space. An incremental clustering algorithm is used to determine the category to which the new document should belong. If the similarity between the new document and any existing category is below the threshold of 0.6, a new subcategory is created. Simultaneously, the knowledge graph is updated, adding new entities and relationships.

[0104] Step 5: A / B Testing and Performance Monitoring After system deployment, a three-month A / B test was conducted. The control group used traditional keyword retrieval methods, while the experimental group used the method of this invention. Results showed that auditors in the experimental group experienced a 58% reduction in average retrieval time, a 16% increase in retrieval accuracy, and a 42% increase in satisfaction with the system's interpretability. Based on the test results, the system was fully rolled out and applied.

[0105] Performance metrics: Training convergence time approximately 48 hours (using 4 A100 GPUs), online learning response time Minutes, knowledge base update delay The system availability is 99.5% for 24 hours.

[0106] To fully verify the effectiveness of the method of this invention, this embodiment also designed a large-scale comparative experiment. The experiment was conducted on a standard audit knowledge base, which contains 15,000+ audit provisions and 8,000+ case documents, covering eight major audit scenarios, including financial audit, compliance audit, risk audit, and internal control audit. The experiment generated 500 test queries, which were divided into four levels of complexity: simple (20%), medium (40%), complex (30%), and extremely complex (10%), to comprehensively examine the performance of the three methods in different scenarios.

[0107] The experiment compared three retrieval methods: 1) an inverted index method based on TF-IDF and a semantic similarity model; 2) a knowledge graph method employing entity recognition and relation reasoning; and 3) the reinforcement learning path optimization method of this invention, based on semantic information gain reward and a hierarchical path space. Each method was run on the same test set, and key metrics such as retrieval accuracy, retrieval steps, response time, Top-5 recall, and interpretability score were recorded. Furthermore, 10 rounds of iterative training experiments were conducted to observe the learning convergence characteristics of each method.

[0108] Figure 2 The comprehensive analysis results of the experiments are presented, including nine subplots comparing the performance of the three methods from different dimensions. The experimental data underwent rigorous statistical significance testing (t-test), and the results show that the method of this invention has a highly significant performance advantage (p<0.001). Specifically, this includes: 1. Significantly Improved Retrieval Accuracy (Sub-figure A): The average retrieval accuracy of the method described in this invention reaches 81.52% (±8.91%), which is 15.81% higher than the 70.39% of the knowledge graph method and 26.24% higher than the 64.58% of the inverted index method. The violin plot and box plot show that the method described in this invention not only has a higher mean but also a more concentrated distribution, indicating better performance stability. Statistical tests show a highly significant difference (p<0.001), proving that the method described in this invention has a substantial advantage in accuracy.

[0109] 2. Verification of the effectiveness of the information gain mechanism (sub-figure B): Information gain and uncertainty reduction show a strong positive correlation. The data points of the method of this invention are distributed in the upper right region, indicating that it can achieve greater uncertainty reduction through higher information gain. This directly verifies the core innovation of this invention—the reward mechanism based on semantic information gain can effectively guide the optimization of the retrieval path, enabling the system to minimize the uncertainty of auditors' risk assessment at each step of the retrieval.

[0110] 3. Rapid Convergence and Learning Ability (Subgraph C): The learning curve shows that the method of this invention converges to a high-performance level (above 87%) after 3-4 iterations, while knowledge graph methods require 7-8 iterations to reach a similar level, and inverted index methods have virtually no learning ability. This proves that the reinforcement learning framework of this invention has excellent sample efficiency, can quickly learn and optimize retrieval strategies from auditing practice, and adapt to dynamically changing audit knowledge bases.

[0111] 4. Significantly Improved Retrieval Efficiency (Subgraph E): The Cumulative Distribution Function (CDF) shows that the method of this invention requires no more than 5 retrieval steps in 90% of queries (median 3 steps), while the knowledge graph method requires 9 steps and the inverted index method requires 11.68 steps. This means that the method of this invention can achieve higher accuracy with 62.70% fewer retrieval steps, significantly shortening the auditing operation time and verifying the actual effect of path optimization.

[0112] 5. Robustness in Complex Scenarios (Subgraph H): The method of this invention maintains optimal performance across scenarios of varying complexity. Particularly in "extremely complex" scenarios, the accuracy drop of the method of this invention is minimal (approximately 15%), while the comparative method experiences a drop of over 30%. This demonstrates that the hierarchical path space and reinforcement learning optimization mechanism can effectively address complex multi-hop inference requirements and maintain high accuracy even in challenging auditing scenarios.

[0113] 6. Optimal Trade-off Between Response Time and Accuracy (Subgraph G): The average response time of the method in this invention is only 0.46 seconds, far lower than the 2.04 seconds of the knowledge graph method and the 1.41 seconds of the inverted index method, while achieving the highest accuracy. The scatter plot shows that the data points of the method in this invention are located in the upper left corner (low time, high accuracy), achieving Pareto optimality, proving that the method in this invention achieves the best balance between efficiency and effectiveness.

[0114] 7. Explainability Advantage (Sub-graph F): The radar chart shows that the method of this invention scores 4.59 / 5.0 in the explainability dimension, significantly higher than the comparative methods. This is due to the generation of hierarchical category paths. The system not only returns search results, but also provides a complete reasoning path of "audit area → audit subject → risk point → specific details", enabling auditors to clearly understand the search basis and enhance the credibility of audit conclusions.

[0115] In summary, large-scale comparative experiments have fully verified the technical advantages of the method of this invention. By constructing a hierarchical audit knowledge path space and adopting a reinforcement learning optimization strategy based on semantic information gain, the method of this invention significantly outperforms existing technologies in all key indicators, including retrieval accuracy, retrieval efficiency, response speed, and interpretability. Statistical significance tests (p<0.001) and multi-dimensional performance analysis demonstrate that the method of this invention is not only theoretically innovative but also exhibits excellent performance and robustness in practical applications. It can effectively solve the path optimization problem in audit knowledge retrieval, providing efficient, accurate, and interpretable intelligent retrieval support for audit operations.

[0116] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for optimizing audit knowledge retrieval paths based on reinforcement learning, characterized in that, include: The documents in the audit knowledge base are hierarchically clustered and semantically annotated to form a four-level category structure from audit domain to specific rules, and a corresponding complete path index is generated for each document; Based on the audit query input by the user, and based on a four-layer category structure, multiple candidate search paths are generated through similarity calculation and collaborative filtering. Based on the generated candidate retrieval paths, the information gain brought by each retrieval step is quantified by comparing the uncertainty changes in the model's answer distribution before and after retrieval, serving as an intrinsic reward signal for reinforcement learning. The retrieval path selection is modeled as a Markov decision process. The intrinsic reward signal of reinforcement learning is used to train the policy network through the GRPO algorithm to learn the optimal path selection policy. During the reasoning phase, multiple paths are selected using the trained policy network. After retrieving the corresponding documents, the path scores and document scores are combined and sorted to generate an interpretable answer.

2. The audit knowledge retrieval path optimization method based on reinforcement learning as described in claim 1, characterized in that, The process involves hierarchical clustering and semantic annotation of documents in the audit knowledge base, forming a four-layer category structure from audit domains to specific rules, and generating a corresponding complete path index for each document, including: The documents in the audit knowledge base are modeled hierarchically, and a four-layer semantic category structure is constructed, which includes at least the audit domain layer, audit subject layer, risk point layer and detailed rule layer. In this process, each document is encoded into a dense vector using a pre-trained language model; A hierarchical clustering algorithm is used to perform multi-level clustering of document vectors, refining the clusters layer by layer from the top to the bottom, so that each document belongs to a unique complete path from the root node to the leaf node.

3. The audit knowledge retrieval path optimization method based on reinforcement learning as described in claim 2, characterized in that, The process of hierarchically clustering and semantically annotating documents in the audit knowledge base to form a four-level category structure from audit domain to specific rules, and generating a corresponding complete path index for each document, also includes: The common themes of documents within each category are analyzed using a large language model, and corresponding semantic tags are generated as category names; Based on the clustering results, a hierarchical path index is built for each document. The path index is composed of the category numbers of each level in sequence, and a mapping relationship between the path and the document is established, mapping each document in the audit knowledge base to a specific hierarchical semantic path.

4. The audit knowledge retrieval path optimization method based on reinforcement learning as described in claim 3, characterized in that, The process involves generating multiple candidate search paths based on the user-input audit query, using a four-layer category structure, through similarity calculation and collaborative filtering. These paths include: When generating multiple candidate search paths, at least one or more of the following strategies are used in combination: Based on the semantic similarity between the query and various categories, the most similar categories are selected layer by layer from the top to the bottom to expand and form a complete path; Based on the co-occurrence relationship between historical queries and paths, similar historical queries are matched for the current query, and corresponding paths are recommended. If a knowledge graph exists, entities are identified from the query, multi-hop reasoning is performed along the relation edges in the graph, and the reasoning results are mapped to hierarchical category paths.

5. The audit knowledge retrieval path optimization method based on reinforcement learning as described in claim 4, characterized in that, The process of generating multiple candidate search paths based on the user-input audit query, using a four-layer category structure, through similarity calculation and collaborative filtering, also includes: The discrete category sequence of each generated candidate path is input into a path encoder based on the Transformer architecture. The categories of each layer are converted into embedding vectors through a learnable embedding matrix, and the continuous vector representation of the path is encoded.

6. The audit knowledge retrieval path optimization method based on reinforcement learning as described in claim 5, characterized in that, The generated candidate retrieval paths, by comparing the changes in uncertainty of the model's answer distribution before and after retrieval, quantify the information gain brought by each retrieval step, serving as the intrinsic reward signal for reinforcement learning, including: The information gain brought about by each step of the retrieval process is used as the intrinsic reward signal of reinforcement learning, where information gain is defined as the reduction in uncertainty of the large language model regarding the answer distribution before and after retrieval. The information gain is estimated using the following method: Before retrieval, a large language model generates multiple candidate answers for the current query, and a bidirectional textual entailment model is used to perform semantic clustering on the generated candidate answers. Based on the clustering results, the answer distribution entropy before retrieval is estimated. After obtaining evidence through retrieval, the evidence is concatenated with the query to form an enhanced query. Candidate answers are then generated again by the large language model and semantic clustering is performed to estimate the entropy of the retrieved answer distribution. The difference in entropy before and after retrieval is calculated as the information gain reward for the retrieval step; The information gain reward is weighted and combined with the external task reward and the step penalty to form a comprehensive reward function.

7. The audit knowledge retrieval path optimization method based on reinforcement learning as described in claim 6, characterized in that, The process of modeling retrieval path selection as a Markov decision process, utilizing the intrinsic reward signal of reinforcement learning, and training a policy network using the GRPO algorithm to learn the optimal path selection policy includes: The audit knowledge retrieval path selection problem is modeled as a Markov decision process, defining states, actions, state transitions, reward functions, and discount factors. The state includes the current query, the history of retrieved paths, the set of evidence obtained, and the entropy of the current answer distribution. The action is to select a path from the candidate path set for retrieval, and the state transition is deterministic.

8. The audit knowledge retrieval path optimization method based on reinforcement learning as described in claim 7, characterized in that, The method of modeling retrieval path selection as a Markov decision process, utilizing the intrinsic reward signal of reinforcement learning, training a policy network through the GRPO algorithm, and learning the optimal path selection policy also includes: The policy network is trained using a group relative policy optimization algorithm. For each query, multiple retrieval trajectories are sampled from the current policy, the cumulative reward of each trajectory is calculated, and the advantage function is calculated based on the relative ranking of the cumulative rewards of each trajectory within the same group. The pruned policy objective function is constructed using the aforementioned advantage function and importance sampling ratio, and a KL divergence regularization term is added to constrain the policy update magnitude. During training, an optimal retrieval path selection strategy that maximizes the expected cumulative discount reward is learned through entropy regularization, ε-greedy sampling, and high-quality trajectory mixing sampling in the experience replay buffer.

9. The audit knowledge retrieval path optimization method based on reinforcement learning as described in claim 8, characterized in that, During the inference phase, the trained policy network selects multiple paths, retrieves corresponding documents, and sorts them based on a combination of path scores and document scores to generate an interpretable answer, including: During the inference phase, the trained policy network is used to generate multiple candidate paths for each query, and each path corresponds to a set of retrieved documents; Calculate the path score for each path and the document score for each document under the path. The path score is determined by a combination of the semantic similarity between the query and the path, the information gain of the path, and the path diversity. The document score is determined by the product of the path score and the semantic similarity between the query and the document.

10. The audit knowledge retrieval path optimization method based on reinforcement learning as described in claim 9, characterized in that, In the inference phase, the trained policy network is used to select multiple paths, retrieve corresponding documents, and then sort them based on a combination of path scores and document scores to generate an interpretable answer. This also includes: Aggregate documents from all paths, remove duplicates, sort by document score, and output a list of the top-ranked documents. The query and the output document list are concatenated to construct an enhanced suggestion, and the large language model is called to generate the final answer; The natural language reasoning model is used to verify the implication relationship between the generated answer and the retrieved document. If the implication probability exceeds a threshold, the answer is confirmed to be credible. An interpretability report is output.

Images