A reinforcement learning method for electronic forensics heterogeneous time sequence evidence chain graph exploration

Abstract

The application discloses a reinforcement learning method for electronic forensics heterogeneous time sequence evidence chain graph exploration, which receives a forensics query, calculates evidence gaps and focuses from an admitted evidence chain graph, encodes four-layer nodes, typed relations and time attributes by using a heterogeneous graph neural network, fuses a remaining budget, a processing footprint and a verification residual to generate a state vector, generates a dynamic action mask and an executable action set according to the gap type, the focus, the mode, the authority and the budget, selects an action in a constrained action space by a strategy network, performs admission verification on a candidate increment after the environment executes, calculates a multi-objective dynamic reward according to an admission closure reward, a clue discovery reward, an invalid candidate penalty, an actual cost and a step penalty, updates a state to form a transition sample, outputs a result with a proof path when a query condition is met, and outputs an evidence deficiency report otherwise. The application effectively improves the exploration efficiency and strategy convergence by dynamically masking the action space and guiding the evidence chain closure through the multi-objective reward.

Description

Technical Field

[0001] This invention relates to the field of electronic forensics, and in particular to a reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics. Background Technology

[0002] Electronic forensics activities revolve around electronic data generated by carriers such as computers, mobile terminals, servers, network devices, and cloud platforms, encompassing data collection, preservation, examination, analysis, and presentation. A single case often involves the simultaneous presence of heterogeneous materials such as device images, file systems, application databases, instant messaging records, emails, financial transaction logs, system logs, and documents, tables, images, and audio / video recordings. The precise location methods for each type of material differ, including file paths, database tables and primary keys, message identifiers, table row and column coordinates, image regions, or audio / video timecodes. The fundamental requirements for forensic analysis are to locate case-related information within massive amounts of material while maintaining the integrity of the electronic data, the traceability of its original source, the verifiability of the acquisition process, and the traceability of its precise location.

[0003] Heterogeneous time-series graphs provide a modeling foundation for structured data that simultaneously represents multiple node types, relationship types, and their evolution over time. These graph models typically set separate feature mappings and message passing rules for different node and relationship types, and combine techniques such as graph traversal, path search, subgraph retrieval, link prediction, time-series modeling, and incremental updates to extract structural features. In electronic forensics scenarios, heterogeneous time-series graphs can incorporate information from different semantic layers, such as original document scope, analytical observations, evidentiary claims, entities, and events, into a single graph structure, recording event occurrence time, material acquisition time, analytical time, and graph update time respectively. This provides a unified data foundation for structured associations and state tracking between multi-source materials. When forensic analysis requires performing multi-step location, evidence collection, source tracing, and verification operations on this type of graph, the choice of each action depends on the results of previous steps and the unmet evidentiary requirements in the current graph, thus constituting a typical sequential decision problem.

[0004] Reinforcement learning provides a systematic learning framework for solving sequential decision-making problems. In this framework, the agent selects an action based on the current environmental state, and the environment returns an immediate reward and subsequent state after the action is executed. The training process continuously updates policy parameters through transition samples composed of states, actions, rewards, and subsequent states. When reinforcement learning is applied to graph-structured environments, it faces several unique challenges: First, the action space dynamically expands or shrinks with changes in the graph's topology, requiring the use of action masks or parameterized action representations to limit the currently executable range; second, the heterogeneity of graph nodes and relationships requires state encoding to distinguish different types of structural information; finally, achieving the goal often requires multiple steps, with sparse and delayed feedback from intermediate steps, necessitating carefully designed reward functions to provide effective training signals.

[0005] Regarding strategy learning mechanisms, manual analysis, retrieval enhancement generation, and graph retrieval methods with fixed strategies primarily rely on human experience, pre-defined processes, or single-round relevance ranking to select the next operation. They lack a learning mechanism that continuously adjusts subsequent strategies based on the results of preceding actions. When faced with the multi-step process of locating, gathering evidence, tracing the source, and verifying evidence required for the same query, these methods struggle to comprehensively consider the current progress of evidence, the long-term benefits of each candidate action, and the constraints of remaining resources. This can easily lead to problems such as fixed-sequence processing, repeated global searches, premature cessation of exploration, or continuous consumption of budget on low-value materials.

[0006] Regarding action space construction, existing graph retrieval schemes, even if they maintain the graph structure during exploration, typically fail to encode the topology of the heterogeneous temporal evidence chain graph, the current evidence gap, graph focus, processing footprint, verification failure reasons, and remaining budget into reinforcement learning states. Action selection also lacks a legal action domain that is dynamically recalculated with state changes. When all action types, such as localization, navigation, expansion, evidence gathering, source tracing, and verification, are always open, the policy network needs to evaluate and experiment with a large number of actions that are irrelevant to the current gap, incompatible in type, involve repetitive processing, exceed the authorized scope, or exceed the budget. This leads to a reduced proportion of effective explorations, an increase in invalid component calls, and significantly delays policy convergence.

[0007] Regarding reward function design, existing solutions lack multi-layered training feedback suitable for evidence graph exploration. Using only semantic relevance, the number of new nodes, or the number of candidate edges as rewards fails to distinguish between the fundamentally different progress states of "discovering new clues" and "forming verified formal evidence," easily inducing the strategy to repeatedly expand similar nodes or derived content. Using only whether an answer is ultimately obtained as a reward lacks timely training signals for the intermediate location and evidence-gathering steps. Furthermore, if constraint verification failures and actual resource consumption (including read volume, computation volume, component call counts, and manual review volume) do not enter the reward and subsequent states, the strategy struggles to learn to avoid the same failure paths or adjust its exploration direction in a timely manner when budget is insufficient. Summary of the Invention

[0008] The purpose of this invention is to address the aforementioned problems by providing a reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics. This method uses a dynamic action mask driven by evidence gaps to restrict each legal action to a range relevant to the current gap and focus, satisfying constraints such as permissions, budget, and history, significantly reducing the number of decision branches. It employs a heterogeneous graph neural network to encode a four-layer separation structure and proof / navigation edge differences, ensuring the state vector accurately reflects evidence progress. By designing a multi-objective dynamic reward function including admission closure potential difference, gap closure bonus, clue quota reward, multi-class verification residual penalty, and actual cost penalty, the agent is guided to prioritize action paths that can pass strict admission verification and close evidence gaps, while simultaneously suppressing redundant exploration, invalid access, and excessive budget consumption. This invention enables autonomous learning and optimization of multi-step positioning, evidence collection, tracing, and verification strategies while ensuring the integrity, traceability, and backtrackability of electronic data, thereby improving the exploration efficiency and reliability of results on large-scale heterogeneous temporal evidence chains.

[0009] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0010] According to one aspect of the present invention, a reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics is provided, comprising the following steps:

[0011] S1. Receive and parse the user's evidence query, and calculate the admitted evidence set, candidate set, evidence gap set and graph focus from the heterogeneous temporal evidence chain graph that has completed evidence admission verification;

[0012] S2. The four-layer nodes, typified relationships, time attributes and claim states of the heterogeneous temporal evidence chain graph are heterogeneously encoded, and the calculated evidence gap set and graph focus, as well as the remaining budget, original document processing footprint, and structured verification residuals from the previous round, are fused together to obtain the state vector.

[0013] S3. Generate a dynamic action mask and a set of executable actions based on the gap closure conditions, graph focus, graph mode, processing footprint, permissions, and budget.

[0014] S4. The reinforcement learning policy network selects the bound action from the set of executable actions;

[0015] S5. After the environment executes the bound action, it forms a candidate graph increment. The environment performs admission verification on the candidate increment and returns the admission result, candidate clue changes, updated heterogeneous temporal evidence chain graph, structured verification residuals and actual costs.

[0016] S6. Calculate multi-objective dynamic rewards based on the following items: admission and closure rewards, clue discovery rewards, invalid candidate penalties, actual exploration costs, and step penalties.

[0017] S7. Update the evidence gap set, graph focus, remaining budget and original document processing footprint, and the next round of action mask to form a transfer sample;

[0018] S8. When the updated heterogeneous temporal evidence chain graph satisfies the query response conditions, output the query result with the original proof path; when the remaining budget is exhausted or there are no action to be performed, output the evidence insufficiency report and the typified evidence gap.

[0019] Preferably, in step S1,

[0020] The current user's evidence retrieval status is indicated as follows:

[0021]

[0022] in, For the first The current query status during round-robin decision-making; For users' evidence collection and inquiry; The set of evidence that has been approved and is permitted to be used in the response; This is a candidate set that can be further explored but has not yet been granted access; This is a set of unmet evidence gaps. The focus of the graph corresponding to the current priority gap; For the remaining budget; Process the footprints of the original document; This is the residual from the previous round of structured verification.

[0023] Preferably, in step S2,

[0024] The four-layer nodes include the original document scope node, observation node, evidence claim node, and entity, event, and slot node;

[0025] The heterogeneous graph encoding sets different initial feature mappings for different node types, and the initial features of the nodes are represented as follows:

[0026]

[0027] in, For nodes Type identifier; This is the projection matrix corresponding to the node type; For nodes The original feature vector;

[0028] The state vector is generated in the following way:

[0029] For each gap, a gap vector is generated. Using the target node or focus of the gap as the attention query, the conditional graph representation of the gap is obtained.

[0030]

[0031] in, This is the attention pooling function; The gap vector; The set of node features to be aggregated; For the process The output vector of a layered graphical neural network after message passing; To focus Centered on the gap, within a preset number of jumps Subgraphs reachable by type-related relationships;

[0032] The final state is:

[0033]

[0034] in, For the first The complete state vector of the wheel; This is the fusion function; The conditional graph representation of the highest priority gap; This is the highest priority gap at present; For global graph pooling representation; This is an embedded function.

[0035] Preferably, in step S3,

[0036] The dynamic action mask consists of the product of seven binary sub-masks, represented as:

[0037]

[0038] in, For the first Round of candidate actions The overall action mask value; For the gap mask; As a focus mask; For pattern mask; To prove the mask; This is a permission mask; For budget mask; To mask history;

[0039] The set of executable actions is represented as follows:

[0040]

[0041] in, No. The legally executable action domain of the wheel; A generator for the complete set of candidate actions; To stop the exploration action.

[0042] Preferably, in step S4,

[0043] The reinforcement learning policy network calculates the log probability for the bound action, as follows:

[0044]

[0045] in, For action The logarithmic probability; For policy networks; For action The embedding vector;

[0046] The final output of the reinforcement learning policy network is:

[0047]

[0048] in, For the state vector and mask Under the given conditions, select the action. The probability of the strategy; For action Exponentialized scores; This is to iterate through all candidate actions.

[0049] Preferably, in step S5,

[0050] The environment performs admission verification on candidate increments, as expressed by the following formula:

[0051]

[0052] in, This is the admission verification function; Increment the candidate graph; For admission results; This is the updated heterogeneous temporal evidence chain diagram; This refers to actual resource consumption.

[0053] Preferably, in step S6,

[0054] The multi-objective dynamic reward is represented as follows:

[0055]

[0056] in, For the first The total reward obtained per step by the reinforcement learning strategy network; , , , and These are dynamic weighting coefficients; For entry and exit rewards; A reward will be given for finding clues; Penalty for invalid candidates.

[0057] Preferably, the dynamic weighting coefficients are calculated by a deterministic environment function, expressed as:

[0058]

[0059] in, For deterministic weighted scheduling functions; The type of gap with the highest priority; The priority value for the highest priority gap; The degree to which the gap is satisfied in the current official graph; This represents the percentage of the remaining budget. This marks the current exploratory phase.

[0060] Preferably, in step S6,

[0061] The admission and closure rewards are represented as follows:

[0062]

[0063] in, Let the closure potential function be the one that allows entry. For the first A formal heterogeneous temporal evidence chain diagram prior to the execution of the wheel action; For the first A formal heterogeneous temporal evidence chain diagram after the execution of the wheel action; The base amount for closing the single gap bonus; For the specific gaps in the traversal; For the gap The weights; For indicator functions; For the gap In the official diagram Satisfaction level; For the gap In the official diagram Satisfaction level; The bonus will be terminated for the entire company. For query response status indicators;

[0064] The reward for finding clues is expressed as follows:

[0065]

[0066] in, This represents the remaining reward quota for clues in the current query. This is the set of nodes that have newly entered the candidate set in this round; As a new candidate node; For relevance; To ensure stable availability of identifiers; For novelty; For subsequent executability;

[0067] The penalty for invalid candidates is expressed as follows:

[0068]

[0069] in, For the first The total penalty value resulting from the failure of the candidate increment to pass the admission verification; This is a dynamic penalty weight vector; To verify the residual vector in a structured manner;

[0070] The structured verification residual vector is represented as:

[0071]

[0072] in, The hashes are inconsistent. Location failure; The fracture originates from the root. The parsing version does not meet the requirements; The time constraint is not met; Unauthorized access; Unable to be reliably connected; Graph pattern violation; This is a repetition of the same root.

[0073] Preferably, in step S7,

[0074] The transferred sample is represented as:

[0075]

[0076] in, This is the state before a decision is made; This is the set of legal actions taken before a decision is made. The actions actually performed by the intelligent agent; For instant rewards; This is the state after the decision; It is the set of legal actions following a decision; For termination status indicator; This is the admission result.

[0077] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0078] 1. This invention addresses the exploration process of heterogeneous temporal evidence chain graphs in electronic forensics scenarios. It designs a multi-objective dynamic reward function to guide reinforcement learning agents in balancing evidence chain closure, exploration cost control, and constraint compliance. The multi-objective reward uses formal evidence admission and evidence gap closure as the main positive feedback signals, avoiding misinterpreting the number of new candidates, semantic similarity, or navigation path length as evidence progress. Clue discovery only receives a lightweight reward with a cumulative cap; navigation edges, unadmitted increments, and derived content do not generate closure rewards, thus suppressing the formation of false proof paths from the reward level. Constraint verification failures and actual resource consumption both enter into negative feedback, enabling the strategy to learn to avoid high-cost, ineffective actions.

[0079] 2. The dynamic action mask of this invention masks actions unrelated to the current gap based on the type of evidence gap and the focal position. The policy network selects only from the set of legal actions, effectively reducing the number of decision branches at each step. The mask, as a hard constraint of the environment, is independent of the policy parameters, ensuring that illegal actions cannot be selected during both the training and deployment phases.

[0080] 3. This invention maps the four-layer separation structure of the heterogeneous temporal evidence chain graph in each round into a continuous state representation through a heterogeneous graph neural network, and integrates it with the current evidence gap set, focal position, remaining budget, and verification residual into a unified state vector. This ensures that each step of exploration, evidence collection, or verification is constrained by the current evidence gap and graph structure, reducing ineffective exploration on large-scale heterogeneous graphs and accelerating policy convergence. Attached Figure Description

[0081] Figure 1 This is a schematic diagram of the overall process flow of the present invention;

[0082] Figure 2 This is a schematic diagram of the heterogeneous graph state encoding and dynamic action mask of the present invention;

[0083] Figure 3 This is a schematic diagram of the multi-objective dynamic reward function structure of the present invention;

[0084] Figure 4 This is a schematic diagram illustrating an example of the evidence gap-driven reward calculation method of the present invention. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and preferred embodiments. However, it should be noted that many details listed in the specification are merely to provide the reader with a thorough understanding of one or more aspects of the invention, and these aspects of the invention can be achieved even without these specific details.

[0086] Please see Figures 1 to 3 This invention provides a reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics. The technical solution is as follows:

[0087] S1. Parse the query and calculate the initial state elements.

[0088] Specifically, it receives and parses the user's evidence query, and calculates the admitted evidence set, candidate set, evidence gap set, and graph focus from the heterogeneous temporal evidence chain graph that has completed evidence admission verification. The user's evidence query is denoted as... The current query status is represented as:

[0089]

[0090] For the first The current query status during round-robin decision-making; The set of evidence that has been approved and is permitted to be used in the response; This is a candidate set that can be further explored but has not yet been granted access; For the set of evidence gaps that have not yet been met, each gap should at least record the target claim or event slot, gap type (source closure, time closure, conflict description or candidate range), satisfaction level, closure condition and priority; The focus of the graph corresponding to the current priority gap; Budget for remaining time, I / O, computation, component calls, and manual review; Process the footprints of the original document; This is the residual from the previous round of structured verification.

[0091] The heterogeneous temporal evidence chain graph that has completed evidence admission verification is a heterogeneous temporal evidence chain graph that satisfies the four-layer separation structure and proof path admission constraints. This heterogeneous temporal evidence chain graph is used as a reinforcement learning environment. The four-layer separation structure includes the original scope layer, the observation layer, the evidence claim layer, and the entity / event and slot layer. The relationships in the graph are divided into mutually exclusive navigation edge sets and proof / constraint edge sets.

[0092] The admitted evidence set consists of reliable evidence that has been approved to answer queries. The candidate set consists of potential material locations that have been discovered but have not yet passed admission verification. The evidence gap set identifies which key pieces of evidence are currently missing (e.g., unclosed sources, unexplained time conflicts), which are the core objectives driving subsequent actions. The graph focus is the specific location on the graph corresponding to the highest priority gap, used to guide the direction of exploration.

[0093] It should be noted that this embodiment does not involve the construction and update rules of the graph itself, but rather learns and explores strategies on an already constructed graph. The policy network is not directly written into the formal graph. After an action is executed, only an increment in the candidate graph is generated. The environment verifies the incremental graph pattern, component hashing, precise location, root source, parsing version, access permissions, entity identification, time constraints, edge type, and claim state transitions. Only incremental changes that pass all verifications are submitted to the updated official graph. .

[0094] S2. Construct heterogeneous sequence diagram state coding.

[0095] Specifically, heterogeneous graph neural networks are used to encode the four-layer structure (original layer, observation layer, claim layer, and entity layer) of the heterogeneous temporal evidence chain graph, and navigation edges (such as similarity) and proof edges (such as support / refutation) are strictly distinguished.

[0096] For different node types Set different initial feature mappings. The original scope node (original layer) should at least encode the material type, normalized locator, hash availability, processed granularity, and access status; the observation node (observation layer) should at least encode the parsed component, version, precise location, time field, and source closure status; the evidence claim node (claim layer) should at least encode the normalized assertion, support / rebuttal / dispute / unknown status, and applicable time; the entity / event and slot node (entity layer) should at least encode the object type, slot completeness, time interval, number of contested fills, and query relevance.

[0097] node The initial representation is:

[0098]

[0099] in, For nodes Type identifier; This is the projection matrix corresponding to the node type; For nodes The original feature vector.

[0100] The above formula is used to represent each original node in the heterogeneous temporal evidence chain graph. The original feature vector By mapping the node to a unified latent space through linear projection, the initial embedding representation of the node is obtained. This serves as the starting point for subsequent message passing in the graph neural network.

[0101] Heterogeneous message passing is as follows:

[0102]

[0103] in, For nodes In the The hidden state feature vector of the next layer; It is a non-linear activation function; For nodes In the The current feature vector of the layer; For the first Within the layer, there are specific options for node types. The weight matrix; For the set of all relation types, At least distinguish between original document inclusion, original document location, source derivation, support, rebuttal, conflict, candidate filling, claim value, event sequence, navigation, and version revision relationship; For nodes In a specific relationship type The set of all neighboring nodes; For the first Within the layers, there is a specific focus on relation types. The weight matrix; the first In the layer, neighboring nodes Through relationships To the node Attention coefficient when transmitting messages.

[0104] The above formula defines the update rule for a single-layer graph neural network in a heterogeneous temporal evidence chain graph. Through this formula, the feature vector of each node... It can aggregate its own information and information about different types of neighbors while strictly distinguishing between node types and relationship types, providing semantically rich structured state encoding for reinforcement learning agents. Specifically, navigation relationships and proof / constraint relationships use different parameter matrices or independent channels to prevent navigation edges (such as similarity relationships and co-occurrence relationships) and proof edges (such as support and rebuttal relationships) from being treated equivalently in message passing.

[0105] For each gap Generate gap vector Using the target node or focus as the attention query, the gap conditional graph representation is obtained:

[0106]

[0107] in, This is the attention pooling function; The gap vector; The set of node features to be aggregated; For the process The output vector of a layered graphical neural network after message passing; To focus Centered on the gap, within a preset number of jumps Subgraphs reachable by type-related relationships.

[0108] The above formula is used to address the highest priority evidence gap. The algorithm focuses on local subgraphs related to the gap from the heterogeneous graph and aggregates them through an attention mechanism to generate a conditional graph representation vector. .

[0109] The final state is:

[0110]

[0111] in, For the first The complete state vector of the wheel; As a fusion function, the gap-conditional graph representation, the full-graph pooling representation, and the embedding vectors of each state component are mapped to a unified state vector; The conditional graph representation of the highest priority gap; This is the highest priority gap at present; For global graph pooling representation; This is an embedded function.

[0112] The above formula serves to combine the gaps and focal points calculated in the previous round, along with the remaining budget, historical processing footprint, and residuals from previous verification failures, to generate the final input state vector of the reinforcement learning policy network. The encoder employs a relational graph convolutional network, a heterogeneous graph attention network, a heterogeneous graph Transformer, or other graph neural network structures capable of distinguishing between node and relation types.

[0113] S3, Generate dynamic bounded motion space

[0114] Specifically, the instantiation action is represented as a quadruple:

[0115]

[0116] in, The action type is derived from Locate, Navigate, Expand, Fetch, Trace, or Contrast. For the target node or component range; For parsing, retrieving, or verifying operators; Parameters such as granularity, orientation, time window, or resolution accuracy are allowed. Additionally, a stopping action is defined. .

[0117] For each candidate action, compute a composite mask consisting of the product of seven binary submasks:

[0118]

[0119] in, For the first Round of candidate actions The overall action mask value; This is a gap mask used to determine whether an action is likely to advance the closing of the current gap. This is a focus mask used to determine whether a target is at the level, direction, and granularity of focus reachability. This is a pattern mask used to determine compatibility of node types, relation types, operator input / output signatures, and parameters. To prove the mask, fields are needed to indicate whether the action may produce original location, source chain, time reference, independent verification or conflict description; This is the permission mask, used to determine whether the target is within the scope of authorized materials and meets the component prerequisites. This is a budget mask used to estimate whether costs will not exceed the remaining budget. This is a history mask used to determine whether a request is a duplicate request with an existing reusable result, or a duplicate action with unchanged failure conditions.

[0120] Each sub-mask takes a value of 0 or 1, and the multiplication operation indicates that a candidate action must simultaneously satisfy the pass condition of all sub-masks. This determines whether the action can be selected and executed by the reinforcement learning agent in the current state.

[0121] The current legal action domain is:

[0122]

[0123] in, No. The legally executable action domain of the wheel; A generator for the complete set of candidate actions; To stop the exploration action.

[0124] The aforementioned dynamic action mask is directly based on the current type of evidence gap. and its focal position This generation ensures that the action space changes with the evidence gap in each round. Specifically:

[0125] When the gap type is "the source path is not yet closed", the gap mask blocks the Expand action, which is not related to source completion, the Trace action, and the Navigate and Fetch actions necessary to reach or read the source breakpoint.

[0126] When the gap type is "Conflict not yet specified", the gap mask enables the Contrast action and the Trace and Fetch actions required to obtain the adjudication material, and masks only the expansion action that adds similar material in the same direction;

[0127] When the gap type is "candidate range not yet determined", the gap mask enables the Locate, Navigate and Expand actions to discover new candidate materials, and masks the Trace and Contrast actions that cannot be performed when there are no candidates.

[0128] The agent performs hard filtering using a seven-fold binary submask (gap, focus, pattern, proof, authority, budget, and history). Only actions that pass all seven criteria are added to the executable set. .

[0129] S4. The policy network selects actions within the constraint space.

[0130] Policy network pairs Generate action vectors and calculate action log probabilities:

[0131]

[0132] in, For action The logarithmic probability; For policy networks; For action The embedding vector.

[0133] The above formula is used for the currently selected legal actions. Calculate an initial preference score (i.e., log probability). This score represents the policy network's initial evaluation of whether the action is good or bad before normalization.

[0134] The final output of the reinforcement learning policy network is:

[0135]

[0136] in, For the state vector and mask Under the given conditions, select the action. The probability of the strategy; For action Exponentialized scores; This is to iterate through all candidate actions.

[0137] The above formula is used to calculate the action preference score. This is transformed into an effective probability distribution over the set of legal actions. Essentially, it's a masked Softmax normalization operation, ensuring that the final action selection probabilities output by the policy network both reflect the relative merits of each action and strictly adhere to the hard constraints of the action mask. In implementation, the logarithmic probability of actions excluded by the mask can be set to negative infinity before Softmax; value-based policies only compare... Value estimation of actions. Action masks, as hard constraints imposed by the environment, cannot be overridden by the scoring of the policy network.

[0138] The input to a reinforcement learning policy network is a state vector. The output is the action probability distribution or Q-value. Softmax calculation or value comparison only considers the filtered valid set. Actions excluded by the mask have their logarithmic probability set to negative infinity and are never included in the decision-making process. This ensures that no matter how the policy network is optimized, it will not select illegal or invalid actions.

[0139] S5. The environment executes and performs access verification.

[0140] Execute action Standardized results were then obtained. Standardization results This should include at least stable node identifiers, target granularity, relationship type, source metadata, action basis, and actual cost. Environment's incremental impact on candidate graphs. Perform access verification and return:

[0141]

[0142] in, This is the admission verification function; Increment the candidate graph; For admission results; This is the updated heterogeneous temporal evidence chain diagram; This represents the actual resource consumption. Only admitted nodes and edges participate in the formal graph update and subsequent closure reward calculation.

[0143] It should be noted that if no element in the candidate increment passes the admission test, the final graph remains unchanged. But the candidate set Processing footprints Remaining budget and verification residuals It is still being updated. Therefore, the invariance of the formal graph does not mean that the reinforcement learning state remains unchanged. The policy can distinguish between two fundamentally different states: "not yet tried a certain action" and "tried the action but failed to pass the admission under the current conditions".

[0144] After the agent selects an action, the environment actually executes it (such as reading a file or querying a database) and generates candidate new nodes / edges. Subsequently, the environment performs extremely strict admission verification on the candidate increments (checking original hashes, precise location, root source chain, permissions, timestamps, etc.).

[0145] S6. Calculate multi-objective dynamic rewards

[0146] Specifically, based on the execution results of S5, a unique feedback value is calculated. This is used to train the agent. The primary reward objective is whether the candidate increment passes the admission verification and closes the evidence gap, while the intermediate reward objective is whether new clues that can be further verified are discovered. Penalties are also applied to constraint verification failures and budget consumption. The single-step reward is represented as:

[0147]

[0148] in, For the first The total reward obtained per step by the reinforcement learning strategy network; , , , and These are dynamic weighting coefficients; For entry and exit rewards; A reward will be given for finding clues; Penalty for invalid candidates.

[0149] Entry and Closure Rewards: Only those who pass verification and truly close the evidence gap will be given a high positive reward.

[0150] For query Each gap in evidence involved The definition is based solely on the formal diagram. Prove the satisfaction of the calculated constraint edges (excluding navigation edges). Based on this, define the quasi-closed potential function:

[0151]

[0152] in, For the gap The weights are determined by the potential function, which is monotonically increasing and bounded, satisfying the policy invariance condition for potential-based reward shaping.

[0153] The admission and closure rewards are:

[0154]

[0155] in, Let the closure potential function be the one that allows entry. For the first A formal heterogeneous temporal evidence chain diagram prior to the execution of the wheel action; For the first A formal heterogeneous temporal evidence chain diagram after the execution of the wheel action; The base amount for closing the single gap bonus; For the specific gaps in the traversal; For the gap The weights; For indicator functions; For the gap In the official diagram Satisfaction level; For the gap In the official diagram Satisfaction level; The bonus will be terminated for the entire company. This is a query response status indicator.

[0156] The above formula contains three components: the potential difference term reflects the marginal increase in gap satisfaction; the single gap closure bonus; and the potential difference term reflects the marginal increase in gap satisfaction. Triggered when a gap that was never satisfied becomes fully satisfied; global termination of bonuses. Triggered when the official graph first supports query answers. This is because unadmitted elements are not included. Navigation side does not participate Therefore, only formal observations, claims, and proofs verified through original hashing, precise location, root source, parsing version, permissions, time, and state transitions can generate positive closed-loop rewards.

[0157] To prevent screenshots, OCR texts, transcriptions, or summaries of the same original from repeatedly adding closure rewards, the system uses "query gap identifier - root original immutable identifier and normalized position - normalized assertion" as the contribution key, and adopts the maximum value within the class or other non-cumulative aggregation methods for closure contributions under the same contribution key.

[0158] Clue Reward: A small, capped reward will be given for discovering new, verifiable candidate locations.

[0159] The newly added node to the candidate set is denoted as . For each new candidate node Calculate the following four indicators: their correlation with the current gap and focus. Availability of stable identifiers or standardized positioning Novelty within this query And whether there are any feasible follow-up verification actions. The reward for finding clues is:

[0160]

[0161] in, This represents the remaining reward quota for clues in the current query. This is the set of nodes that have newly entered the candidate set in this round; As a new candidate node; For relevance; To ensure stable availability of identifiers; For novelty; For subsequent feasibility.

[0162] This reward only indicates that "a new candidate location has been discovered that can be further verified," and does not change the satisfaction level of any gaps. Candidate nodes are not included in the evidence set of the formal response. Navigation paths are not allowed to enter the proof path. The same stable identifier or normalized location can only be rewarded once per query. The single-step reward and cumulative cap for clues are both lower than the closure reward for critical gaps. This ensures that the strategy of repeatedly expanding candidate nodes cannot replace the closure of formal evidence in terms of total reward.

[0163] The penalty for invalid candidates is the weighted sum of the residuals:

[0164]

[0165] in, For the first The total penalty value resulting from the failure of the candidate increment to pass the admission verification; This is a dynamic penalty weight vector; To verify the residual vector in a structured manner.

[0166] Actual exploration cost vector This should include at least the actual read volume, I / O duration, computation time, number of component calls, and amount of manual review, calculated based on actual execution volume rather than estimated by action name. Step penalty. Apply a fixed, small negative reward to each step to limit meaningless, long exploration paths.

[0167] The structured validation residuals returned by admission validation must contain at least the following nine types of constraint violation indicators:

[0168]

[0169] in, The hashes are inconsistent. Location failure; The fracture originates from the root. The parsing version does not meet the requirements; The time constraint is not met; Unauthorized access; Unable to be reliably connected; Graph pattern violation; This is a repetition of the same root.

[0170] The weight coefficients of each component in the reward function are calculated using a deterministic environment function:

[0171]

[0172] in, For deterministic weighted scheduling functions; The type of gap with the highest priority; The priority value for the highest priority gap; The degree to which the gap is satisfied in the current official graph; This represents the percentage of the remaining budget. This marks the current exploratory phase.

[0173] Weighted scheduling follows these principles:

[0174] In the stage where the candidate range has not yet been defined, the weight of clue discovery should be appropriately increased. However, the cumulative reward cap for clues remains unchanged;

[0175] After entering the original document reading, source completion, or time verification stage, increase the weighting of admission and closure. and reduce ;

[0176] When the same type of validation failure occurs repeatedly, the penalty weight of the corresponding residual type is increased;

[0177] Increase the weight of actual costs when the budget is nearing exhaustion. and step-by-step punishment .

[0178] The weighting function can be implemented as a piecewise rule function, a lookup table function, or a parameterized function with range restrictions, and its version can be saved so that the same reward value can be recalculated at different times for the same state transition. Any candidate increment that fails to pass the admission test must not change the admission closed potential. .

[0179] S7. Update the state and generate training samples.

[0180] The results of the environmental feedback will be incorporated into the next round, updating the evidence gap status, focus, remaining budget, and processing footprint. Simultaneously, the complete interaction record for this round will be maintained: Status Legal Action Set Selected action ,award Next state Next Action Set Termination mark Access Results residual These samples are packaged into a transfer sample and stored in the experience pool. The sample also includes the version number of each module to ensure that subsequent training can fully reproduce the results.

[0181] Each round of decision-making generates the following transition samples:

[0182]

[0183] in, This is the state before a decision is made; This is the set of legal actions taken before a decision is made. The actions actually performed by the intelligent agent; For instant rewards; This is the state after the decision; It is the set of legal actions following a decision; For termination status indicator; This is the admission result.

[0184] The samples simultaneously store the pre- and post-production graph version identifiers, graph pattern version, state encoding version, mask generation version, reward function version, and cost measurement version to ensure sample replayability. Value-based, policy-based, actor-critic, online, or offline reinforcement learning algorithms can be used. Regardless of the algorithm used, the same mask semantics must be used during training and deployment: actions excluded by the mask cannot participate in the target value calculation. This constraint aims to prevent excluded high-value violations from artificially inflating the training target value, leading to a systematic underestimation of the value of legitimate actions.

[0185] S8. Termination Judgment and Result Output

[0186] Specifically, the termination condition is defined as follows:

[0187] When all necessary claims or event slots required for the query are covered by formal evidence, the claims upon which the answer depends can be closed along the proof / constraint edges to the root source, key temporal relationships can be explained, and competing claims affecting the answer have been adjudicated or explicitly reserved as disputes, let... The system outputs a result containing the original document proof path. If the budget is exhausted and only the stop action remains in the valid action domain... If the expected benefits of any remaining legal actions do not exceed their estimated costs, the system will stop exploring and output an insufficient evidence report, including admitted evidence, unmet typological evidence gaps, final focus position, reason for stopping, and official graph version identifier.

[0188] like Figure 4 The following is an example of reward function calculation centered on closing the evidence gap:

[0189] Taking the verification of "after account A transfers 50,000 yuan to account R, whether account R then diverts 30,000 yuan to account B and 20,000 yuan to account C respectively" as an example. (Current official diagram) The A→R transfer record has been approved, but the fact that R→B and R→C were split only exists in the summary document and has not yet been supported by original-level evidence, therefore it cannot be used as formal proof. The system thus generates two unmet evidentiary gaps, as shown in Table 1 below:

[0190] Table 1

[0191]

[0192] The current gap set is To illustrate the composition and differentiating role of rewards, let's assume... , , Step-by-step punishment Cost items are represented by weighted values. The following values ​​are only used to illustrate the combination of each reward item.

[0193] First outcome: Clues found, but the gap is not closed. The agent executes the Navigate action, locating a candidate data table containing the transaction history of account R. This table has a stable locator and can continue to be fetched, thus entering the candidate set. However, it has not yet undergone original-level access verification. At this time... Both evidentiary gaps remain unmet. Assuming the number of clues discovered is 0.8 and the actual cost is 0.2, then:

[0194]

[0195] The reward is a small positive value, derived solely from the clue discovery sub-item. This signal informs the strategy that "the current action has successfully located a candidate location that can be further verified," but it does not constitute progress in evidence. The strategy thus learns that simply discovering candidates through navigation cannot yield a high reward; subsequent evidence gathering and verification actions are required to close the gap.

[0196] The second outcome: Formal admission and gap closure. The agent continues to execute the Fetch action, obtaining the original flows R→B and R→C. These two flows, after passing admission verification through original hash, precise location, root source, parsed version, amount, and time order, are transformed into formal observer nodes and evidence claim nodes for writing. .gap and All were shut down. Assuming the entry and closure reward is 3 and the actual cost is 0.6, then:

[0197]

[0198] The reward is a high positive value. The reward comes from "the candidate increment passing admission verification and closing the evidence gap," not from the number of records read. The strategy thus learns that performing forensic actions that close the gap is the primary way to obtain a high reward.

[0199] The third outcome: Verification fails and the gap remains unclosed. Assuming the records returned by Fetch lack stable location and parsed version information, the candidate increment fails admission because it cannot be reliably attached to the original. The final graph remains unchanged. Assuming the verification residual weighting is 2 and the actual cost is 0.4, then:

[0200]

[0201] The reward is significantly negative. The strategy learns that performing a fetch on materials lacking necessary metadata leads to admission failure and wasted resources; therefore, in future encounters with similar conditions, alternative actions should be prioritized or the location information should be supplemented first.

[0202] In summary, the reward function clearly distinguishes between three state transition outcomes—"candidate found but gap remains unchanged," "admission granted and gap closed," and "verification failed but gap remains unchanged"—through three different reward values. This guides the agent to prioritize closing gaps while controlling exploration costs. Action path.

[0203] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics, characterized in that, Includes the following steps: S1. Receive and parse the user's evidence query, and calculate the admitted evidence set, candidate set, evidence gap set and graph focus from the heterogeneous temporal evidence chain graph that has completed evidence admission verification; S2. The four-layer nodes, typified relationships, time attributes and claim states of the heterogeneous temporal evidence chain graph are heterogeneously encoded, and the calculated evidence gap set and graph focus, as well as the remaining budget, original document processing footprint, and structured verification residuals from the previous round, are fused together to obtain the state vector. S3. Generate a dynamic action mask and a set of executable actions based on the gap closure conditions, graph focus, graph mode, processing footprint, permissions, and budget. S4. The reinforcement learning policy network selects the bound action from the set of executable actions; S5. After the environment executes the bound action, it forms a candidate graph increment. The environment performs admission verification on the candidate increment and returns the admission result, candidate clue changes, updated heterogeneous temporal evidence chain graph, structured verification residuals and actual costs. S6. Calculate multi-objective dynamic rewards based on the following items: admission and closure rewards, clue discovery rewards, invalid candidate penalties, actual exploration costs, and step penalties. S7. Update the evidence gap set, graph focus, remaining budget and original document processing footprint, and the next round of action mask to form a transfer sample; S8. When the updated heterogeneous temporal evidence chain graph satisfies the query response conditions, output the query result with the original proof path; when the remaining budget is exhausted or there are no action to be performed, output the evidence insufficiency report and the typified evidence gap.

2. The reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics according to claim 1, characterized in that: In step S1, The current user's evidence retrieval status is indicated as follows: in, For the first The current query status during round-robin decision-making; For users' evidence collection and inquiry; The set of evidence that has been approved and is permitted to be used in the response; This is a candidate set that can be further explored but has not yet been granted access; This is a set of unmet evidence gaps. The focus of the graph corresponding to the current priority gap; For the remaining budget; Process the footprints of the original document; This is the residual from the previous round of structured verification.

3. The reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics according to claim 1, characterized in that: In step S2, The four-layer nodes include the original document scope node, observation node, evidence claim node, and entity, event, and slot node; The heterogeneous graph encoding sets different initial feature mappings for different node types, and the initial features of the nodes are represented as follows: in, For nodes Type identifier; This is the projection matrix corresponding to the node type; For nodes The original feature vector; The state vector is generated in the following way: For each gap, a gap vector is generated. Using the target node or focus of the gap as the attention query, the conditional graph representation of the gap is obtained: in, This is the attention pooling function; The gap vector; This is the set of node features to be aggregated; For the process The output vector of a layered graphical neural network after message passing; To focus Centered on the gap, within a preset number of jumps Subgraphs reachable by type-related relationships; The final state is: in, For the first The complete state vector of the wheel; This is the fusion function; The conditional graph representation of the highest priority gap; This is the highest priority gap at present; For global graph pooling representation; This is an embedded function.

4. The reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics according to claim 1, characterized in that: In step S3, The dynamic action mask consists of the product of seven binary sub-masks, represented as: in, For the first Round of candidate actions The overall action mask value; For the gap mask; As a focus mask; For pattern mask; To prove the mask; This is a permission mask; For budget masking; To mask history; The set of executable actions is represented as follows: in, No. The legally executable action domain of the wheel; A generator for the complete set of candidate actions; To stop the exploration action.

5. The reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics according to claim 1, characterized in that: In step S4, The reinforcement learning policy network calculates the log probability for the bound action, as follows: in, For action The logarithmic probability; For policy networks; For action The embedding vector; The final output of the reinforcement learning policy network is: in, For the state vector and mask Under the given conditions, select the action. The probability of the strategy; For action Exponentialized scores; This is to iterate through all candidate actions.

6. The reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics according to claim 1, characterized in that: In step S5, The environment performs admission verification on candidate increments, as expressed by the following formula: in, This is the admission verification function; Increment the candidate graph; For admission results; This is the updated heterogeneous temporal evidence chain diagram; This refers to actual resource consumption.

7. The reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics according to claim 1, characterized in that: In step S6, The multi-objective dynamic reward is represented as follows: in, For the first The total reward obtained per step by the reinforcement learning strategy network; , , , and These are dynamic weighting coefficients; For entry and exit rewards; A reward will be given for finding clues; Penalty for invalid candidates.

8. The reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics according to claim 7, characterized in that: The dynamic weighting coefficients are calculated by a deterministic environment function and are expressed as follows: in, For deterministic weighted scheduling functions; The type of gap with the highest priority; The priority value for the highest priority gap; The degree to which the gap is satisfied in the current official graph; This represents the percentage of the remaining budget. This marks the current exploratory phase.

9. The reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics according to claim 7, characterized in that: In step S6, The admission and closure rewards are represented as follows: in, Let the closure potential function be the one that allows entry. For the first A formal heterogeneous temporal evidence chain diagram prior to the execution of the wheel action; For the first A formal heterogeneous temporal evidence chain diagram after the execution of the wheel action; The base amount for closing the single gap bonus; For the specific gaps in the traversal; For the gap The weights; For indicator functions; For the gap In the official diagram Satisfaction level; For the gap In the official diagram Satisfaction level; The bonus will be terminated for the entire company. For query response status indicators; The reward for finding clues is expressed as follows: in, This represents the remaining reward quota for clues in the current query. This is the set of nodes that have newly entered the candidate set in this round; As a new candidate node; For relevance; To ensure stable availability of identifiers; For novelty; For subsequent executability; The penalty for invalid candidates is expressed as follows: in, For the first The total penalty value resulting from the failure of the candidate increment to pass the admission verification; This is a dynamic penalty weight vector; To verify the residual vector in a structured manner; The structured verification residual vector is represented as: in, The hashes are inconsistent. Location failure; The fracture originates from the root. The parsing version does not meet the requirements; The time constraint is not met; This constitutes unauthorized access. Unable to be reliably connected; Graph pattern violation; This is a repetition of the same root.

10. The reinforcement learning method for exploring heterogeneous temporal evidence chain graphs in electronic forensics according to claim 1, characterized in that: In step S7, The transferred sample is represented as: in, This is the state before a decision is made; This is the set of legal actions taken before a decision is made. The actions actually performed by the intelligent agent; For instant rewards; This is the state after the decision; It is the set of legal actions following a decision; For termination status indicator; This is the admission result.

Images