A method and system for optimizing a financial transaction execution strategy based on reinforcement learning

Abstract

This invention discloses a method and system for optimizing financial transaction execution strategies based on reinforcement learning, relating to the field of financial transaction technology. It includes a competitive event analysis module, a sequential deduction module, and an execution action determination module. The competitive event analysis module receives order data and communication latency boundaries, and identifies groups of competitive events triggered by changes in the same order book based on the possible generation time intervals of each order event. The sequential deduction module inserts candidate execution actions generated by the reinforcement learning model into the causal partial order relationship. This invention establishes a causal partial order relationship between low-latency competitive events based on the generation time intervals of order events, inserts reinforcement learning candidate execution actions into multiple legitimate processing sequences corresponding to the causal partial order relationship for matching deduction, and outputs only candidate execution actions whose execution results are consistent under all legitimate processing sequences.

Description

Technical Field

[0001] This invention relates to the field of financial transaction technology, specifically to a method and system for optimizing financial transaction execution strategies based on reinforcement learning. Background Technology

[0002] Financial trading execution systems typically receive order data from exchanges on a per-transaction basis. Based on the order book status formed by order additions, cancellations, modifications, and executions, they generate instructions to submit limit orders, cancel existing orders, and modify the quantity of existing orders. Reinforcement learning techniques can use order book status as input and trading execution actions as output. By constructing a training environment using historical order data, the reinforcement learning model learns the state transition relationships between different order book states and trading execution actions. For low-latency trading scenarios, order data typically also records the exchange's processing time, message sequence number, order identifier, order price, order quantity, and order direction, thus supporting the replay and analysis of the trading execution process.

[0003] In a low-latency competitive scenario where multiple trading participants generate order instructions almost simultaneously for the same order book change event, the exchange processes each order instruction sequentially according to its actual arrival order, forming a unique message order in the historical order data. When the reinforcement learning training system replays the data one by one according to this unique message order, it may mistakenly identify the accidental processing order caused by communication latency differences as a definite causal order between order instructions. This causes the reinforcement learning model to learn state transition relationships that can only be established under a specific message arrangement. Consequently, in actual deployment, it generates trading execution actions with uncertain transaction quantities, uncertain queue positions, and uncertain instruction acceptance states, resulting in inconsistencies between the execution results in the training environment and the execution results in the real trading system. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for optimizing financial transaction execution strategies based on reinforcement learning, so as to solve the problems mentioned in the background art.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a financial transaction execution strategy optimization system based on reinforcement learning, comprising a competition event analysis module, a sequence deduction module, and an execution action determination module; the competition event analysis module is used to receive order data and communication delay boundaries, identify competition event groups triggered by changes in the same order book based on the possible generation time intervals of each order event, and establish causal partial order relationships between order events in the competition event group; the sequence deduction module is used to insert candidate execution actions generated by the reinforcement learning model into the causal partial order relationships, generate and process legal processing sequences that satisfy the causal partial order relationships according to the exchange matching rules, and obtain the order of each candidate execution action in each order book. The execution results under a legal processing order; the execution action determination module is used to compare the execution result records formed under each legal processing order for the same candidate execution action, and to determine the candidate execution actions in which the instruction acceptance status, cumulative transaction quantity, remaining order quantity, remaining order price and front queue quantity in each execution result record are consistent as order-invariant actions, and to add the order-invariant actions to the set of allowed training actions for the corresponding order book state, and to output the set of allowed training actions to the training process of the reinforcement learning model. The causal partial order relationship is used to record the precedence relationship that must be maintained between each order event. If there is no precedence relationship between two order events, it means that the two order events are allowed to be processed in different orders.

[0006] According to the above technical solution, the competition event parsing module includes a time interval construction submodule and a partial order relationship construction submodule. The time interval construction submodule is used to determine the possible generation time interval and possible transaction processing time interval of the order event based on the transaction processing time and the order uplink delay boundary. The partial order relationship construction submodule is used to retain the precedence relationship determined by the order lifecycle and remove the historical order relationship between order events whose possible transaction processing time intervals intersect and do not have order processing dependency. The sequence deduction module includes an action insertion submodule and a matching deduction submodule. The action insertion submodule is used to determine the prior relationship between the candidate action and each order event in the competing event group based on the possible arrival time interval of the candidate action. The matching deduction submodule is used to generate a legal processing order according to the causal partial order relationship and process each legal processing order in sequence according to the exchange matching rules of price priority and time priority. The action determination module includes a result comparison submodule and an action output submodule. The result comparison submodule is used to compare the instruction acceptance status, cumulative transaction quantity, remaining order quantity, remaining order price and front queue quantity of candidate action under each legal processing order. The action output submodule is used to output the candidate action whose execution results are consistent.

[0007] A method for optimizing financial trading execution strategies based on reinforcement learning includes the following steps: S1. Receive order data and pre-defined communication latency boundaries. Extract the exchange processing time, message sequence number, order identifier, target order identifier, order price, order quantity, order direction, and event type for each order event from the order data. Determine the possible generation time interval for each order event based on the exchange processing time and communication latency boundaries. Determine the response generation interval corresponding to the order book change event based on the downlink latency boundary and the participant processing time boundary. Identify the response event based on the response generation interval and the possible generation time interval of the order event. The order data consists of order instruction data recorded according to a unified reference clock. Extract new order instructions and cancel order instructions from the order instruction data as order events to be sorted. The transaction results generated by the exchange based on the order events are not considered as independent nodes to be sorted. For new order instructions, the target order identifier is empty. For cancel order instructions, the target order identifier is the order identifier of the canceled order. S2. Group order events associated with the same order book change event into a competing event group. Determine the necessary precedence relationships to be maintained based on the reference relationships between order identifiers and the dependencies between order creation and order processing. For order events that do not have the aforementioned precedence relationships but may have overlapping transaction processing time intervals, remove the order constraints of the order events in the historical order data. Form a causal partial order relationship for the competing event group based on the retained precedence relationships. S3. Obtain the confirmed order book status before the start of the competition event group, input the confirmed order book status into the reinforcement learning model to obtain candidate execution actions; determine the time precedence relationship between the action node and each response event based on the possible transaction processing time interval of the candidate execution actions; when the candidate execution action is to cancel the existing order of the party, establish the precedence relationship between the corresponding order creation event and the action node based on the target order identifier carried by the candidate execution action. S4. Generate a valid processing order based on the causal partial order relationship after inserting the action node, and process the order events and action nodes in each valid processing order in sequence according to the exchange matching rules; record the order book status, the status of the local order, and the event nodes that have not yet been processed during the processing; when different valid processing orders result in the same order book status, the status of the local order, and the event nodes that have not yet been processed, merge the corresponding processing branches. S5. Generate execution result records for each candidate action under each legal processing order, and determine the candidate actions with consistent execution result records as order-invariant actions; determine the set of candidate actions as the initial set of allowed training actions, and delete order-sensitive actions from the initial set of allowed training actions; input the corresponding order book status and the deleted set of allowed training actions into the training process of the reinforcement learning model, and when the set of allowed training actions is empty, determine the postponement instruction as the only allowed training action.

[0008] According to the above technical solution, step S1 specifically includes: S1-1. Before calculating the time interval, the transaction processing time in each order data and the local time recorded by the reinforcement learning model's running device are converted to the same reference clock. Subsequent time parameters are all represented using this unified reference clock. Since the transaction processing time of an order event is determined by the time the order event is generated at the participating end and the total time taken for the order event to pass through the order uplink, the upper and lower bounds of the order uplink delay are used to inversely limit the possible generation time range of the order event. The order data is read according to the transaction processing order, and the first... The exchange processing time for each order event is recorded as follows: The minimum delay of the entrusted uplink is recorded as The maximum delay of the entrusted uplink is denoted as And determine the first according to the following formula Possible time intervals for generating an order event: , , ,in, Indicates the first The possible time intervals for the generation of an order event This indicates the start time of the possible time interval to be generated. This indicates the end time of the possible generation time interval; the possible generation time interval represents the time range within which an order event is generated on the participating end and enters the delegated uplink. To ensure that the processing order of each historical order event and the candidate execution action is compared within the same time dimension, in the first... Based on the possible generation time intervals of each order event, the uplink latency boundary of the entrustment is superimposed to determine the first... The possible exchange processing time range for each order event is calculated using the following formula: , , ,in Indicates the first The possible exchange processing time range for each order event. This indicates the earliest possible time when the order event will be processed by the exchange. This indicates the latest possible time when the order event is processed by the exchange; the possible exchange processing time interval is used to reconstruct the range of possible processing times for the order event under the allowed combination of order uplink delays, while maintaining the uncertainty of the possible generation time of the order event. S1-2. After receiving the market information corresponding to the triggering event and completing local processing, the participating end generates a response order. Therefore, the exchange processing time of the triggering event is sequentially superimposed with the downlink latency of the market data and the processing time of the participating end to obtain the possible time range for the generation of the response order. This will cause the change in the number of price tiers in the order book. The order event is determined as the triggering event, and the first order event is determined as the triggering event. The transaction processing time for each triggered event is recorded as follows: The minimum delay of the downward trend in the market is denoted as The maximum downlink latency of the market is denoted as . The shortest processing time required for the participating end to process order book changes is denoted as The longest processing time required for a participant to handle order book changes is recorded as . The response generation range corresponding to the triggering event is determined according to the following formula: , , ,in, Indicates the first The response generation range corresponding to each triggering event. This indicates the earliest time when the participating end completes the response order generation in response to the triggering event. The latest time that the participating end completes the response order generation in response to the triggering event is indicated. The downlink latency of the market data indicates the total time elapsed from when the exchange completes the processing of the triggering event and starts generating the corresponding market data message until the participating end receives the market data message, including the time spent generating the market data message and the time spent transmitting the market data message. S1-3. When the possible generation time interval of an order event and the response generation interval of a triggering event share at least one common time point, the order event has the temporal possibility of being generated in response to the triggering event. The following formula is used to determine the... The possible generation time interval of the first order event and the first Do the response generation intervals of the triggered events intersect? When the above formula is true, the first... The order event was determined to be related to the first... The trigger event is associated with the response event. min means taking the earlier of the two times within the parentheses, and max means taking the later of the two times within the parentheses.

[0009] According to the above technical solution, step S2 specifically includes: S2-1. Group response events associated with the same triggering event into a competing event group, and determine the order processing dependency between each response event according to the order identifier, target order identifier, entrustment price, entrustment quantity and event type. S2-2. When the order identifier created by the first response event is the target order identifier for the second response event to perform the cancellation process, a precedence relationship is established between the first response event and the second response event. For new order instructions submitted by the participating parties, a precedence relationship is not established based on historical transaction results, historical queue positions, and changes in the number of historical price tiers. S2-3. To determine whether the possible transaction processing time intervals of two response events intersect, for the i-th and j-th response events that do not have the preceding relationship described in step S2-2, the overlap determination value of the two possible transaction processing time intervals is calculated according to the following formula: ,in, Indicates the first The response event and the first The overlapping value of possible exchange processing time intervals for a response event; when When it is greater than or equal to zero, it indicates that the first The response event and the first The possible transaction processing time intervals of two response events intersect, ensuring that, in the absence of order processing dependencies, the two response events maintain no sequential relationship in a causal partial order; when At that time, the first The response event points to the first The preceding relationship of a response event; when At that time, the first The response event points to the first The preceding relationships of each response event.

[0010] According to the above technical solution, step S3 specifically includes: S3-1. Determine the order book state before the earliest response event in the competitive event group is processed, and before any response event in the competitive event group has been processed, as the determined order book state before the start of the competitive event group; input the determined order book state and the current order state of the party into the reinforcement learning model to obtain a set of candidate execution actions, the set of candidate execution actions includes submitting a limit order and canceling the current order of the party, the submission of a limit order includes the order direction, order price and order quantity, and the cancellation of the current order of the party includes the target order identifier; S3-2. The total time required for a candidate action to be generated locally and processed by the exchange is limited by the uplink latency boundary. Therefore, the local generation time of the candidate action is added to the latency boundary to obtain the possible exchange processing time interval for the candidate action. The reinforcement learning model is started when each order instruction data is replayed to the unified reference time corresponding to the determined order book status. The reinforcement learning model completes the first step. The time when a candidate action is generated and output is determined as follows: And determine the first according to the following formula Possible transaction processing time ranges for each candidate action: , , ,in, Indicates the first The possible exchange processing time range for each candidate action. This indicates the earliest possible time when the candidate action will be processed by the exchange. This indicates the latest possible time when the candidate action will be processed by the exchange; S3-3. For each candidate action in the candidate action set, copy the causal partial order relation of the competing event group, and insert only the current candidate action as an action node into the corresponding copied causal partial order relation; when When, it indicates the first All candidate actions are executed before the first one under all allowed time delay combinations. The response event is processed by the exchange, establishing the action node pointing to the first... The preceding relationship of a response event; when When the time interval is equal, it indicates that the i-th response event is processed by the exchange before the action node under all allowed time interval combinations, and a precedence relationship is established between the i-th response event and the action node; when the possible exchange processing time interval of the candidate action intersects with the possible exchange processing time interval of the i-th response event, the action node and the i-th response event are kept without precedence. When the candidate action is to cancel an existing order, a prior relationship is established between the creation event of the corresponding order and the action node, based on the target order identifier carried by the candidate action.

[0011] According to the above technical solution, step S4 specifically includes: S4-1. Select a node from the event nodes and action nodes that do not have a predecessor node for processing. After processing, delete the node from the unprocessed nodes and delete the preceding relationships that the node points to other nodes. Repeat the node selection and preceding relationship deletion operations until all nodes in the competing event group have been processed to obtain a legal processing order that satisfies the causal partial order relationship. When there are two or more nodes without a predecessor node in the same processing stage, establish an independent processing branch with each node without a predecessor node as the current processing node, and continue to execute the node selection and preceding relationship deletion operations in each independent processing branch. The predecessor node refers to an event node or action node that has not yet been processed and points to the current node through an preceding relationship. The absence of a predecessor node means that all the preceding conditions corresponding to the current node have been processed.

[0012] S4-2. For each legal processing order, establish a corresponding order book copy and read order events and candidate execution actions sequentially according to the legal processing order. For limit orders entering the order book, first determine the corresponding price tier according to the order price, and then determine the queuing position according to the processing order of entering that price tier. For limit orders that can be matched with orders in the opposite direction, update the cumulative number of orders and the remaining number of orders participating in the matching according to the price priority and time priority matching rules, and update the total number of orders in the corresponding price tier according to the updated remaining number of orders. For order cancellation instructions, search for the corresponding order in the current order book copy according to the target order identifier carried by the order cancellation instruction. When the corresponding order exists, delete the remaining number of orders of the order from the corresponding price tier and record the instruction acceptance status of the order cancellation instruction as accepted. When the corresponding order does not exist, record the instruction acceptance status of the order cancellation instruction as rejected. S4-3. When the order identifiers arranged in the processing order within each price tier in different processing branches, the remaining order quantity corresponding to each order identifier, the instruction acceptance status of the local order, the cumulative transaction quantity, the remaining order quantity, the remaining order price, the queue quantity ahead, and the unprocessed event nodes and action nodes are all consistent, the different processing branches are merged into one processing branch; when any of them are inconsistent, the corresponding processing branches are retained and node processing continues.

[0013] According to the above technical solution, step S5 specifically includes: S5-1. After a valid processing sequence is completed, an execution result record is generated for each candidate execution action. The execution result record consists of the instruction acceptance status, cumulative transaction quantity, remaining order quantity, remaining order price, and queue size. The instruction acceptance status is either accepted or rejected. The queue size is the sum of the remaining quantities of all orders at the same price level as the current remaining order and preceding the current remaining order. When the order corresponding to the candidate execution action is not retained in the order book, the remaining order quantity, remaining order price, and queue size are all recorded as zero. S5-2. Copy the set of candidate execution actions obtained in step S3-1 to the initial set of allowed training actions corresponding to the determined order book state. Compare the execution result records of the same candidate execution action under all legal processing orders item by item. When the instruction acceptance status, cumulative transaction quantity, remaining order quantity, remaining order price and the number of orders in front are all the same in all execution result records, the candidate execution action is determined to be an order-invariant action. When there is a difference in one of the contents, the candidate execution action is determined to be an order-sensitive action, and the candidate execution action is deleted from the set of allowed training actions corresponding to the determined order book state. S5-3. Establish a correspondence between the determined order book status and the set of allowed training actions after deleting order-sensitive actions, and input the determined order book status and the set of allowed training actions into the training process of the reinforcement learning model; when the reinforcement learning model reads the determined order book status, it blocks candidate execution actions that do not belong to the set of allowed training actions, and selects training actions only from the set of allowed training actions; when the set of allowed training actions is empty, it uses the postponement of sending instruction as the only allowed training action, and the postponement of sending instruction means that no order instruction to be sent to the exchange will be generated in the current action decision cycle.

[0014] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: By distinguishing the deterministic causal relationship between the historical processing order of the exchange and order events, this invention avoids the reinforcement learning training system from using the accidental sequence caused by the communication link as a repeatable state transition condition. By reproducing the exchange processing results of candidate execution actions under multiple legal processing sequences, it can identify sequence-sensitive actions that depend on specific message arrangements and prevent such actions from entering the actual execution link. This reduces the probability of changes in the number of transactions, queue positions, and instruction state conflicts in the actual trading system, thereby improving the consistency between reinforcement learning training results and real trading execution results. This processing is completed based on order data, communication latency boundaries, order lifecycle, and exchange matching rules. It does not require predicting the communication speed of competing parties, nor does it require using the probability of competitive transactions to compensate for missing order data. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the present invention; Figure 2 This is a schematic diagram of the overall modular structure of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1 and Figure 2 This invention provides a technical solution: a method and system for optimizing financial transaction execution strategies based on reinforcement learning, comprising a competition event analysis module, a sequence deduction module, and an execution action determination module. The competition event analysis module receives order data and communication latency boundaries, identifies groups of competition events triggered by changes in the same order book based on the possible generation time intervals of each order event, and establishes a causal partial order relationship between each order event in the competition event group. The sequence deduction module inserts candidate execution actions generated by the reinforcement learning model into the causal partial order relationship, generates and processes legal processing sequences that satisfy the causal partial order relationship according to the exchange's matching rules, and obtains the order of each candidate execution action in each legal processing sequence. The execution results under the sequence; the execution action determination module is used to compare the execution result records formed under each legal processing sequence for the same candidate execution action, and to determine the candidate execution actions in which the instruction acceptance status, cumulative transaction quantity, remaining order quantity, remaining order price and the number of orders in front are consistent in each execution result record as order-invariant actions. The order-invariant actions are added to the set of allowed training actions for the corresponding order book state, and the set of allowed training actions is output to the training process of the reinforcement learning model. The causal partial order relationship is used to record the precedence relationship that must be maintained between each order event. If there is no precedence relationship between two order events, it means that the two order events are allowed to be processed in different orders. The competition event analysis module includes a time interval construction submodule and a partial order relation construction submodule. The time interval construction submodule is used to determine the possible generation time interval and possible transaction processing time interval of the order event based on the transaction processing time and the order uplink delay boundary. The partial order relation construction submodule is used to retain the precedence relationship determined by the order lifecycle and remove the historical order relationship between order events whose possible transaction processing time intervals intersect and have no order processing dependency relationship. The sequence deduction module includes an action insertion submodule and a matching deduction submodule. The action insertion submodule is used to determine the prior relationship between the candidate action and each order event in the competing event group based on the possible arrival time interval of the candidate action. The matching deduction submodule is used to generate a legal processing order according to the causal partial order relationship and process each legal processing order in turn according to the exchange matching rules of price priority and time priority. The action determination module includes a result comparison submodule and an action output submodule. The result comparison submodule is used to compare the instruction acceptance status, cumulative number of transactions, number of remaining orders, price of remaining orders and number of orders ahead of the candidate action under each legal processing order. The action output submodule is used to output the candidate action whose execution results are consistent. A method for optimizing financial trading execution strategies based on reinforcement learning includes the following steps: S1. Receive order data and pre-defined communication latency boundaries. Extract the exchange processing time, message sequence number, order identifier, target order identifier, order price, order quantity, order direction, and event type for each order event from the order data. Determine the possible generation time interval for each order event based on the exchange processing time and communication latency boundaries. Determine the response generation interval for order book change events based on the downlink latency boundary and participant processing time boundary, and identify the response event based on the response generation interval and the possible generation time interval of the order event. The order data consists of order instruction data recorded according to a unified reference clock. Extract new order instructions and cancel order instructions from the order instruction data as order events to be sorted. The transaction results generated by the exchange based on the order events are not considered as independent nodes to be sorted. For new order instructions, the target order identifier is empty; for cancel order instructions, the target order identifier is the order identifier of the canceled order. S2. Group order events associated with the same order book change event into a competing event group. Determine the event precedence relationships that must be maintained based on the reference relationships between order identifiers and the dependencies between order creation and order processing. For order events that do not have an event precedence relationship and whose transaction processing time intervals may intersect, remove the order constraints of the order events in the historical order data. Form a causal partial order relationship for the competing event group based on the retained event precedence relationships. S3. Obtain the confirmed order book status before the start of the competition event group, input the confirmed order book status into the reinforcement learning model to obtain candidate execution actions; determine the time precedence relationship between the action node and each response event based on the possible transaction processing time interval of the candidate execution actions; when the candidate execution action is to cancel the existing order of the party, establish the precedence relationship between the corresponding order creation event and the action node based on the target order identifier carried by the candidate execution action. S4. Generate a valid processing order based on the causal partial order relationship after inserting the action node, and process the order events and action nodes in each valid processing order in sequence according to the exchange matching rules; record the order book status, the status of the local order, and the event nodes that have not yet been processed during the processing; when different valid processing orders result in the same order book status, the status of the local order, and the event nodes that have not yet been processed, merge the corresponding processing branches. S5. Generate execution result records for each candidate action under each legal processing order, and determine the candidate actions with consistent execution result records as order-invariant actions; determine the set of candidate actions as the initial set of allowed training actions, and delete order-sensitive actions from the initial set of allowed training actions; input the corresponding order book status and the deleted set of allowed training actions into the training process of the reinforcement learning model; when the set of allowed training actions is empty, determine the postponement instruction as the only allowed training action; Step S1 specifically includes: S1-1. Before calculating the time interval, the transaction processing time in each order data and the local time recorded by the reinforcement learning model's running device are converted to the same reference clock. Subsequent time parameters are all represented using the unified reference clock. Since the transaction processing time of an order event is determined by the time the order event is generated at the participating end and the total time taken for the order event to pass through the order uplink, the upper and lower bounds of the order uplink delay are used to inversely limit the possible generation time range of the order event. The order data is read according to the transaction processing order, and the first... The exchange processing time for each order event is recorded as follows: The minimum delay of the entrusted uplink is recorded as The maximum delay of the entrusted uplink is denoted as And determine the first according to the following formula Possible time intervals for generating an order event: , , ,in, Indicates the first The possible time intervals for the generation of an order event Indicates the start time of the possible time interval to be generated. Indicates the end time of the possible generation time interval; the possible generation time interval represents the time range within which the order event is generated on the participating end and enters the delegated uplink; To ensure that the processing order of each historical order event and the candidate execution action is compared within the same time dimension, in the first... Based on the possible generation time intervals of each order event, the uplink latency boundary of the entrustment is superimposed to determine the first... The possible exchange processing time range for each order event is calculated using the following formula: , , ,in Indicates the first The possible exchange processing time range for each order event. This indicates the earliest possible time when an order event is processed by the exchange. Indicates the latest possible time when the order event is processed by the exchange; the possible exchange processing time interval is used to reconstruct the range of possible processing times for the order event under the allowed combination of order uplink delays, while maintaining the uncertainty of the possible generation time of the order event. S1-2. After receiving the market information corresponding to the triggering event and completing local processing, the participating end generates a response order. Therefore, the exchange processing time of the triggering event is sequentially superimposed with the downlink latency of the market data and the processing time of the participating end to obtain the possible time range for the generation of the response order. This will cause the change in the number of price tiers in the order book. The order event is determined as the triggering event, and the first order event is determined as the triggering event. The transaction processing time for each triggered event is recorded as follows: The minimum delay of the downward trend in the market is denoted as The maximum downlink latency of the market is denoted as . The shortest processing time required for the participating end to process order book changes is denoted as The longest processing time required for a participant to handle order book changes is recorded as . The response generation range corresponding to the triggering event is determined according to the following formula: , , ,in, Indicates the first The response generation range corresponding to each triggering event. This indicates the earliest time when the participating end completes its response to the triggered event and generates an order. The latest time that the participant completes the response order generation in response to the triggered event is indicated. The downlink latency of the market data indicates the total time elapsed from when the exchange completes the processing of the triggered event and starts generating the corresponding market data message until the participant receives the market data message, including the time spent generating the market data message from the exchange and the time spent transmitting the market data message. S1-3. When the possible generation time interval of an order event and the response generation interval of a triggering event share at least one common time point, the order event has the temporal possibility of being generated in response to the triggering event. The following formula is used to determine the... The possible generation time interval of the first order event and the first Do the response generation intervals of the triggered events intersect? When the above formula is true, the first... The order event was determined to be related to the first... The response event associated with each triggering event, min means taking the earlier of the two times within the parentheses, and max means taking the later of the two times within the parentheses; Step S1 uses the exchange's processing time, communication link latency range, and participant processing time to reconstruct the possible time range within which an order event could have been generated and processed by the exchange, and thereby identifies response events that might be triggered by changes in the same order book. This process provides a temporal basis for the formation of subsequent competing event groups, enabling the reconstruction of individual processing times recorded in historical data into possible time ranges consistent with the communication process. While conventional historical data replays typically use a fixed sequence represented by message numbers, this step further considers the actual time required for market data propagation, order generation, and order upload, reducing the likelihood of misinterpreting order events that are close in time but lack physical response conditions as consecutive causal events.

[0018] Step S2 specifically includes: S2-1. Group response events associated with the same triggering event into a competing event group, and determine the order processing dependency between each response event according to the order identifier, target order identifier, entrustment price, entrustment quantity and event type. S2-2. When the order identifier created by the first response event is the target order identifier for the second response event to perform the cancellation process, a precedence relationship is established between the first response event and the second response event. For new order instructions submitted by the participating parties, a precedence relationship is not established based on historical transaction results, historical queue positions, and changes in the number of historical price tiers. S2-3. To determine whether the possible transaction processing time intervals of two response events intersect, for the i-th and j-th response events that do not have a priori relationship as described in step S2-2, the overlap determination value of the two possible transaction processing time intervals is calculated according to the following formula: ,in, Indicates the first The response event and the first The overlapping value of possible exchange processing time intervals for a response event; when When it is greater than or equal to zero, it indicates that the first The response event and the first The possible transaction processing time intervals of two response events intersect, ensuring that, in the absence of order processing dependencies, the two response events maintain no sequential relationship in a causal partial order; when At that time, the first The response event points to the first The preceding relationship of a response event; when At that time, the first The response event points to the first The preceding relationships of each response event; Step S2 preserves the necessary processing sequence between order creation events and their corresponding cancellation events, while simultaneously removing historical order constraints between order events with overlapping processing time ranges and no order identifier dependency. This results in a causal partial order relationship containing only necessary preceding relationships. This causal partial order relationship maintains the processing conditions required for the order lifecycle while allowing independent response events to enter the matching process in different orders. This step re-distinguishes between deterministic and indeterminate orders based on order identifier dependency and time range, ensuring that subsequent deductions cover event arrangements that still conform to communication latency and transaction processing rules outside of historical records.

[0019] Step S3 specifically includes: S3-1. Determine the order book state before the earliest response event in the competitive event group is processed, and before any response event in the competitive event group has been processed, as the determined order book state before the start of the competitive event group; input the determined order book state and the current order state of the party into the reinforcement learning model to obtain a set of candidate execution actions. The set of candidate execution actions includes submitting a limit order and canceling the current order of the party. Submitting a limit order includes the order direction, order price and order quantity, and canceling the current order of the party includes the target order identifier; S3-2. The total time required for a candidate action to be generated locally and processed by the exchange is limited by the uplink latency boundary. Therefore, the local generation time of the candidate action is added to the latency boundary to obtain the possible exchange processing time interval for the candidate action. The reinforcement learning model is started when each order instruction data is replayed to the unified reference time corresponding to the determined order book status. The reinforcement learning model completes its first step. The time when a candidate action is generated and output is determined as follows: And determine the first according to the following formula Possible transaction processing time ranges for each candidate action: , , ,in, Indicates the first The possible exchange processing time range for each candidate action. This indicates the earliest possible moment when the candidate action will be processed by the exchange. This indicates the latest possible time when the candidate action will be processed by the exchange; S3-3. For each candidate action in the candidate action set, copy the causal partial order relation of the competing event group, and insert only the current candidate action as an action node into the corresponding copied causal partial order relation; when When, it indicates the first All candidate actions are executed before the first one under all allowed time delay combinations. The first response event is processed by the exchange, and an action node is established pointing to the first... The preceding relationship of a response event; when When the time interval is equal to 1, it indicates that the i-th response event is processed by the exchange before the action node under all allowed time interval combinations, and a precedence relationship is established between the i-th response event and the action node; when the possible exchange processing time interval of the candidate action intersects with the possible exchange processing time interval of the i-th response event, the action node and the i-th response event are kept without precedence. Step S3 replicates the causal partial order relationship of the competing event group for each candidate execution action, and inserts only one candidate execution action into each causal partial order relationship, allowing different candidate execution actions to be independently deduced under the same competing event conditions. The precedence relationship between action nodes and response event nodes is determined based on their respective time ranges that may be processed by the exchange. Revocation actions also establish necessary dependencies through the target order identifier and the corresponding order creation event. This process avoids multiple mutually exclusive candidate actions from affecting each other in the same processing branch and places candidate execution actions and historical response events within the same event processing structure, thereby enabling the determination of whether the action execution result depends on a certain accidental arrival order.

[0020] When the candidate action is to cancel an existing order, a prior relationship is established between the creation event of the corresponding order and the action node, based on the target order identifier carried by the candidate action. Step S4 specifically includes: S4-1. Select a node from the event nodes and action nodes that do not have a predecessor node for processing. After processing, delete the node from the nodes that have not yet been processed and delete the predecessor relationship that the node points to other nodes. Repeat the node selection and predecessor relationship deletion operations until all nodes in the competing event group have been processed to obtain a legal processing order that satisfies the causal partial order relationship. When there are two or more nodes that do not have a predecessor node in the same processing stage, establish an independent processing branch with each node that does not have a predecessor node as the current processing node, and continue to execute the node selection and predecessor relationship deletion operations in each independent processing branch. S4-2. For each legal processing order, create a corresponding order book copy and read order events and candidate execution actions sequentially according to the legal processing order. For limit orders entering the order book, first determine the corresponding price tier according to the order price, and then determine the queuing position according to the processing order of those price tiers. For limit orders that can be matched with orders in the opposite direction, update the cumulative number of orders and the remaining number of orders participating in the matching according to the price priority and time priority matching rules, and update the total number of orders in the corresponding price tier according to the updated remaining number of orders. For order cancellation instructions, search for the corresponding order in the current order book copy according to the target order identifier carried by the order cancellation instruction. When the corresponding order exists, delete the remaining number of orders for that order from the corresponding price tier and record the order cancellation instruction as accepted. When the corresponding order does not exist, record the order cancellation instruction as rejected. S4-3. When the order identifiers arranged in the processing order for each price tier in different processing branches, the remaining order quantity corresponding to each order identifier, the instruction acceptance status of our order, the cumulative transaction quantity, the remaining order quantity, the remaining order price, the queue quantity ahead, and the unprocessed event nodes and action nodes are all consistent, the different processing branches are merged into one processing branch; when any of them are inconsistent, the corresponding processing branches are retained and node processing continues. Step S4 generates a legal processing order from nodes that currently lack a predecessor node, level by level. When multiple processable nodes exist in the same stage, different processing branches are formed, and the order book and the status of the local order are updated in each branch according to the exchange's matching rules. Different branches are only merged when the order arrangement, remaining order quantity, local order execution results, and unprocessed nodes are all consistent across price tiers. This merging method reduces redundant deductions while preserving state differences that affect subsequent transactions, cancellations, and queue positions. Simply comparing the total number of price tiers can easily overlook the differences in subsequent results caused by changes in the order arrangement within the same price tier. This step ensures that branch compression does not change the actual deduction results of candidate execution actions by comparing order-level states.

[0021] Step S5 specifically includes: S5-1. After a valid processing sequence is completed, an execution result record is generated for each candidate action. The execution result record consists of the instruction acceptance status, cumulative transaction quantity, remaining order quantity, remaining order price, and queue size. The instruction acceptance status is either accepted or rejected. The queue size is the sum of the remaining quantities of all orders at the same price level as the current order and preceding the current order. When the order corresponding to the candidate action is not retained in the order book, the remaining order quantity, remaining order price, and queue size are all recorded as zero. S5-2. Copy the set of candidate execution actions obtained in step S3-1 to the initial set of allowed training actions corresponding to the determined order book state. Compare the execution result records of the same candidate execution action under all legal processing orders item by item. When the instruction acceptance status, cumulative transaction quantity, remaining order quantity, remaining order price and the number of orders in front are all the same in all execution result records, the candidate execution action is determined to be an order-invariant action. When there is a difference in one of the contents, the candidate execution action is determined to be an order-sensitive action, and the candidate execution action is deleted from the set of allowed training actions corresponding to the determined order book state. S5-3. Establish a correspondence between the determined order book status and the set of allowed training actions after deleting order-sensitive actions, and input the determined order book status and the set of allowed training actions into the training process of the reinforcement learning model; when the reinforcement learning model reads the determined order book status, it blocks candidate execution actions that do not belong to the set of allowed training actions, and selects training actions only from the set of allowed training actions; when the set of allowed training actions is empty, it uses the postponement of sending instructions as the only allowed training action, and the postponement of sending instructions means that no order instructions to be sent to the exchange will be generated in the current action decision cycle.

[0022] Step S5 compares the instruction acceptance status, cumulative transaction volume, remaining order volume, remaining order price, and queue size for the same candidate execution action under all legal processing sequences. If all results remain consistent, it indicates that the action is independent of the random processing order of competing events and is therefore retained in the set of allowed training actions for the corresponding order book state. If any result changes, the action is removed from the set of allowed training actions. This screening process directly utilizes the consistency constraints of exchange processing results to reinforce the range of selectable actions for the reinforcement learning model. It eliminates the need to estimate the speed of competing participants or calculate the probability of early execution, thereby reducing the likelihood of the model learning non-repeatable execution conditions from a single historical permutation.

[0023] Each step is progressively linked around the potential state transition distortion caused by the unique historical message sequence, subjecting the reinforcement learning model's training actions to the combined constraints of communication latency, order lifecycle, and exchange matching rules. The entire process focuses on the dependence of candidate execution actions on the random arrangement of events, improving the consistency between the training and actual execution environments without altering the exchange matching rules.

[0024] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0025] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A reinforcement learning based financial transaction execution strategy optimization system, characterized in that: It includes a competition event parsing module, a sequence deduction module, and an execution action determination module; the competition event parsing module is used to receive order data and communication delay boundaries, identify competition event groups triggered by changes in the same order book based on the possible generation time intervals of each order event, and establish causal partial order relationships between each order event in the competition event group; The sequential deduction module is used to insert the candidate execution actions generated by the reinforcement learning model into the causal partial order relationship, generate and process a legal processing order that satisfies the causal partial order relationship according to the exchange matching rules, and obtain the execution result of each candidate execution action under each legal processing order. The execution action determination module is used to compare the execution result records formed under each legal processing order for the same candidate execution action, and to determine the candidate execution actions in which the instruction acceptance status, cumulative transaction quantity, remaining order quantity, remaining order price and front queue quantity are consistent as order-invariant actions. The order-invariant actions are added to the set of allowed training actions for the corresponding order book status, and the set of allowed training actions is output to the training process of the reinforcement learning model.

2. The financial transaction execution strategy optimization system based on reinforcement learning according to claim 1, characterized in that: The competition event parsing module includes a time interval construction submodule and a partial order relation construction submodule. The time interval construction submodule is used to determine the possible generation time interval and possible transaction processing time interval of the order event based on the transaction processing time and the order uplink delay boundary. The partial order relation construction submodule is used to retain the precedence relationship determined by the order lifecycle and remove the historical order relationship between order events whose possible transaction processing time intervals intersect and have no order processing dependency relationship. The sequence deduction module includes an action insertion submodule and a matching deduction submodule. The action insertion submodule is used to determine the prior relationship between the candidate action and each order event in the competing event group based on the possible arrival time interval of the candidate action. The matching deduction submodule is used to generate a legal processing order according to the causal partial order relationship and process each legal processing order in sequence according to the exchange matching rules of price priority and time priority. The action determination module includes a result comparison submodule and an action output submodule. The result comparison submodule is used to compare the instruction acceptance status, cumulative transaction quantity, remaining order quantity, remaining order price and front queue quantity of candidate action under each legal processing order. The action output submodule is used to output the candidate action whose execution results are consistent.

3. A method for optimizing financial transaction execution strategies based on reinforcement learning, characterized in that: The method, when applied to the system of claim 2, includes the following steps: S1. Receive order data and pre-defined communication latency boundaries; extract the transaction processing time, message sequence number, order identifier, target order identifier, order price, order quantity, order direction, and event type for each order event from the order data; determine the possible generation time interval for each order event based on the transaction processing time and communication latency boundaries; determine the response generation interval corresponding to the order book change event based on the downlink latency boundary and the participant processing time boundary; and identify the response event based on the response generation interval and the possible generation time interval of the order event. S2. Group order events associated with the same order book change event into a competing event group, and determine the event precedence relationships that must be maintained based on the reference relationships between order identifiers and the dependencies between order creation and order processing. For order events that do not have the aforementioned event precedence relationship and whose transaction processing time intervals may overlap, the order constraint of the order events in the historical order data is removed; a causal partial order relationship is formed for the competing event groups based on the retained event precedence relationship. S3. Obtain the confirmed order book status before the start of the competitive event group, input the confirmed order book status into the reinforcement learning model to obtain candidate execution actions; determine the time precedence relationship between the action node and each response event based on the possible transaction processing time interval of the candidate execution actions. When the candidate action is to cancel an existing order, a precedence relationship is established between the corresponding order creation event and the action node based on the target order identifier carried by the candidate action. S4. Generate a valid processing order based on the causal partial order relationship after inserting the action node, and process the order events and action nodes in each valid processing order in sequence according to the exchange matching rules; record the order book status, the status of the local order, and the event nodes that have not yet been processed during the processing; when different valid processing orders result in the same order book status, the status of the local order, and the event nodes that have not yet been processed, merge the corresponding processing branches. S5. Generate the execution result record of each candidate execution action under each legal processing order, and determine the candidate execution action whose execution result record is consistent as the order-invariant action; The candidate set of actions to be executed is determined as the initial set of allowed training actions, and order-sensitive actions are removed from the initial set of allowed training actions. The corresponding order book status and the set of allowed training actions after deletion are input into the training process of the reinforcement learning model. When the set of allowed training actions is empty, the postponement instruction is determined as the only allowed training action.

4. The method for optimizing financial transaction execution strategies based on reinforcement learning according to claim 3, characterized in that: Step S1 specifically includes: S1-1, the first The exchange processing time for each order event is recorded as follows: The minimum delay of the entrusted uplink is recorded as The maximum delay of the entrusted uplink is denoted as And determine the first according to the following formula Possible time intervals for the generation of an order event: , , ,in, Indicates the first The possible time intervals for the generation of an order event This indicates the start time of the possible time interval to be generated. Indicates the end time of the possible generation time interval, at the th Based on the possible generation time intervals of each order event, the uplink latency boundary of the entrustment is superimposed to determine the first... The possible exchange processing time range for each order event is calculated using the following formula: , , ,in Indicates the first The possible exchange processing time range for each order event. This indicates the earliest possible time when the order event will be processed by the exchange. This indicates the latest possible time when the order event will be processed by the exchange; S1-2, the first one that will cause changes in the quantity of price tiers in the order book. The order event is determined as the triggering event, and the first order event is determined as the triggering event. The transaction processing time for each triggered event is recorded as follows: The minimum delay of the downward trend in the market is denoted as The maximum delay of the downlink in the market is denoted as The shortest processing time required for the participating end to process order book changes is denoted as The longest processing time required for a participant to handle order book changes is recorded as . The response generation range corresponding to the triggering event is determined according to the following formula: , , ,in, Indicates the first The response generation range corresponding to each triggering event. This indicates the earliest time when the participating end completes the response order generation in response to the triggering event. This indicates the latest time when the participating end completes the response order generation in response to the triggered event; S1-3, determine the first according to the following formula. The possible generation time interval of the first order event and the first Do the response generation intervals of the triggered events intersect? When the above formula is true, the first... The order event was determined to be related to the first... The trigger event is associated with the response event. min means taking the earlier of the two times within the parentheses, and max means taking the later of the two times within the parentheses.

5. The method for optimizing financial transaction execution strategies based on reinforcement learning according to claim 4, characterized in that: Step S2 specifically includes: S2-1. Group response events associated with the same triggering event into a competing event group, and determine the order processing dependency between each response event according to the order identifier, target order identifier, entrustment price, entrustment quantity and event type. S2-2. When the order identifier created by the first response event is the target order identifier for the second response event to perform the cancellation process, a precedence relationship is established between the first response event and the second response event. For new order instructions submitted by the participating parties, a precedence relationship is not established based on historical transaction results, historical queue positions, and changes in the number of historical price tiers. S2-3. For the i-th and j-th response events that do not have the preceding relationship described in step S2-2, calculate the overlap determination value of the two possible transaction processing time intervals according to the following formula: ,in, Indicates the first The response event and the first The overlap determination value of possible transaction processing time intervals for each response event; when When it is greater than or equal to zero, it indicates that the first The response event and the first The possible transaction processing time intervals of two response events intersect, ensuring that, in the absence of order processing dependencies, the two response events maintain no sequential relationship in a causal partial order; when At that time, the first The response event points to the first The preceding relationship of a response event; when At that time, the first The response event points to the first The preceding relationships of each response event.

6. The method for optimizing financial transaction execution strategies based on reinforcement learning according to claim 5, characterized in that: Step S3 specifically includes: S3-1. Determine the order book state before the earliest response event in the competitive event group is processed, and before any response event in the competitive event group has been processed, as the determined order book state before the start of the competitive event group; input the determined order book state and the existing order state of the party into the reinforcement learning model to obtain a set of candidate execution actions; S3-2. When the data of each order instruction is replayed to the unified reference time corresponding to the determined order book status, the reinforcement learning model is started, and the reinforcement learning model completes the first step. The time when a candidate action is generated and output is determined as follows: And determine the first according to the following formula Possible transaction processing time ranges for each candidate action: , , ,in, Indicates the first The possible exchange processing time range for each candidate action. This indicates the earliest possible time when the candidate action will be processed by the exchange. This indicates the latest possible time when the candidate action will be processed by the exchange; S3-3. For each candidate action in the candidate action set, copy the causal partial order relation of the competing event group, and insert only the current candidate action as an action node into the corresponding copied causal partial order relation; when When, it indicates the first All candidate actions are executed before the first one under all allowed time delay combinations. The response event is processed by the exchange, establishing the action node pointing to the first... The preceding relationship of a response event; when When the time is specified, it indicates that the i-th response event is processed by the exchange before the action node under all allowed time delay combinations, and a precedence relationship is established between the i-th response event and the action node.

7. The method for optimizing financial transaction execution strategies based on reinforcement learning according to claim 6, characterized in that: Step S4 specifically includes: S4-1. Select a node from the event nodes and action nodes that do not have a predecessor node for processing. After processing, delete the node from the nodes that have not yet been processed and delete the predecessor relationship that the node points to other nodes. Repeat the node selection and predecessor relationship deletion operations until all nodes in the competing event group have been processed to obtain a legal processing order that satisfies the causal partial order relationship. When there are two or more nodes that do not have a predecessor node in the same processing stage, establish an independent processing branch with each node that does not have a predecessor node as the current processing node, and continue to execute the node selection and predecessor relationship deletion operations in each independent processing branch. S4-2. For each legal processing order, establish a corresponding order book copy and read order events and candidate execution actions sequentially according to the legal processing order. For limit orders entering the order book, first determine the corresponding price tier according to the order price, and then determine the queuing position according to the processing order of entering that price tier. For limit orders that can be matched with orders in the opposite direction, update the cumulative number of orders and the remaining number of orders participating in the matching according to the price priority and time priority matching rules, and update the total number of orders in the corresponding price tier according to the updated remaining number of orders. For order cancellation instructions, search for the corresponding order in the current order book copy according to the target order identifier carried by the order cancellation instruction. When the corresponding order exists, delete the remaining number of orders of the order from the corresponding price tier and record the instruction acceptance status of the order cancellation instruction as accepted. When the corresponding order does not exist, record the instruction acceptance status of the order cancellation instruction as rejected. S4-3. When the order identifiers arranged in the processing order within each price tier in different processing branches, the remaining order quantity corresponding to each order identifier, the instruction acceptance status of the local order, the cumulative transaction quantity, the remaining order quantity, the remaining order price, the queue quantity ahead, and the unprocessed event nodes and action nodes are all consistent, the different processing branches are merged into one processing branch; when any of them are inconsistent, the corresponding processing branches are retained and node processing continues.

8. The method for optimizing financial transaction execution strategies based on reinforcement learning according to claim 7, characterized in that: Step S5 specifically includes: S5-1. After a legal processing sequence is completed, an execution result record is generated for each candidate execution action. The execution result record consists of the instruction acceptance status, cumulative number of transactions, number of remaining orders, price of remaining orders, and number of orders in the queue ahead. S5-2. Copy the set of candidate execution actions obtained in step S3-1 to the initial set of allowed training actions corresponding to the determined order book state. Compare the execution result records of the same candidate execution action under all legal processing orders item by item. When the instruction acceptance status, cumulative transaction quantity, remaining order quantity, remaining order price and the number of orders in front are all the same in all execution result records, the candidate execution action is determined to be an order-invariant action. When there is a difference in one of the contents, the candidate execution action is determined to be an order-sensitive action, and the candidate execution action is deleted from the set of allowed training actions corresponding to the determined order book state. S5-3. Establish a correspondence between the determined order book state and the set of allowed training actions after deleting order-sensitive actions, and input the determined order book state and the set of allowed training actions into the training process of the reinforcement learning model; when the reinforcement learning model reads the determined order book state, it blocks candidate execution actions that do not belong to the set of allowed training actions, and selects training actions only from the set of allowed training actions.

Images