Foreign exchange loop inverse detection method and system based on parallel graph calculation method

Abstract

The invention discloses a parallel graph calculation method-based foreign exchange loop reciprocation detection method and system, which are used for realizing quick detection response to loop reciprocation under the condition that the data volume of a detection result set is very large. On the basis of an existing technical scheme using a Read-Tarjan loop detection graph algorithm, an innovative fine-grained parallel loop detection scheme is provided, and the method is suitable for detecting loops and the like during historical snapshot of foreign exchange transactions of 12 months in the past. The method is especially suitable for the field of foreign exchange transaction detection and the like, and has good response timeliness control and high expandability.

Description

Technical Field

[0001] The present invention relates to the field of foreign exchange transaction data processing, and in particular to a foreign exchange loop back-and-forth detection method and system based on a parallel graph calculation method. Background Art

[0002] In foreign exchange trading, a ring trade (or "wash trade") is when traders create false market activity by trading with each other. This behavior may mislead other traders and cause abnormal fluctuations in market prices.

[0003] In order to detect foreign exchange loops, the technical approach is generally to model the associations in complex business scenarios into a graph structure. A graph is a data structure consisting of nodes (Vertex) and edges (Edge). When modeling business scenarios, the attribute graph model is generally chosen. In an attribute graph, nodes represent entities and edges represent relationships. A node or edge can have zero, one, or more attributes, and the attribute key of an entity is unique. In the business scenario of foreign exchange transactions, the attribute graph model treats institutions as nodes and transactions that have occurred as edges; a "loop" refers to a foreign exchange transaction link of the same foreign exchange product with the same foreign exchange trading institution at the beginning and end, which has a sequential relationship in time. The attributes on the edge can record the details of the transaction (transaction time, transaction amount, etc.). If multiple transactions occur between two institutions, multiple edges will be established to indicate the relationship. Loop detection is performed by finding loops.

[0004] The loop search technology solution for modeling a general attribute graph model for loop back detection with timing attributes generally adopts the Johnson algorithm and the Read-Tarjan algorithm. These algorithms and their coarse-grained parallel methods (i.e., simply executing loop detection starting with different start edges in parallel) have defects, mainly:

[0005] The time complexity is O(ec) (e is the number of edges, c is the size of the result set). When the query result set has a large amount of data, the response time is long and cannot meet the business requirements for response time. In real foreign exchange trading business scenarios, when querying historical snapshot data for the past 6 or 12 months, there will be a long response time due to the large number of loop results. Summary of the invention

[0006] The present invention aims to address the deficiencies in the prior art and to propose a foreign exchange loop back-and-forth detection method and system based on a parallel graph calculation method.

[0007] The object of the present invention is achieved through the following technical solution: a foreign exchange loop back-and-forth detection method based on a parallel graph calculation method, comprising:

[0008] Process the time-series foreign exchange transaction history data in the data warehouse into a transaction relationship graph based on the time window specified by the user.

[0009] For the generated graph relationship, the foreign exchange loop reverse detection method based on the parallel graph calculation method is used to quickly find the loop, and the result is output to the message queue for transmission;

[0010] The detection result set is filtered using the time sequence relationship, loop transactions that do not conform to the time sequence relationship are deleted, and the final detection results are reported.

[0011] Furthermore, the processing of the time-series foreign exchange transaction history data in the data warehouse into a transaction relationship diagram based on a time window specified by the user is specifically as follows: different foreign exchange transaction products are respectively constructed through the big data platform ETL, and the institutions are regarded as nodes, and the transactions that have occurred in the time window are regarded as edges; the attributes on the edges are based on the attribute fields pre-selected and defined in the business, and the details of the transactions are recorded; the points and edges and the corresponding attributes are temporarily stored in the big data platform.

[0012] Furthermore, the parallel graph-based computing method includes:

[0013] Perform parallel loop iterations on each edge in the transaction relationship graph. During the loop process, search for any path from the current vertex to the starting vertex without repeated vertices as an initial path extension. If an initial path extension exists, perform a loop search.

[0014] Furthermore, the loop search includes: receiving a copy of the current path and the blocked vertex set, deleting redundant parts, continuously adding the next vertex t of the initial path extension to the current path, and then searching for another path extension starting from t whose next vertex is different from the original path extension, recursively creating a new task with t, the initial vertex and the new path extension as input parameters, exiting the loop after all new path extensions are found, and when the initial path extension is traversed and no new path extension is found, reporting that the current path is a loop, and entering the message queue as a result.

[0015] Furthermore, in the receiving of the copy of the current path and the blocked vertex set: each loop search task will allocate and maintain its own current path and blocked vertex set, and only when the task is stolen can these sets be actually copied to reduce data overhead.

[0016] Furthermore, in the parallel graph-based computing method, the dynamic thread scheduling framework Intel TBB is used to execute each recursive call as a separate task.

[0017] Furthermore, the deletion of redundant parts is specifically as follows: when the last vertex in the received path is not the vertex of the current iteration, the last vertex is deleted in a continuous loop until the last vertex is the vertex of the current iteration; and the points added when the depth of the blocked vertex set is greater than or equal to the current iteration depth are deleted.

[0018] According to another aspect of the specification, there is provided a system for the method, the system comprising: a graph generation module, a graph data loading module, a parallel graph calculation module and a detection result filtering module;

[0019] The graph generation module is used to process the time-series foreign exchange transaction history data in the data warehouse into a transaction relationship graph based on a time window specified by a user;

[0020] The graph data loading module is used to load and store the graph data generated by modeling into the memory for subsequent calls;

[0021] The parallel graph calculation module uses the foreign exchange loop reverse detection method to quickly find loops and output the results to the message queue for transmission;

[0022] The detection result filtering module uses the time sequence relationship to filter the detection result set obtained by the parallel graph calculation module, deletes the loop transactions that do not conform to the time sequence relationship, and reports the final detection result.

[0023] According to another aspect of the specification, a foreign exchange loop back-to-back detection device based on a parallel graph calculation method is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the foreign exchange loop back-to-back detection method based on a parallel graph calculation method is implemented.

[0024] According to another aspect of the specification, a computer-readable storage medium is provided, on which a program is stored. When the program is executed by a processor, the foreign exchange loop back-to-back detection method based on a parallel graph calculation method is implemented.

[0025] Beneficial effects of the present invention: The foreign exchange loop back-to-back detection method and system based on the parallel method proposed by the present invention has a time complexity of O(ne) (n is the number of nodes, e is the number of edges) in the case of sufficient number of threads in business, and is not affected by the size of the result set. It is very suitable for fields such as foreign exchange transaction risk control, and enhances the supervision ability. The advantages it brings are self-evident. In summary, they mainly include:

[0026] 1) Good timeliness control. During the detection process, fine-grained parallelization is used to greatly reduce the time complexity, thereby greatly reducing the system response time.

[0027] 2) High scalability. The fine-grained parallel loop detection method proposed in the present invention has strong scalability, that is, when the result set is large enough, the parallelization acceleration is of the same order as the number of threads. Therefore, when the business scale is increased, the computing power can be improved by simply adding computing devices and distributed storage memory, thereby ensuring that the delay in dealing with complex timing loop searches is controllable. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of a method provided by an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of the operation of the method provided by an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0032] like Figure 1 As shown, the present invention provides a foreign exchange loop back-and-forth detection method and system based on a parallel graph calculation method.

[0033] The existing Read-Tarjan algorithm maintains a set of blocked vertices Blk for recursive tree pruning. Blk only tracks those vertices that cannot lead to new cycles when exploring the current path extension in the same recursive call. Vertices in Blk are avoided when searching for other path extensions branching off the current path extension. When it is fine-grained parallelized, since Blk only tracks vertices when in the same recursive call, it allows independent exploration of different subtrees of the recursive tree.

[0034] The foreign exchange loop back-and-forth detection method based on the parallel graph calculation method and the system operation process proposed by the present invention are as follows.

[0035] First, the system calculates and processes the time-series foreign exchange transaction history data in the data warehouse through the big data platform ETL job to form a transaction relationship graph based on the user-specified time window for different foreign exchange transaction products, and temporarily stores the processing points and edges and corresponding attributes in the big data platform; the graph data loading module is used to load and store the graph data of the modeling results in the memory. Figure 1 As shown, the institution is regarded as a node and the transactions that occurred in the time window are regarded as edges. The attributes on the edge can record the details of the transaction (transaction time, transaction amount, etc.) according to the attribute fields pre-selected and defined in the business. If multiple foreign exchange transactions occur between two institutions, multiple edges will be established to indicate the relationship.

[0036] Then, for the generated graph relationship, the foreign exchange loop back detection method based on the parallel graph calculation method is used to quickly find the loop, and the result is output to the message queue for transmission. This is the innovation point of the present invention, as shown in Algorithm 12 3, which is described in detail below.

[0037]

[0038]

[0039]

[0040]

[0041] Each recursive call is executed as a separate task using the dynamic thread scheduling framework Intel TBB.

[0042] There are neither data dependencies nor ordering requirements between different calls, except between parent and child calls. To exploit the parallelism available during recursive tree exploration, each path extension exploration is performed separately in each recursive call, and all these tasks can be executed independently.

[0043] First, as shown in Algorithm 1, to find all cycles of the graph, a parallel loop iteration is performed for each edge v0→v, which expands E with the path from v to v0 obtained by the search of Algorithm 3. If such E exists, a task is created to call Algorithm 2 with v, v0, and E as its input parameters to discover all cycles starting with the edge v0→v.

[0044] Algorithm 3 is a simple process that searches for any path from v to v0 without repeated vertices.

[0045] In Algorithm 2, in order to prevent different threads from modifying Π and Blk at the same time, each task will allocate and maintain its own Π and Blk sets. When a task is created, the task can directly receive a copy of Π and Blk from its parent task. However, these sets can only be truly copied when the task is "stolen" (Algorithm 2, line 3), thereby minimizing the copying overhead. After copying, their redundant parts will be deleted to ensure correctness (Algorithm 2, lines 4 and 5). After obtaining the correct Π and Blk sets, the next vertex t of E is continuously added to the current path Π, and then another path extension E' is searched for the next vertex starting from t that is different from E (while maintaining the point set Blk that should be blocked and can no longer generate a loop, the method is the same as the Read-Tarjan algorithm), and t, v0 and E' are used as input parameters to recursively create a new task, as shown in line 15 of Algorithm 2. After E' is found, the loop should be exited, and the task is copied again to create a new task with a deeper depth of d+1, as shown in line 20 of Algorithm 2, to avoid the situation where the π and Blk set information of the d+1 layer has been destroyed by the parent task in a different thread T1 when T1 processes the subtask with a deeper depth (such as d+2). When E is traversed and no new path extension E' is found (line 19 of Algorithm 2), the current path is reported as a loop, and the result is entered into the message queue.

[0046] Finally, the detection result set is filtered using the time sequence relationship, loop transactions that do not conform to the time sequence relationship are deleted, and the final detection results are reported.

[0047] The following is described by an example.

[0048] like Figure 1 , the foreign exchange transaction information of the same foreign exchange product in the time interval [2,11] has been integrated into a graph. When searching for a loop with institution 1->institution 2 as the starting edge, such as Figure 2After finding the path extension E of mechanism 2, 3, 4, 5, 1, enter the Solve_Task part, recorded as Task0, at this time the thread is T0, and the current path Π is {1}. After adding {2} to the current path, when searching for other extensions from mechanism 2, the path extension E' of mechanism 6, 7, 8, 1 is searched, and a new task Task1 is created; since the new extension is found, Task0 ends and a new copy is made to form a new task Task2. Assume that Task2 is also executed by thread T0, while Task1 is executed by thread T1, and Task2 is executed before Task1. Then in Task2, when searching for other extensions from mechanism 3, the path through mechanisms 9 and 10 cannot form a loop, so mechanisms 9 and 10 are blocked; finally, Task2 is executed and no new tasks are generated, and a loop {1, 2, 3, 4, 5, 1} is reported; in this process, nodes 3, 4, 5 are also blocked as path E is traversed, because repeated points cannot be passed in a loop. Then, Task1 is executed, and it is found that the task created by thread T0 is "stolen" by T1, so it is actually copied and the path is restored to {1, 2}, and {3, 4, 5, 9, 10} added to the Blk set when the depth is greater than or equal to the depth of Task1 is restored, and the correct path and Blk set are obtained. Then, when searching for other extensions from mechanism 3, the path through mechanisms 9 and 10 cannot form a loop, so mechanisms 9 and 10 are blocked; finally, Task1 is executed and no new tasks are generated, and a loop {1, 2, 6, 7, 8, 1} is reported. All loops with institution 1->institution 2 as the starting edge are successfully found and output to the message queue for transmission. Finally, since transaction 8->1 in the loop {1, 2, 6, 7, 8, 1} is completed before transaction 7->8, it does not conform to the time sequence relationship, so it is deleted in the filtering part, and the final result set {{1, 2, 3, 4, 5, 1}} is obtained.

[0049] In order to verify the performance improvement of the fine-grained parallel loop detection method proposed in the present invention for foreign exchange transaction loop back-to-back detection, a comparative test was carried out in a business system scenario.

[0050] The foreign exchange transaction flow data structure used in the test is shown in the following table:

[0051] Fields type Remark transTime Long Trading Hours from String(32) Transaction Seller to String(32) Buyer equityId Long Foreign exchange product number transAmt Long Transaction amount status Integer Transaction Status

[0052] In the map construction, the method adopted is to load the foreign exchange transaction flow of the last 12 months through the actual data set to build the map.

[0053] When 1024 threads are used, the fine-grained parallel loop detection method proposed in the present invention improves the performance of loop search by an order of magnitude compared to the traditional coarse-grained parallel Read-Tarjan method, and the improvement effect is 8-35 times in different actual data sets.

[0054] It can be seen that the response time of the present invention is shorter, which confirms the advantage of the present invention in fast detection and response of foreign exchange transaction loop when the query result set has a large amount of data.

[0055] Corresponding to the aforementioned embodiment of a foreign exchange loop back-to-back detection method based on a parallel graph calculation method, the present invention also provides an embodiment of a foreign exchange loop back-to-back detection system based on a parallel graph calculation method, the system comprising: a graph generation module, a graph data loading module, a parallel graph calculation module and a detection result filtering module;

[0056] The graph generation module is used to process the time-series foreign exchange transaction history data in the data warehouse into a transaction relationship graph based on a time window specified by a user;

[0057] The graph data loading module is used to load and store the graph data generated by modeling into the memory for subsequent calls;

[0058] The parallel graph calculation module uses the foreign exchange loop reverse detection method to quickly find loops and output the results to the message queue for transmission;

[0059] The detection result filtering module uses the time sequence relationship to filter the detection result set obtained by the parallel graph calculation module, deletes the loop transactions that do not conform to the time sequence relationship, and reports the final detection result.

[0060] Corresponding to the aforementioned embodiment of a foreign exchange loop back-and-forth detection method based on a parallel graph calculation method, the present invention further provides an embodiment of a foreign exchange loop back-and-forth detection device based on a parallel graph calculation method.

[0061] See also Figure 3 An embodiment of the present invention provides a foreign exchange loop back-to-back detection device based on a parallel graph calculation method, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a foreign exchange loop back-to-back detection method based on a parallel graph calculation method in the above embodiment.

[0062] The embodiment of the foreign exchange loop arbitrage detection device based on the parallel graph calculation method provided by the present invention can be applied to any device with data processing capabilities, and the any device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 3 As shown, it is a hardware structure diagram of a foreign exchange loop back-to-back detection device based on a parallel graph calculation method provided by the present invention, in which any device with data processing capability is located, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0063] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0064] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0065] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a foreign exchange loop trad detection method based on a parallel graph calculation method in the above embodiment is implemented.

[0066] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0067] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for detecting foreign exchange loop tradability based on a parallel graph calculation method is implemented.

[0068] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0069] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A foreign exchange loop back-and-forth detection method based on a parallel graph calculation method, characterized in that: include: Process the time-series foreign exchange transaction history data in the data warehouse into a transaction relationship graph based on the time window specified by the user. For the generated graph relationship, the foreign exchange loop reverse detection method based on the parallel graph calculation method is used to quickly find the loop, and the result is output to the message queue for transmission; The detection result set is filtered using the time sequence relationship, loop transactions that do not conform to the time sequence relationship are deleted, and the final detection results are reported.

2. According to the parallel graph calculation method of claim 1, the method is characterized in that: The processing of the time-series foreign exchange transaction history data in the data warehouse into a transaction relationship diagram based on a time window specified by the user is specifically as follows: different foreign exchange transaction products are respectively constructed through the big data platform ETL, and the institutions are regarded as nodes, and the transactions that occurred in the time window are regarded as edges; the attributes on the edges are based on the attribute fields pre-selected and defined in the business, and the details of the transactions are recorded; the points and edges and the corresponding attributes are temporarily stored in the big data platform.

3. The foreign exchange loop back-and-forth detection method based on the parallel graph calculation method according to claim 1 is characterized in that: The parallel graph-based computing method includes: Perform parallel loop iterations on each edge in the transaction relationship graph. During the loop process, search for any path from the current vertex to the starting vertex without repeated vertices as an initial path extension. If an initial path extension exists, perform a loop search.

4. The foreign exchange loop back-and-forth detection method based on the parallel graph calculation method according to claim 3 is characterized in that: The loop search includes: receiving a copy of the current path and the blocked vertex set, deleting redundant parts, continuously adding the next vertex t of the initial path extension to the current path, then searching for another path extension starting from t that is different from the original path extension, recursively creating a new task with t, the initial vertex and the new path extension as input parameters, exiting the loop after all new path extensions are found, and when the initial path extension is traversed and no new path extension is found, reporting that the current path is a loop and entering the message queue as a result.

5. The foreign exchange loop back-and-forth detection method based on the parallel graph calculation method according to claim 4 is characterized in that: In the receiving copy of the current path and the blocked vertex set: each loop search task will allocate and maintain its own current path and blocked vertex set, and only when the task is stolen can these sets be actually copied to reduce data overhead.

6. The foreign exchange loop back-and-forth detection method based on the parallel graph calculation method according to claim 3 is characterized in that: In the parallel graph computing method, the dynamic thread scheduling framework Intel TBB is used to execute each recursive call as a separate task.

7. The foreign exchange loop back-and-forth detection method based on parallel graph calculation method according to claim 4 is characterized in that: The redundant part is specifically deleted as follows: when the last vertex in the received path is not the vertex of the current iteration, the last vertex is deleted repeatedly until the last vertex is the vertex of the current iteration; and the points added when the depth of the blocked vertex set is greater than or equal to the depth of the current iteration are deleted.

8. A system for implementing the method according to any one of claims 1 to 7, characterized in that: The system includes: a graph generation module, a graph data loading module, a parallel graph calculation module and a detection result filtering module; The graph generation module is used to process the time-series foreign exchange transaction history data in the data warehouse into a transaction relationship graph based on a time window specified by a user; The graph data loading module is used to load and store the graph data generated by modeling into the memory for subsequent calls; The parallel graph calculation module uses the foreign exchange loop reverse detection method to quickly find loops and output the results to the message queue for transmission; The detection result filtering module uses the time sequence relationship to filter the detection result set obtained by the parallel graph calculation module, deletes the loop transactions that do not conform to the time sequence relationship, and reports the final detection result.

9. A foreign exchange loop back-to-back detection device based on a parallel graph calculation method, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a foreign exchange loop back-and-forth detection method based on a parallel graph calculation method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a foreign exchange loop back-to-back detection method based on a parallel graph calculation method as described in any one of claims 1 to 7 is implemented.

Images