An external data set operation method and set operation device

Abstract

An external data set operation method and set operation device, the present application relates to the bank system in the process of reconciliation, data processing, there are a large number of set operation of two files, intersection operation. Using database technology, time-consuming more. The present application is provided with three hash mapping areas in the memory of the set operation device, each hash mapping area includes a plurality of linked lists: and read the first file and the second file contrast line by line, and the hash value, the primary key and the primary key value information of each row data are stored in the hash mapping area. Then, the left and right association algorithm is carried out on each first chain table and each second chain table, and the set operation is carried out, and the position information of the content mismatch is stored in the third hash mapping area, so that the set operation of the external data is realized. No database resources are occupied, no other commercial software is used, and no external data storage, result export and other links are needed, which is 50 times faster than the existing mode, and is of great significance to the improvement of the work efficiency of the bank system.

Description

I. TECHNICAL FIELD

[0001] The present application relates to the field of information processing of data in structured storage files in the data processing process of reconciliation in a banking system, and in particular to a set operation method and set operation device for memory and file system based on a HASH algorithm. II. BACKGROUND

[0002] With the sharp increase in the volume of business in the banking industry, in the process of reconciliation and data processing, it is often necessary to perform set union and intersection operations on data in two files of generally structured storage files, and the performance of the operation will inevitably affect the performance of the entire banking system. How to maximize the performance of external data set has become an urgent problem to be solved.

[0003] The purpose of improving the efficiency of external data set is to perform set union and intersection operations on data in two files of generally structured storage files and improve the efficiency in the process of reconciliation and data processing. In the previous system, after the two files are imported into the database, set operation is performed, mainly by processing data through SQL. After processing, the file is exported. If the import calculation needs to frequently load the file and frequently operate the database, the efficiency will inevitably be affected by the performance of the database.

[0004] At present, banks need to reconcile their own account files and the other party's account information every day in the centralized reconciliation process. Each day can be up to millions of groups, and inconsistent or missing, multiple records of each party are found. Such a large amount of data needs to be compared. If database technology is used, a large amount of computer resources will be occupied, and it will take a lot of time. The SQL processing operation is low in efficiency and consumes a large amount of database computing resources. III. SUMMARY

[0005] The present application aims to solve the above problems. Without occupying data resources and using other commercial software, a universal and efficient external data set algorithm is provided. The problem of low efficiency of set union and intersection operations on data in two files is solved. The problem of external data correlation operation operation independent of the database is solved. Without occupying database resources and using other commercial software, a universal and efficient external data set algorithm and technology are provided, and there is no need to perform external data import, result export and other links. It is 50 times faster than the existing import first and then association method. The time complexity is linearly related to the data volume.

[0006] The technical solution of the present application is: an external data set operation method. It is applied to a set operation device, and three hash mapping areas are arranged in the memory of the set operation device. Each hash mapping area includes a plurality of linked lists. The set operation method comprises:

[0007] read the first file to be operated line by line, and store the hash value, primary key and primary key value information of each row of data in the first hash mapping area corresponding to the hash value of each row of data in the first chain table;

[0008] read the second file to be operated line by line, and store the hash value, primary key and primary key value information of each row of data in the second hash mapping area corresponding to the hash value of each row of data in the second chain table;

[0009] Based on the hash value, primary key and primary key value information stored in each chain table, the intersection and union operation is performed on each first chain table and each second chain table, and the operation result is stored in the third hash mapping area;

[0010] Based on the third hash mapping group, an operation result file is generated to realize the set operation of external data.

[0011] Further, the reading of the first file to be operated line by line and the storage of the hash value, primary key and primary key value information of each row of data in the first hash mapping area corresponding to the hash value of each row of data in the first chain table comprises:

[0012] For any row of data in the first file, read the row of data and extract the primary key and primary key value information of the row of data;

[0013] Based on the hash algorithm, calculate the hash value of the row of data;

[0014] Based on the hash value of the row of data and the number of first chain tables in the first hash mapping area, determine the first chain table corresponding to the hash value of the row of data;

[0015] Store the hash value, primary key and primary key value information of the row of data in the first chain table corresponding to the hash value of the row of data.

[0016] More specifically, in the storage structure, there are two Hash bucket lists for storing the indexes of the two files respectively, and the set operation is realized by comparing the two indexes.

[0017] Including but not limited to the following steps:

[0018] Step 1: Prepare two comparison files to be operated in the memory;

[0019] Step 2: Set the primary keys of the two files respectively;

[0020] Step 3: Build HashMap structure HashMap1, HashMap2, HashJoinRes in memory, which is composed of link table (Link). The number of link table in HashMap is N, and the value of N is recommended to be 0.01-0.1 times of the amount of data to be processed, and is a prime number.

[0021] Step 4: Read the data of file one, extract the primary key F1k1, F1k2, and calculate the Hash value H after connecting. Calculate the remainder S of H divided by the number of link table N; store the Hash value H and part of the primary key value information Data in the corresponding S number link table of HashMap1 until the reading of file one is completed.

[0022] Step 5: Similarly, read the data of comparison file two, extract the primary key F2k1, F2k2, calculate the Hash value and the remainder S, and then store the Hash value H and part of the primary key value information Data in the corresponding S number link table of HashMap2 until the reading of comparison file two is completed.

[0023] Step 6: Traverse HashMap1, HashMap2, and compare whether the hash value and primary key information on both sides are consistent. If consistent, create HashJoin structure, and void*data1, void*data2 point to the data in HashMap1, HashMap2 respectively; and add the HashJoin structure to HashJoinRes.

[0024] Step 7: Traverse the HashJoinRes structure, output to the file, which completes the two file independent database inner join operation;

[0025] Step 8: If the corresponding hash value and primary key information of HashMap1 cannot be found in HashMap2, void*data1 points to the data in HashMap1, and void*data2 points to empty; realize the function of left join of HashMap1 and HashMap2, that is, the algorithm of database left join (Left Join).

[0026] Step 9: If you want to realize right join (Right Join), assign null value to void*data1 when the corresponding information of HashMap2 cannot be found in HashMap1, and void*data2 points to the information in HashMap2, which can be realized.

[0027] Among them, the HashJoin structure is composed of hash value, part of original data, data1 data pointer, data2 data pointer, next node pointer.

[0028] Further, the determining the first chain table corresponding to the hash value of the row data based on the hash value of the row data and the number of the first chain tables in the first hash mapping area comprises:

[0029] performing a remainder operation on the hash value of the row data based on the number of the first chain tables to obtain a remainder S1;

[0030] determining the S1th first chain table in the first hash mapping area as the first chain table corresponding to the hash value of the row data.

[0031] Further, the determining the first chain table corresponding to the hash value of the row data based on the hash value of the row data and the number of the first chain tables in the first hash mapping area comprises:

[0032] reading any row data in the second file, reading the row data, and extracting the primary key and the primary key value information of the row data;

[0033] calculating the hash value of the row data based on a hash algorithm;

[0034] determining the second chain table corresponding to the hash value of the row data based on the hash value of the row data and the number of the second chain tables in the second hash mapping area;

[0035] storing the hash value, the primary key and the primary key value information of the row data in the second chain table corresponding to the hash value of the row data.

[0036] Further, the determining the first chain table corresponding to the hash value of the row data based on the hash value of the row data and the number of the first chain tables in the first hash mapping area comprises:

[0037] performing a remainder operation on the hash value of the row data based on the number of the second chain tables to obtain a remainder S2;

[0038] determining the S2th second chain table in the second hash mapping area as the second chain table corresponding to the hash value of the row data.

[0039] Further, each chain table comprises a plurality of nodes, and one node stores the hash value, the primary key and the primary key value information of a row data in the first file or the second file; the hash values of different chain tables in the same hash mapping area are different, and the primary keys of different nodes in the same chain table are different.

[0040] the performing set intersection operation on each first chain table and each second chain table based on the hash value, the primary key and the primary key value information stored in each chain table and storing the operation result in a third hash mapping area comprises:

[0041] Traverse each node of each first chain table and each node of each second chain table, and determine first type nodes, second type nodes and third type nodes based on the hash value and the primary key; the first type nodes include nodes with the same hash value and primary key in each first chain table and each second chain table; the second type nodes are nodes existing in each first chain table but not existing in each second chain table; the third type nodes are nodes existing in each second chain table but not existing in each first chain table;

[0042] Perform inner join operation on the first type nodes, left join operation on the second type nodes, and right join operation on the third type nodes to obtain an operation result;

[0043] Store the operation result in a third hash mapping area.

[0044] Further, the operation result includes a hash value, a primary key, a first pointer and a second pointer; the first pointer is used to point to the primary key value information corresponding to the first file, and the second pointer is used to point to the primary key value information corresponding to the second file;

[0045] Perform inner join operation on the first type nodes, left join operation on the second type nodes, and right join operation on the third type nodes to obtain an operation result, including:

[0046] For two nodes with the same hash value and primary key in the first type nodes, the hash value of the two nodes is used as the hash value of the operation result, the primary key of the two nodes is used as the primary key of the operation result, the first pointer is used to point to the primary key value information of the node belonging to the first chain table in the two nodes, and the second pointer is used to point to the primary key value information of the node belonging to the second chain table in the two nodes, to obtain the operation result and realize inner join operation;

[0047] For any node in the second type nodes, the hash value of the node is used as the hash value of the operation result, the primary key of the node is used as the primary key of the operation result, the first pointer is used to point to the primary key value information of the node, and the second pointer is used to point to nothing, to obtain the operation result and realize left join operation;

[0048] For any node in the third type nodes, the hash value of the node is used as the hash value of the operation result, the primary key of the node is used as the primary key of the operation result, the first pointer is used to point to nothing, and the second pointer is used to point to the primary key value information of the node, to obtain the operation result and realize right join operation.

[0049] Further, the operation result includes a hash value, a primary key, a first pointer and a second pointer; the first pointer is used to point to the primary key value information corresponding to the first file, and the second pointer is used to point to the primary key value information corresponding to the second file;

[0050] Based on the hash value and the primary key of each operation result, the operation result including the hash value, the primary key, the first pointer and the second pointer is stored in the third hash mapping area.

[0051] Further, the data unloaded by the upstream system is directly used in the files respectively set up.

[0052] Further, from another perspective, the primary keys of the two files are respectively set up. The fields capable of uniquely identifying the records of the file 1 and the file 2 are selected, which can be a combination of multiple fields. And the identification code generated after deformation by any processing rule is supported. Since it is a set operation, the position of the record line in the file is irrelevant to the result.

[0053] Further, from another perspective, the HashJoinRes structure is created, which includes but is not limited to two void* data1 and void* data2. The associated data result can be organized with as little memory occupation as possible, and any data type is supported.

[0054] Further, from another perspective, the Link structure is int iHash, char caData[D], long iLine, Link* psNext. It is respectively used for storing the hash value, part of the original data (length D), the line number of the data and the next node pointer. The HashMap structure is Link* linkHead[N], unsigned long iNum[N]. It is respectively used for constructing the HashMap structure using the link set and for calculating the usage of each bucket. N is the number of Hash buckets, and the larger N is, the higher the query efficiency is, but the memory consumption is increased.

[0055] Further, from another perspective, the HashJoinRes and the HashJoin declared in step 6 are declared, and the relationship therebetween is that the HashJoinRes is an instance of the HashMap structure, and any data can be saved in the HashMap form; the HashJoin is a variable recording the relationship between two data, which is saved in the HashMap form in the HashJoinRes, so that the output is facilitated and the information organization form is unified.

[0056] Further, from another perspective, the HashJoin structure is created, which is int iHash, char caData[D], void* data1, void* data2, Node* pNext. It is respectively used for storing the hash value, part of the original data, the memory address of the primary information and the memory address of the associated information.

[0057] Further, from another perspective, the left-right correlation algorithm can be applied to code comparison and other scenarios as the position information of the content mismatch in the two files. As the result of step 7, the lines of code file 1 and code file 2 are completely consistent; the lines newly added or modified in code file 1 can be obtained by using step 9; and the lines newly added or modified in code file 2 can be obtained by using step 10, which are characterized by being mirror images of each other, and the position information of the content mismatch in the two files can be further determined.

[0058] Further, from another perspective, the whole process includes but is not limited to implementation in multiple compiled languages such as java, c, python, etc.

[0059] Further, it is, the method is applied to a set operation device, three hash mapping areas are arranged in the memory of the set operation device, each hash mapping area includes a plurality of linked lists, and the set operation device includes:

[0060] A communication module is configured to read a first file to be operated line by line;

[0061] A processing module is configured to store the hash value, the primary key and the primary key value information of each row of data in a first linked list corresponding to the hash value of each row of data in the first hash mapping area;

[0062] The communication module is further configured to read a second file to be operated line by line;

[0063] The processing module is further configured to store the hash value, the primary key and the primary key value information of each row of data in a second linked list corresponding to the hash value of each row of data in the second hash mapping area;

[0064] The processing module is further configured to perform intersection and union set operations on each first linked list and each second linked list based on the hash value, the primary key and the primary key value information stored in each linked list, and store the operation result in a third hash mapping area; and generate an operation result file based on the third hash mapping group to realize set operation of external data. Further, the set operation device includes a memory and a processor, the memory stores a computer program, and the processor is configured to call and run the computer program stored in the memory to execute any of the above methods and operations.

[0065] The present application solves the problems of dependence on a database and resource consumption by adopting a mode of being independent of a database system. The present application solves the problem of low efficiency caused by processing data through SQL of a database and exporting a file after processing. The present application adopts the structure of HashMap and the mode of data correlation search algorithm. Any length of data stored in a general structure outside a database is converted into a fixed length of string or numerical value output through a hash algorithm. The output is a hash value, i.e., the characteristic value or hash value of the data.

[0066] The characteristic value or hash value of the data in the file 1 is loaded into the memory, and a hash (HASH) chain table structure is used to store the information. The record file 2 to be subjected to the union and intersection operation is subjected to the union and intersection operation with the file 1 in the memory in advance, and a set operation is generated. The whole process is operated in the memory, and does not depend on the external data association operation of the database. The universal and efficient external data set algorithm and technology are provided under the premise of not occupying the database resources and not using other commercial software, and without the steps of external data warehousing and result exporting, and the applicability is high. The application scenarios are various

[0067] The string described in the present application is named for convenience, and is not a limitation of the present application. Changing the string will not affect the essence of the present application, and should be considered as the protection scope of the present application.

[0068] The present application has the advantages that the above problems are solved, and a universal and efficient external data set algorithm is provided under the premise of not occupying data resources and not using other commercial software. In the past, the data is calculated by warehousing, the system resource consumption is large, and the time consumption is long, so that the efficiency of the set operation of two files and the like can be significantly improved. IV. DESCRIPTION OF DRAWINGS

[0069] Figure 1 It is the overall flow of the embodiment.

[0070] Figure 2 It is the data in the table of the embodiment.

[0071] Figure 3 It is the structure diagram in the memory of HashMap1 and HashMap2.

[0072] Figure 4 It is example data.

[0073] Figure 5 It is the representation diagram of the example data in the memory of HashMap1 and HashMap2 respectively.

[0074] Figure 6 It is a HashJoin structure diagram.

[0075] Figure 7 It is a database inner join (Inner Join) algorithm, and the HashJoinRes memory data after operation.

[0076] Figure 8 It is a database left join (Left Join) algorithm, and the HashJoinRes memory data after operation. V. DETAILED DESCRIPTION

[0077] The application will be further described below in connection with the accompanying drawings and examples.

[0078] Specific implementation one:

[0079] A bank needs to report the results of the association processing of 5 million data and 1 million data tables every day according to the requirements of the regulatory agency. Since the result data required by the regulatory agency is a Json message, and there are a large number of object arrays, the database operation is slow, and it also needs to be processed by other languages. At present, the set operation device using the present technology is provided with three hash mapping areas in the memory of the set operation device, each hash mapping area includes a plurality of linked lists, specifically including:

[0080] The first file to be operated is read line by line, and the hash value, primary key and primary key value information of each row of data are stored in the first linked list corresponding to the hash value of each row of data in the first hash mapping area.

[0081] The second file to be operated is read line by line, and the hash value, primary key and primary key value information of each row of data are stored in the second linked list corresponding to the hash value of each row of data in the second hash mapping area.

[0082] Based on the hash value, primary key and primary key value information stored in each linked list, the intersection and union set operation is performed on each first linked list and each second linked list, and the operation result is stored in the third hash mapping area.

[0083] Based on the third hash mapping group, an operation result file is generated to realize the set operation of external data.

[0084] The process is as shown in Figure 1 .

[0085] The specific execution of this example is as follows:

[0086] Step one: prepare the files F1, F2 to be operated.

[0087] Step two: set the primary key of F1 in the F1k1, F1k2 columns; the primary key of F2 in the F2k1, F2k2 columns. The memory and file system are as shown in Figure 3 .

[0088] Step three: build HashMap structures HashMap1, HashMap2, HashJoinRes in the memory, which are composed of linked lists (Link). The number of linked lists in HashMap is N, and the value of N is recommended to be 0.01-0.1 times the amount of data to be processed, and is a prime number. The linked list bucket structure is as shown in Figure 2 . The linked list node structure is as shown in Figure 3

[0089] Step four: read F1 line data, extract the primary key F1k1, F1k2, and calculate the Hash value H. Calculate H divided by the number of linked lists N remainder S. Store the Hash value H, partial primary key value information Data in HashMap1 corresponding to the S number of linked list, until F1 read.

[0090] Step five: read F2 line data, extract the primary key F2k1, F2k2, calculate the Hash value, remainder S, and store the Hash value H, partial primary key value information Data in HashMap2 corresponding to the S number of linked list, until F2 read. Take the data in the table as an example: Figure 4 At this time, the structure in HashMap1, HashMap2 memory is as follows: Figure 5 Traverse HashMap1, HashMap2, and compare the hash value and primary key information of the nodes on both sides. If they are consistent, create a structure HashJoin as follows: Figure 6 And add nodes to HashJoinRes. And void*data1, void*data2 point to the data in HashMap1, HashMap2 respectively. At this time, the database inner join (Inner Join) algorithm is realized, and the operation of HashJoinRes is as follows: Figure 7 Traverse the HashJoinRes structure and output to the file to complete the two file inner join operation without relying on the database.

[0091] If the corresponding hash value and primary key information of HashMap1 cannot be found in HashMap2, void*data1 points to the data in HashMap12, and void*data2 points to empty. At this time, the database left join (Left Join) algorithm is realized, and the operation of HashJoinRes is as follows: Figure 8 If you want to realize right join (Right Join), you can assign void*data1 to empty value and void*data2 to point to the information in HashMap2 to achieve it.

[0092] In order to speed up the work process and improve efficiency, the output form is flexible and various. This operation also uses the data unloaded by the upstream system. And it does not occupy database resources, and does not need to perform external data warehousing, result export and other links. The time consumed for association and result output is within 2 minutes. It has been running stably and accurately for two years. It is more than 50 times faster than the existing method of first warehousing and then association. The time complexity is linearly related to the data volume, and the whole process is realized by java and c programming languages. And batch commands containing compiled independent EXE files are used to realize it.

[0093] The above merely describes the preferred embodiments of the present application, and it should be pointed out that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should also be considered as falling within the protection scope of the present application.

Claims

1. A method for operating on an external data set, characterized in that, This is applied to a collection computing device, which has three hash mapping areas in its memory. Each hash mapping area includes multiple linked lists, and each linked list includes multiple nodes. Each node stores the hash value, primary key, and primary key value information of a line of data in a first file or a second file. The hash values ​​of different linked lists in the same hash mapping area are different, and the primary keys of different nodes in the same linked list are different. The set operation methods include: Read the first file to be processed line by line, and store the hash value, primary key and primary key value information of each line of data in the first linked list corresponding to the hash value of each line of data in the first hash mapping area; Read the second file to be processed line by line, and store the hash value, primary key and primary key value information of each line of data in the second linked list corresponding to the hash value of each line of data in the second hash mapping area; Based on the hash values, primary keys, and primary key value information stored in each linked list, intersection and union operations are performed on each first linked list and each second linked list, and the results are stored in the third hash mapping area. This includes: traversing each node of each first linked list and each node of each second linked list; determining first-type nodes, second-type nodes, and third-type nodes based on hash values ​​and primary keys; first-type nodes include nodes in each first linked list and each second linked list that have the same hash value and primary key; second-type nodes are nodes that exist in each first linked list but not in each second linked list; third-type nodes are nodes that exist in each second linked list but not in each first linked list; performing an inner join operation on the first-type nodes, a left join operation on the second-type nodes, and a right join operation on the third-type nodes to obtain the operation results; and storing the operation results in the third hash mapping area. Based on the third hash mapping area, a calculation result file is generated to realize the collection operation of external data.

2. The external data set operation method according to claim 1, characterized in that, The step of reading the first file to be processed line by line and storing the hash value, primary key, and primary key value information of each line of data in the first linked list corresponding to the hash value of each line of data in the first hash mapping area includes: For any line of data in the first file, read that line of data and extract its primary key and primary key value information; Calculate the hash value of this row of data based on the hash algorithm; Based on the hash value of the data in that row and the number of the first linked lists in the first hash mapping area, determine the first linked list corresponding to the hash value of the data in that row; The hash value, primary key, and primary key value of the row of data are stored in the first linked list corresponding to the hash value of the row of data.

3. The external data set operation method according to claim 2, characterized in that, The step of determining the first linked list corresponding to the hash value of the row of data based on the hash value of the row of data and the number of first linked lists in the first hash mapping area includes: Based on the number of items in the first linked list, perform a modulo operation on the hash value of the row of data to obtain the remainder S1; The S1th first linked list in the first hash mapping area is determined as the first linked list corresponding to the hash value of the data in that row.

4. The external data set operation method according to claim 1, characterized in that, The step of reading the second file to be processed line by line and storing the hash value, primary key, and primary key value information of each line of data in the second linked list corresponding to the hash value of each line of data in the second hash mapping area includes: For any line of data in the second file, read that line of data and extract its primary key and primary key value information; Calculate the hash value of this row of data based on the hash algorithm; Based on the hash value of the row of data and the number of the second linked lists in the second hash mapping area, determine the second linked list corresponding to the hash value of the row of data. The hash value, primary key, and primary key value of the row of data are stored in the second linked list corresponding to the hash value of the row of data.

5. The external data set operation method according to claim 4, characterized in that, The step of determining the second linked list corresponding to the hash value of the row of data based on the hash value of the row of data and the number of second linked lists in the second hash mapping area includes: Based on the number of items in the second linked list, perform a modulo operation on the hash value of the row of data to obtain the remainder S2; The S2nd second linked list in the second hash mapping area is determined as the second linked list corresponding to the hash value of the data in that row.

6. The external data set operation method according to claim 1, characterized in that, The calculation result includes a hash value, a primary key, a first pointer, and a second pointer; wherein the first pointer is used to point to the primary key value information corresponding to the first file, and the second pointer is used to point to the primary key value information corresponding to the second file; Perform inner join operation on the first type of nodes, left join operation on the second type of nodes, and right join operation on the third type of nodes to obtain the operation results, including: For two nodes in the first type of nodes that have the same hash value and primary key, the hash value of the two nodes is used as the hash value of the operation result, the primary key of the two nodes is used as the primary key of the operation result, the first pointer points to the primary key value information of the node belonging to the first linked list among the two nodes, and the second pointer points to the primary key value information of the node belonging to the second linked list among the two nodes, so as to obtain the operation result and realize the internal association operation. For any node in the second type of node, the hash value of the node is used as the hash value of the operation result, the primary key of the node is used as the primary key of the operation result, the first pointer points to the primary key value information of the node, and the second pointer points to null, so as to obtain the operation result and realize the left association operation; For any node in the third type of node, the hash value of the node is used as the hash value of the operation result, the primary key of the node is used as the primary key of the operation result, the first pointer points to null, and the second pointer points to the primary key value information of the node to obtain the operation result, thus realizing the right join operation.

7. The external data set operation method according to claim 6, characterized in that, The step of storing the calculation result in the third hash mapping area includes: Based on the hash value and primary key of each operation result, the results are stored in the third hash mapping area; the nodes in the third hash mapping area are used to store the operation results, including the hash value, primary key, first pointer and second pointer.

8. An external data set processing device, characterized in that, This is applied to a collection computing device, which has three hash mapping areas in its memory. Each hash mapping area includes multiple linked lists, and each linked list includes multiple nodes. Each node stores the hash value, primary key, and primary key value information of a line of data in a first file or a second file. The hash values ​​of different linked lists in the same hash mapping area are different, and the primary keys of different nodes in the same linked list are different. The set operation device includes: The communication module is used to read the first file to be processed line by line; The processing module is used to store the hash value, primary key, and primary key value information of each row of data in the first linked list corresponding to the hash value of each row of data in the first hash mapping area; The communication module is also used to read the second file to be processed line by line; The processing module is also used to store the hash value, primary key, and primary key value information of each row of data in the second linked list corresponding to the hash value of each row of data in the second hash mapping area; The processing module is also used to perform intersection and union operations on each first linked list and each second linked list based on the hash value, primary key and primary key value information stored in each linked list, and store the operation results in the third hash mapping area; and generate an operation result file based on the third hash mapping area to realize the set operation of external data; The processing module is specifically used to traverse each node of each first linked list and each node of each second linked list, and determine first-type nodes, second-type nodes, and third-type nodes based on hash values ​​and primary keys. The first-type nodes include nodes in each first linked list and each second linked list that have the same hash value and primary key. The second-type nodes are nodes that exist in each first linked list but not in each second linked list. The third-type nodes are nodes that exist in each second linked list but not in each first linked list. An inner join operation is performed on the first-type nodes, a left join operation is performed on the second-type nodes, and a right join operation is performed on the third-type nodes to obtain the operation results. The operation results are stored in the third hash mapping area.

9. A collective computing device, characterized in that, The aggregate computing device includes a memory and a processor. The memory stores a computer program, and the processor is used to call and run the computer program stored in the memory to perform the method as described in any one of claims 1 to 7.

Images