Pika-based reconciliation method, Pika-based reconciliation device, and storage medium

Abstract

The present invention discloses a Pika-based reconciliation method, a Pika-based reconciliation device, and a storage medium, belonging to the technical field of transaction data processing. The present invention reduces the memory used for data processing during the reconciliation process by obtaining local first transaction data and its corresponding channel transaction data, wherein the channel transaction data is transaction data from an external system, and both the first transaction data and the channel transaction data are user transaction data within a preset time period; importing the first transaction data and the channel transaction data into Pika; and determining the difference between the first transaction data and the channel transaction data through Pika's analysis commands.

Description

Technical Field

[0001] The present invention relates to the technical field of transaction data processing, and in particular to a Pika-based reconciliation method, a Pika-based reconciliation device, and a storage medium. Background Art

[0002] Reconciliation is the verification of transaction information from the previous settlement cycle, which can ensure the quality of accounting information.

[0003] Existing reconciliation uses Redis to achieve data reconciliation. However, Redis is a memory-based database with extremely high memory requirements. Excessive data volume can easily lead to memory overflow. Therefore, large-scale data reconciliation on Redis requires a lot of memory, which can easily lead to insufficient memory.

[0004] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of the present invention is to provide a Pika-based reconciliation method, a Pika-based reconciliation device and a storage medium, aiming to reduce the memory used for data processing in the reconciliation process.

[0006] To achieve the above object, the present invention provides a reconciliation method based on Pika, which includes the following steps:

[0007] Obtaining local first transaction data and its corresponding channel transaction data, where the channel transaction data is transaction data from an external system, and both the first transaction data and the channel transaction data are user transaction data within a preset time period;

[0008] Importing the first transaction data and the channel transaction data into Pika;

[0009] The difference data between the first transaction data and the channel transaction data is determined through the analysis command of Pika.

[0010] Optionally, the analysis command includes a sinterstore command and a sdiffstore command, and the step of determining the difference data between the first transaction data and the channel transaction data through the Pika analysis command includes:

[0011] Determine the intersection of the first transaction data and the channel transaction data by using the sinterstore command;

[0012] Determine a first difference set between the first transaction data and the intersection set by using the sdiffstore command;

[0013] Determine a second difference between the channel transaction data and the intersection using the sdiffstore command;

[0014] The difference data includes the first difference set and the second difference set.

[0015] Optionally, the step of importing the first transaction data and the channel transaction data into Pika includes:

[0016] Splitting the first transaction data according to a preset capacity to obtain a plurality of first sub-data, and splitting the channel transaction data according to the preset capacity to obtain a plurality of second sub-data;

[0017] The plurality of first sub-data and the plurality of second sub-data are imported into the Pika through slots corresponding to the plurality of threads in the thread pool.

[0018] Optionally, before the steps of dividing the first transaction data according to a preset capacity to obtain a plurality of first sub-data and dividing the channel transaction data according to the preset capacity to obtain a plurality of second sub-data, the method further includes:

[0019] Acquire local second transaction data, where the second transaction data is user transaction data before the preset time period;

[0020] Determine the number of shards for the first transaction data and the channel transaction data based on the second transaction data;

[0021] The thread pool is constructed according to the number of shards.

[0022] Optionally, the first transaction data includes a plurality of first sub-data, each of the first sub-data includes first order information, and the first order information is character data. The step of importing the first transaction data into Pika includes:

[0023] Converting the first order information into numerical data;

[0024] Determine a modulo operation result of the numerical data;

[0025] Determine the slot corresponding to each first sub-data according to the modulo operation result;

[0026] The corresponding first sub-data is imported into Pika through the slot.

[0027] Optionally, the first sub-data further includes second order information, and before the step of importing the corresponding first sub-data into Pika through the slot, the step further includes:

[0028] Concatenating the numerical data and the corresponding second order information to obtain a to-be-reconciled field of each first sub-data;

[0029] Determine an FST serialization result of the to-be-reconciled field as a data processing result corresponding to the first sub-data;

[0030] The step of importing the corresponding first sub-data into Pika through the slot includes: importing the corresponding data processing result into Pika through the slot.

[0031] Optionally, the step of importing the corresponding data processing result into Pika through the slot includes:

[0032] Each of the data processing results is imported into the Pika through a slot and a preset mark is added at the starting position and the ending position of each of the data processing results.

[0033] Optionally, the step of determining the difference data between the first transaction data and the channel transaction data through the Pika analysis command includes:

[0034] Allocating a thread corresponding to the slot among multiple threads;

[0035] The analysis command is executed in the thread to determine difference data between each first sub-data and the corresponding second sub-data in the channel transaction data.

[0036] In addition, in order to achieve the above-mentioned purpose, the present invention also provides a Pika-based reconciliation device, which includes: a memory, a processor, and a Pika-based reconciliation program stored on the memory and runnable on the processor, and the Pika-based reconciliation program is configured to implement the steps of the Pika-based reconciliation method described in any of the above items.

[0037] In addition, in order to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which a Pika-based reconciliation program is stored. When the Pika-based reconciliation program is executed by the processor, the steps of the Pika-based reconciliation method as described in any of the above items are implemented.

[0038] The present invention proposes a reconciliation method based on Pika. The method obtains local first transaction data and its corresponding channel transaction data, where the channel transaction data is transaction data from an external system, and both the first transaction data and the channel transaction data are user transaction data within a preset time period. The method then imports the first transaction data and the channel transaction data into Pika, and determines the difference data between the first transaction data and the channel transaction data through Pika's analysis command. Since Pika is a disk-based data storage system, storing data on disk can avoid memory overflow problems during the reconciliation process, thereby resolving the memory shortage problem that can easily occur when storing large amounts of reconciliation data in Redis, and reducing the memory used for data processing during the reconciliation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the hardware operating environment involved in an embodiment of a reconciliation device based on Pika according to the present invention;

[0040] Figure 2 This is a flow chart of the first embodiment of the reconciliation method based on Pika of the present invention;

[0041] Figure 3 This is a flow chart of a second embodiment of the reconciliation method based on Pika of the present invention;

[0042] Figure 4 This is a flow chart of a third embodiment of the reconciliation method based on Pika of the present invention;

[0043] Figure 5 for Figure 2 Detailed flow chart of step S20.

[0044] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] Reference Figure 1 , Figure 1 This is a structural diagram of a Pika-based reconciliation device in the hardware operating environment involved in an embodiment of the present invention.

[0047] like Figure 1As shown, the Pika-based reconciliation device may include: a processor, such as a central processing unit (CPU), and memory. The processor includes at least one CPU 1, which includes a CPU cache 11, a controller 12, registers 13, and an arithmetic unit 14. CPU 1 includes single-core CPUs and multi-core CPUs. A multi-core CPU has multiple logical CPU cores and can physically execute multiple instructions simultaneously. The memory includes memory 2 and disk 3. Memory 2 caches programs containing analysis commands and data. Disk 3 stores the programs containing the analysis commands and data.

[0048] Furthermore, CPU 1 reads the program from memory 2 into CPU cache 11. Controller 12 analyzes the program cached in CPU cache 11 and issues an analysis command to register 13. Register 13 receives the analysis command from controller 12, transmits the analysis command to arithmetic unit 14, and records the data processed by arithmetic unit 14, which then processes the data. The program from memory 2 is read from disk 3. The program includes analysis commands and data. The program corresponds to a process, which includes multiple threads.

[0049] It's important to note that processes and threads are dynamic. A process is the execution of a program, typically containing several threads. The thread's physical structure includes the program, data, and the task control block (TCB). A thread can create and terminate other threads, and multiple threads within the same process can execute concurrently, thus enabling concurrent program execution.

[0050] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the Pika-based reconciliation device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0051] like Figure 1 As shown, the memory as a readable storage medium may include an operating system, a data storage module, a network communication module, a user interface module, and a Pika-based reconciliation program.

[0052] exist Figure 1 In the Pika-based reconciliation device shown, the Pika-based reconciliation device calls a Pika-based reconciliation program stored in a memory through a processor, and executes the Pika-based reconciliation method provided by an embodiment of the present invention.

[0053] The first embodiment of the present invention provides a reconciliation method based on Pika, referring to Figure 2 , Figure 21 is a flow chart of a first embodiment of a Pika-based reconciliation method of the present invention. In this embodiment, the Pika-based reconciliation method includes:

[0054] Step S10: Acquire local first transaction data and its corresponding channel transaction data, where the channel transaction data is transaction data from an external system, and both the first transaction data and the channel transaction data are user transaction data within a preset time period;

[0055] The first transaction data and its corresponding channel transaction data include an order number and transaction information. The preset time period is the last working day after the day cut.

[0056] Specifically, the first transaction data is stored in the local system, obtained from the local system, and queried through multi-threading and paging.

[0057] Channel transaction data can be obtained from banks, third-party payment companies, China UnionPay, or China NetsUnion. Downloading channel transaction data from the internet can be done using the MQ or FTP file transfer protocols. Channel transaction data is imported into Pika using table storage format.

[0058] Step S20: importing the first transaction data and the channel transaction data into Pika;

[0059] It should be noted that Pika is a disk-based No-SQL database that supports multiple types of data structures and high throughput. At the same time, Pika is a Redis-like storage system that is compatible with most interfaces of string, hash, list, zset, and set. The underlying layer is based on RocksDB and is a disk-based data storage.

[0060] Specifically, first transaction data and channel transaction data are identified, where the first transaction data and the channel transaction data include corresponding order numbers, and the first transaction data and the channel transaction data are imported into Pika according to the order numbers.

[0061] Step S30: determining the difference data between the first transaction data and the channel transaction data through the Pika analysis command.

[0062] It should be noted that Pika's analysis commands include key commands, string commands, hash commands, list commands, collection commands, and server commands. Among them, collection commands include sunion commands, sinter commands, sdiffstore commands, sdiff commands, sinterstore commands, and sunionstore commands.

[0063] Specifically, the first transaction data and the channel transaction data are read, compared, and difference data between the first transaction data and the channel transaction data is determined. The difference data between the first transaction data and the channel transaction data is returned via a collection command. The difference data includes data where the first transaction data differs from the channel transaction data and data where the channel transaction data differs from the first transaction data.

[0064] The present invention proposes a reconciliation method based on Pika. The method obtains local first transaction data and its corresponding channel transaction data, where the channel transaction data is transaction data from an external system, and both the first transaction data and the channel transaction data are user transaction data within a preset time period. The method then imports the first transaction data and the channel transaction data into Pika, and determines the difference data between the first transaction data and the channel transaction data through Pika's analysis command. Since Pika is a disk-based data storage system, storing data on disk can avoid memory overflow problems during the reconciliation process, thereby resolving the memory shortage problem that can easily occur when storing large amounts of reconciliation data in Redis, and reducing the memory used for data processing during the reconciliation process.

[0065] Furthermore, based on the above embodiment, a second embodiment of the reconciliation method based on Pika is proposed. In this embodiment, referring to Figure 3 The analysis commands include the sinterstore command and the sdiffstore command, and the step S30 includes:

[0066] Step S31, determining the intersection of the first transaction data and the channel transaction data through the sinterstore command;

[0067] It should be noted that the sinterstore command is used to return the intersection of all given sets and store it at the destination address.

[0068] Specifically, the sinterstore command is executed to compare the first transaction data and the channel transaction data, filter out the same data between the first transaction data and the channel transaction data as an intersection, and store the intersection.

[0069] Step S32, determining a first difference set between the first transaction data and the intersection set by using the sdiffstore command;

[0070] It should be noted that the sdiffstore command is used to return the difference of all given sets and store it to the destination address.

[0071] Specifically, the sdiffstore command is executed to compare the first transaction data and the intersection, filter out different data between the first transaction data and the intersection as a first difference set, and store the first difference set.

[0072] Step S33: determining a second difference set between the channel transaction data and the intersection through the sdiffstore command; the difference data includes the first difference set and the second difference set.

[0073] Specifically, execute the sdiffstore command, compare the channel transaction data and the intersection, filter the different data between the channel transaction data and the intersection as a second difference set, and store the second difference set; execute the union command, merge the first difference set and the second difference set, and return the difference data, which includes the first difference set and the second difference set.

[0074] It should be noted that the union command is used to return the union of all given sets.

[0075] There is no particular order in which step S32 and step S33 are executed.

[0076] In this embodiment, the intersection of the first transaction data and the channel transaction data is determined by the sinterstore command, that is, the identical data of the first transaction data and the channel transaction data is determined, and further the first difference set of the first transaction data and the intersection is determined by the sdiffstore command, and the second difference set of the channel transaction data and the intersection is determined, that is, the difference data existing in the first transaction data and the difference data existing in the second transaction data are respectively determined, so as to merge the first difference set and the second difference set to determine the difference data, ensure the accuracy of the comparison between the first transaction data and the channel transaction data, thereby ensuring the accuracy of the difference data, reducing the omission of difference data, and improving the accuracy of reconciliation.

[0077] In other embodiments, the sdiff command can also be used to determine and return the difference between the first transaction data and the channel transaction data, where the difference is the difference data. Based on this, by comparing the first transaction data and the channel transaction data, the difference data can be quickly determined, thereby achieving rapid reconciliation.

[0078] Furthermore, based on any of the above embodiments, a third embodiment of the reconciliation method based on Pika is proposed. Figure 4 The step of importing the first transaction data and the channel transaction data into Pika includes:

[0079] Step S21: Slicing the first transaction data according to a preset capacity to obtain a plurality of first sub-data, and slicing the channel transaction data according to the preset capacity to obtain a plurality of second sub-data;

[0080] In this embodiment, the preset capacity is the total capacity of the first transaction data and the channel transaction data, which is determined based on the local transaction data before the current moment. In other embodiments, the preset capacity may also be a pre-set fixed value.

[0081] Specifically, the capacity of a sub-data is preset according to the preset capacity, the first transaction data is divided into multiple first sub-data according to the capacity of the sub-data, and the channel transaction data is divided into multiple second sub-data according to the capacity of the sub-data.

[0082] Assume that the preset capacity is 1G, 1G=1024MB, and the preset capacity of a sub-data is 32MB. It is known that the first transaction data is 320MB. According to the 32MB sub-data, the first transaction data is split to obtain multiple first sub-data. It can be seen that 320MB / 32MB=10 first sub-data are obtained. It is known that the second transaction data is 480MB. According to the 32MB sub-data, the channel transaction data is split to obtain multiple second sub-data. It can be seen that 480MB / 32MB=15 second sub-data are obtained.

[0083] Step S22: importing the plurality of first sub-data and the plurality of second sub-data into the Pika through the slots corresponding to the plurality of threads in the thread pool.

[0084] It should be noted that the thread pool is a form of multi-data processing. During processing, tasks are added to a queue and automatically started after a thread is created. Each thread uses the default stack size, runs at the default priority, and resides in a multi-threaded unit. If a thread in managed code is idle, the thread pool inserts another auxiliary thread to keep all processors busy. If all thread pool threads are busy and the queue contains pending work, the thread pool will create another auxiliary thread after a period of time. It is important to note that the number of threads created must be less than or equal to the maximum number of thread pool threads. Threads exceeding the maximum number can be queued and started only after other threads have completed.

[0085] A slot is the smallest storage unit in the local variable table. The local variable table represents a set of variable value storage spaces, used to store method parameters and local variables defined within methods. It is built on the thread stack. Data is distributed to threads via slots.

[0086] Specifically, a modulo operation is performed on the multiple first sub-data and the multiple second sub-data, and corresponding slots of the Pika are allocated according to the modulo operation results. The multiple first sub-data and the multiple second sub-data are allocated to corresponding threads through the corresponding slots.

[0087] In this embodiment, the first transaction data is divided according to the preset capacity to obtain multiple first sub-data, and the channel transaction data is divided according to the preset capacity to obtain multiple second sub-data. Then, the multiple first sub-data and multiple second sub-data are imported into Pika through the slots corresponding to the multiple threads in the thread pool. This ensures that each sub-data can be assigned to the corresponding thread, so that each sub-data can be processed. Multiple threads can ensure that the data is processed quickly, thereby improving the efficiency of data processing and improving the efficiency of reconciliation.

[0088] In other embodiments, the first transaction data and the channel transaction data may also be imported into Pika through a single thread.

[0089] Furthermore, in this embodiment, before step S21, the first transaction data is divided according to the preset capacity to obtain a plurality of first sub-data, the following steps are further included:

[0090] Step S01: Acquire local second transaction data, where the second transaction data is user transaction data before the preset time period;

[0091] It should be noted that the second transaction data is the user transaction data before the last working day after the day cut.

[0092] Specifically, the local database is read and the second transaction data is queried through multi-threading and paging.

[0093] Step S02: determining the number of shards of the first transaction data and the channel transaction data according to the second transaction data;

[0094] It should be noted that the system numbers locally stored transaction data in ascending order, including the second transaction data. A shard is an independent data area on a data storage node, which can be physically or logically isolated from other shards. The number of shards is the number of data areas after data is partitioned.

[0095] Specifically, the serial number of the second transaction data is read, the number of data items of the second transaction data is calculated according to the serial number, and the number of shards of the first transaction data and the channel transaction data is determined according to the number of data items.

[0096] Assume that the second transaction data read is numbered 1-100, the number of data items is calculated to be 100, and the number of shards is determined to be 10 based on the 100 data items. Then, these 100 data items will be divided into 10 and stored in each shard.

[0097] Step S03: construct the thread pool according to the number of shards.

[0098] The size of the thread pool can be fixed or variable.

[0099] Specifically, data is allocated to corresponding threads according to the number of shards, and a thread pool is constructed according to the required number of threads.

[0100] Assume that 1 million pieces of second transaction data are queried through paging, and 10,000 pieces of data are queried each time. The 10,000 pieces of data are divided into 10 shards, that is, each shard contains 1,000 pieces of data. The 1,000 pieces of data contained in each shard are assigned to 1 thread for processing, that is, 10 shards are assigned to 10 threads for processing, so a thread pool with 10 threads is constructed.

[0101] In this embodiment, the number of shards is determined based on the second transaction data, and the data is allocated to corresponding threads based on the number of shards, thereby constructing a thread pool. This method can ensure the rationality of allocating data to corresponding threads. By constructing a thread pool, the rational use of threads can be guaranteed, the convenience of data management can be improved, and resources can be fully utilized for rapid reconciliation.

[0102] In other embodiments, the first transaction data may be segmented according to database nodes to obtain a plurality of first sub-data.

[0103] It should be noted that the execution order of step S01 , step S02 , step S03 and step S10 is not critical.

[0104] Further, in this embodiment, referring to Figure 5 The first transaction data includes a plurality of first sub-data, each of which includes first order information, and the first order information is character data. The step of importing the first transaction data into Pika includes:

[0105] Step S23, converting the first order information into numerical data;

[0106] In this embodiment, the first order information is the order number. In other embodiments, the first order information may also be the order amount, etc. Character data is a text data type that does not have calculation capabilities, while numeric data is a data type that has calculation capabilities.

[0107] Specifically, the first order information is obtained, that is, the character data is obtained, the character data is matched with a known information list respectively, the type information of the character data is obtained, and according to the type information, a corresponding conversion function is called to convert the character data into numerical data.

[0108] Step S24, determining the modulo operation result of the numerical data;

[0109] It should be noted that the modulo operation is to find the remainder of the division of two numbers and is used for addressing. Pika includes a Cluster module, which can be used for task scheduling. The hash algorithm in the Cluster module is used to determine the slot corresponding to the data key and value. The Cluster module contains a fixed 16384 slots. Among them, the hash algorithm is: HASH_SLOT = CRC16(key) mod 16384, the hash algorithm calculates the CRC16 value of the key value, and then takes the modulo 16384 to obtain the HASH_SLOT corresponding to the key, that is, to determine the slot of the data.

[0110] Specifically, the numerical data is read, a modulo operation is performed on the numerical data, and a modulo operation result of the numerical data is determined.

[0111] Step S25, determining the slot corresponding to each first sub-data according to the modulo operation result;

[0112] It should be noted that a slot is the smallest storage unit in the local variable table. Data types less than or equal to 32 bits occupy only one slot, while data types up to 64 bits occupy two slots. Pika is a key-value distributed storage database, where key is the keyword and value is the value. Specifically, the hash value of the key of the first sub-data is stored in the corresponding slot.

[0113] Specifically, the CRC16 value of the key value is calculated according to the hash algorithm HASH_SLOT=CRC16(key)mod 16384, and then the modulus is 16384. The slot corresponding to each key key is determined according to the modulus operation result, that is, the slot corresponding to each first sub-data.

[0114] Step S26: import the corresponding first sub-data into Pika through the slot.

[0115] It's important to note that slots manage the relationship between data and nodes. This can be understood as nodes corresponding to slots, and slots storing data. Nodes are the service clusters that the data subsystem uses to access data storage and are the main components of the data subsystem. Slots solve the mapping problem between data and nodes, ensuring even distribution of data to corresponding nodes.

[0116] Specifically, according to the slot corresponding to the first sub-data, it is allocated to the corresponding node to be stored in Pika.

[0117] Specifically, the corresponding first sub-data is imported into Pika through the slots corresponding to the first sub-data in the plurality of threads in the thread pool.

[0118] In this embodiment, the first order information is converted into numerical data to obtain the modulo operation result, and the slot corresponding to each first sub-data is further determined. The corresponding first sub-data is imported into Pika through the slot to facilitate data allocation and improve the convenience of data processing, thereby optimizing the reconciliation experience.

[0119] In other embodiments, data may be imported into Pika via a write request. Based on this, the first order information may also be numerical data, and the above step S23 may be omitted during the import process into Pika, and step S24 may be directly executed.

[0120] It should be noted that the above step S23 is executed during the process of importing channel transaction data into Pika.

[0121] Among them, when importing channel transaction data into Pika, it can be imported into Pika in an analogous manner to the import method of the first transaction data; or after the first transaction data is imported into Pika, data import rules corresponding to multiple second sub-data in the channel transaction data can be generated according to the corresponding position arrangement rules of different first sub-data in the first transaction data in Pika, and the corresponding second sub-data in the channel transaction data can be extracted according to the data import rules and imported into Pika. Specifically, the corresponding second sub-data can be imported into Pika through the slots corresponding to the multiple threads in the above-mentioned thread pool. Based on this, it is beneficial to ensure the correlation between the first sub-data and the second sub-data in Pika, so as to improve the efficiency of determining the difference data between the subsequent first sub-data and the second sub-data, thereby effectively improving the reconciliation efficiency.

[0122] Furthermore, in this embodiment, the first sub-data further includes second order information. Before the step of importing the corresponding first sub-data into Pika through the slot, the step further includes:

[0123] Step S001: Concatenate the numerical data and the corresponding second order information to obtain a field to be reconciled of each of the first sub-data;

[0124] The numerical data and its corresponding order information specifically refer to the order information belonging to the same first sub-data.

[0125] In this embodiment, the second order information is other order information other than the order number, for example, order content, order time, etc.

[0126] It should be noted that the second order information may be character data, may be numeric data, or may include character data and numeric data.

[0127] Specifically, a data splicing task is received, the numerical data and the corresponding second order information are read, and a splicing result of the field to be reconciled is generated, where the field to be reconciled includes the numerical data and the corresponding second order information.

[0128] Step S002: Determine the FST serialization result of the field to be reconciled as the data processing result corresponding to the first sub-data;

[0129] It should be noted that serialization is the process of converting an object's state information into a form that can be stored or transmitted. During serialization, an object writes its current state to a temporary or persistent storage area. After serialization, the object's state can be recreated by reading or deserializing it from the storage area. Using FST serialization allows for fast data storage or transmission.

[0130] Specifically, an FST serialization instruction is received, where the FST serialization instruction includes type information of the field to be reconciled, an FST serialization interface function is called according to the type information, and an FST serialization field to be reconciled corresponding to the type information is created. The FST serialization interface function is used to determine an FST serialization rule, add an FST serialization instance of the field to be reconciled to instantiate the field to be reconciled, perform data FST serialization on the FST serialization instance of the field to be reconciled, and output an FST serialization result as the data processing result corresponding to the first sub-data.

[0131] Step S003, the step of importing the corresponding first sub-data into Pika through the slot includes: importing the corresponding data processing result into Pika through the slot.

[0132] Specifically, the data processing result corresponding to each first sub-data is imported into Pika through the slot corresponding to each first sub-data.

[0133] In this embodiment, by concatenating numerical data and the second order information to obtain the field to be reconciled, the field to be reconciled is further serialized by FST, and the FST serialization result is imported into Pika as the data processing result corresponding to the first sub-data, which facilitates data storage and thus improves the convenience of data storage.

[0134] It should be noted that step S001, step S002 and step S003 are performed after step S23.

[0135] Furthermore, in this embodiment, the step of importing the corresponding data processing result into Pika through the slot includes:

[0136] Step S004: import each of the data processing results into the Pika through the slot and add a preset mark at the starting position and the ending position of each of the data processing results.

[0137] In this embodiment, the preset tag is specifically hash_tag.

[0138] It's important to note that the first subdata is imported into Pika using hash_tags. Hash tags mark the beginning and end of a partial string used in hashing, such as "{}" and "$$." This allows partial key strings to be used in hash calculations. When a key contains {}, the hash is not calculated for the entire key; only the string enclosed by {} is hashed.

[0139] Specifically, through the slot corresponding to each data processing result, preset marks are added to the starting position and the ending position of each data processing result, and each data processing result is stored in Pika according to the key of each data processing result.

[0140] In this embodiment, preset marks are added to the starting position and the ending position of each data processing result by slots, so as to facilitate the inspection of the integrity of data storage, improve the accuracy of data import, and thus improve the accuracy of reconciliation.

[0141] Furthermore, in this embodiment, the step of determining the difference data between the first transaction data and the channel transaction data through the Pika analysis command includes:

[0142] Step S005: Allocate a thread corresponding to the slot in multiple threads, and execute the analysis command in the thread to determine the difference data between each first sub-data and the corresponding second sub-data in the channel transaction data.

[0143] It should be noted that multiple threads can be executed in parallel, and multiple threads can be executed simultaneously by multiple CPUs or by a multi-core CPU.

[0144] Specifically, among multiple threads, a corresponding thread is allocated according to the slot, and the analysis command is executed in the thread. The intersection of the first sub-data and the channel transaction data is determined by the analysis command, the first difference set between the first sub-data and the intersection is determined by the analysis command, and the second difference set between the channel transaction data and the intersection is determined by the analysis command. The difference data includes the first difference set and the second difference set.

[0145] It should be noted that step S005 is before step S31 and after step S004.

[0146] Specifically, corresponding threads are allocated among multiple threads according to the slots; the sinterstore command is executed in the thread to determine the intersection between each of the data processing results and each of the second sub-data; the sdiffstore command is executed in the thread to determine the first difference set between each of the data processing results and the intersection; the sdiffstore command is executed in the thread to determine the second difference set between each of the second sub-data and the intersection; the difference data includes the first difference set and the second difference set.

[0147] In this embodiment, corresponding threads are allocated among multiple threads according to slots, and the analysis command is executed in the thread to determine the difference data between the first transaction data and the channel transaction data. This multi-threaded data processing method improves the efficiency of program execution, thereby improving the reconciliation efficiency.

[0148] In addition, an embodiment of the present invention also provides a Pika-based reconciliation device, which includes: a memory, a processor, and a Pika-based reconciliation program stored on the memory and runnable on the processor, and the Pika-based reconciliation program is configured to implement the relevant steps of any embodiment of the Pika-based reconciliation method above.

[0149] In addition, an embodiment of the present invention further provides a storage medium on which a Pika-based reconciliation program is stored. When the Pika-based reconciliation program is executed by a processor, the relevant steps of any embodiment of the above Pika-based reconciliation method are implemented.

[0150] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0151] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0152] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0153] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A reconciliation method based on Pika, characterized in that: The Pika-based reconciliation method includes the following steps: Obtaining local first transaction data and its corresponding channel transaction data, where the channel transaction data is transaction data from an external system, and both the first transaction data and the channel transaction data are user transaction data within a preset time period; Importing the first transaction data and the channel transaction data into Pika; Acquire local second transaction data, where the second transaction data is user transaction data before the preset time period; Determine the number of shards for the first transaction data and the channel transaction data based on the second transaction data; Splitting the first transaction data according to a preset capacity to obtain a plurality of first sub-data, and splitting the channel transaction data according to the preset capacity to obtain a plurality of second sub-data; Performing a modulo operation on the plurality of first sub-data and the plurality of second sub-data, allocating corresponding slots of the Pika according to the modulo operation result, and allocating the plurality of first sub-data and the plurality of second sub-data to corresponding threads through the corresponding slots, wherein the data is allocated to corresponding threads according to the number of shards, and a thread pool is constructed according to the required number of threads; Determining the difference data between the first transaction data and the channel transaction data through the Pika analysis command specifically includes: Allocate a corresponding thread according to the slot, and execute the analysis command in the thread; The intersection of the first sub-data and the channel transaction data, the first difference between the first sub-data and the intersection, and the second difference between the channel transaction data and the intersection are determined through the analysis command, wherein the difference data includes the first difference and the second difference.

2. The Pika-based reconciliation method according to claim 1, characterized in that: The analysis command includes a sinterstore command and an sdiffstore command, and the step of determining the intersection of the first sub-data and the channel transaction data, the first difference between the first sub-data and the intersection, and the second difference between the channel transaction data and the intersection through the analysis command includes: Determine the intersection of the first transaction data and the channel transaction data by using the sinterstore command; Determine a first difference set between the first transaction data and the intersection set by using the sdiffstore command; Determine a second difference between the channel transaction data and the intersection using the sdiffstore command; The difference data includes the first difference set and the second difference set.

3. The Pika-based reconciliation method according to any one of claims 1 to 2, characterized in that: The first transaction data includes a plurality of first sub-data, each of which includes first order information, and the first order information is character data. The step of importing the first transaction data into Pika includes: Converting the first order information into numerical data; Determine a modulo operation result of the numerical data; Determine the slot corresponding to each first sub-data according to the modulo operation result; The corresponding first sub-data is imported into the Pika through the slot.

4. The Pika-based reconciliation method according to claim 3, characterized in that: The first sub-data further includes second order information. Before the step of importing the corresponding first sub-data into Pika through the slot, the step further includes: Concatenating the numerical data and the corresponding second order information to obtain a to-be-reconciled field of each first sub-data; Determine an FST serialization result of the to-be-reconciled field as a data processing result corresponding to the first sub-data; The step of importing the corresponding first sub-data into the Pika through the slot includes: The corresponding data processing results are imported into the Pika through the slot.

5. The Pika-based reconciliation method according to claim 4, characterized in that: The step of importing the corresponding data processing result into the Pika through the slot includes: Each of the data processing results is imported into the Pika through the slot and a preset mark is added at the starting position and the ending position of each of the data processing results.

6. A Pika-based reconciliation device, characterized in that: The Pika-based reconciliation device includes: a memory, a processor, and a Pika-based reconciliation program stored in the memory and executable on the processor, wherein the Pika-based reconciliation program is configured to implement the steps of the Pika-based reconciliation method according to any one of claims 1 to 5.

7. A storage medium, characterized in that: The storage medium stores a Pika-based reconciliation program, which, when executed by a processor, implements the steps of the Pika-based reconciliation method according to any one of claims 1 to 5.

Images