A transaction data processing method and device

Abstract

The present invention discloses a transaction data processing method and device, and relates to the field of automatic programming technology. A specific implementation of the method includes receiving a transaction data processing request, obtaining the message field of the request, calling a preset mapping engine to identify the corresponding sub-library key value and partition key value, and then calculating the basket number where the data to be processed is located through a consistent hash algorithm; reading a preset deployment configuration table, and obtaining the database and deployment unit where the data to be processed is located based on the sub-library key value and the basket number; accessing the database and deployment unit where the data to be processed is located, and executing the transaction data processing request. Therefore, the implementation of the present invention splits the data in two dimensions or multiple dimensions to adapt to the multi-level management mode of the enterprise, simplify the processing logic of the aggregation business, and achieve more flexible sub-library and sub-table processing.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic program design, and in particular to a transaction data processing method and device. Background Art

[0002] As the enterprise develops, its backend system needs to manage more and more data and the business becomes more and more complex. A single database can no longer fully carry all the data and business. It is necessary to divide the data into separate databases and tables to improve the capacity and processing power of the entire system.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] However, most of the existing methods of database and table division are one-dimensional, which divides the data once according to a certain dimension. The data after division is completely scattered, which cannot adapt well to certain management business processing scenarios of the system before database and table division, and cannot be coordinated with the company's management structure. For example, in the banking industry, the provincial branch is basically a large management unit. Each provincial branch manages its own business personnel and customers independently and issues management reports. If the one-dimensional database and table division method is used directly to disperse the data of the whole bank, the data of the provincial branch cannot be cohesive, and the business processing of the provincial branch management report is complicated. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a transaction data processing method and device. To solve the problems of the prior art, the present invention splits the data into two dimensions or multiple dimensions to adapt to the multi-level management mode of the enterprise, simplify the processing logic of aggregation business, and achieve more flexible sub-library and sub-table processing.

[0006] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a transaction data processing method is provided, including receiving a transaction data processing request, obtaining the message fields of the request, calling a preset mapping engine to identify the corresponding sub-library key value and partition key value, and then calculating the basket number of the data to be processed through a consistent hashing algorithm; reading a preset deployment configuration table, and obtaining the database and deployment unit where the data to be processed is located based on the sub-library key value and the basket number; accessing the database and deployment unit where the data to be processed is located, and executing the transaction data processing request.

[0007] Optionally, before receiving the transaction data processing request, the process includes:

[0008] Obtaining system information of the database to be created, and based on the attribute identifier in the system information, calling the corresponding capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created;

[0009] According to the business scenario in the system information, a sub-library key value and a partition key value are selected to put system data with the same sub-library key value into the same sub-library;

[0010] The corresponding basket number is obtained by consistent hashing according to the partition key value, so that system data with the same basket number can be put into the same partition table.

[0011] Optionally, based on the attribute identifier in the system information, calling a corresponding capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created includes:

[0012] According to the attribute identifier in the system information, determine whether it is a migration system. If so, call the migration capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created; otherwise, call the new system capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created.

[0013] Optionally, call the new system capacity assessment component to determine the number of sub-databases and sub-tables of the database to be created, including:

[0014] Determine the number of databases deployed in the new system to obtain the number of sub-databases, obtain the expected future data volume of the system, calculate the single database data volume, and then divide the single database data volume by the preset single table maximum data volume, round up, and multiply by the preset expansion threshold to obtain the number of sub-tables; and

[0015] Call the migration capacity assessment component to determine the number of sub-databases and sub-tables of the database to be created, including:

[0016] According to the current system data volume, determine the growth rate in the future preset time, calculate the total data volume, and then calculate the number of sub-databases and sub-tables based on the preset single-database data volume threshold, the preset single-table data volume threshold and the preset expansion threshold.

[0017] Optionally, obtaining the message field of the request and calling a preset mapping engine to identify the corresponding sub-library key value and partition key value includes:

[0018] Obtain the message field of the request, determine whether the partition key and sub-library key exist in the message field and are not empty, if so, directly extract the sub-library key value and partition key value; otherwise, call a preset mapping function or a preset index table to identify the corresponding sub-library key value and partition key value based on the message field.

[0019] Optionally, the preset mapping function includes an interception function and a concatenation function;

[0020] Call the preset interception function or splicing function to obtain the sub-library key value and partition key value from the message field.

[0021] Optionally, it also includes:

[0022] Set the identification strategy field to configure a multi-line strategy; wherein the multi-line strategy includes a mapping function, an index table, and direct acquisition.

[0023] Optionally include:

[0024] The mapping relationship of the partition key of the sub-library in the mapping engine is realized by configuring the SpringEL expression in the transaction rule table.

[0025] In addition, the present invention also provides a transaction data processing device, including an acquisition module, which is used to receive a transaction data processing request, obtain the message field of the request, call a preset mapping engine to identify the corresponding sub-library key value and partition key value, and then calculate the basket number of the data to be processed through a consistent hash algorithm; a processing module, which is used to read a preset deployment configuration table, obtain the database and deployment unit where the data to be processed is located based on the sub-library key value and the basket number; access the database and deployment unit where the data to be processed is located, and execute the transaction data processing request.

[0026] Optionally, before the acquisition module receives the transaction data processing request, it includes:

[0027] Obtaining system information of the database to be created, and based on the attribute identifier in the system information, calling the corresponding capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created;

[0028] According to the business scenario in the system information, a sub-library key value and a partition key value are selected to put system data with the same sub-library key value into the same sub-library;

[0029] The corresponding basket number is obtained by consistent hashing according to the partition key value, so that system data with the same basket number can be put into the same partition table.

[0030] Optionally, the acquisition module calls a corresponding capacity assessment component based on the attribute identifier in the system information to determine the number of sub-libraries and sub-tables of the database to be created, including:

[0031] According to the attribute identifier in the system information, determine whether it is a migration system. If so, call the migration capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created; otherwise, call the new system capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created.

[0032] Optionally, the acquisition module calls a new system capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created, including:

[0033] Determine the number of databases deployed in the new system to obtain the number of sub-databases, obtain the expected future data volume of the system, calculate the single database data volume, and then divide the single database data volume by the preset single table maximum data volume, round up, and multiply by the preset expansion threshold to obtain the number of sub-tables; and

[0034] Call the migration capacity assessment component to determine the number of sub-databases and sub-tables of the database to be created, including:

[0035] According to the current system data volume, determine the growth rate in the future preset time, calculate the total data volume, and then calculate the number of sub-databases and sub-tables based on the preset single-database data volume threshold, the preset single-table data volume threshold and the preset expansion threshold.

[0036] Optionally, the acquisition module acquires the message field of the request, and calls a preset mapping engine to identify the corresponding sub-library key value and partition key value, including:

[0037] Obtain the message field of the request, determine whether the partition key and sub-library key exist in the message field and are not empty, if so, directly extract the sub-library key value and partition key value; otherwise, call a preset mapping function or a preset index table to identify the corresponding sub-library key value and partition key value based on the message field.

[0038] Optionally, the acquisition module is also used to:

[0039] The mapping relationship of the partition key of the sub-library in the mapping engine is realized by configuring the SpringEL expression in the transaction rule table.

[0040] An embodiment of the above invention has the following advantages or beneficial effects: the present invention introduces a sharding key to split data from two or more dimensions, which is convenient for the system to better manage data and better realize complex functions; and, a consistent hashing algorithm is used to locate the basket number, and the system capacity is designed in advance, which reduces the cost of data redistribution during capacity expansion and simplifies the impact of redistribution on the data system; at the same time, the flexible and changeable sharding key and sharding key mapping mechanism ensures that the system transaction implementation function is not constrained by data, greatly improving the diversity of the system's achievable functions; in addition, a changeable sharding key and sharding key mapping mechanism is realized through SpringEL expression.

[0041] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0043] Figure 1 is a schematic diagram of the main flow of the transaction data processing method according to the first embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of an example of a consistent hashing algorithm according to an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of introducing a node on a basket number according to an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of determining a sub-library key and a partition key instance according to an embodiment of the present invention;

[0047] Figure 5 is a schematic diagram of application data distribution after the database sharding key and partition key are determined according to an embodiment of the present invention;

[0048] Figure 6 is a schematic diagram of the main flow of a transaction data processing method according to a second embodiment of the present invention;

[0049] Figure 7 is a schematic diagram of main modules of a transaction data processing device according to an embodiment of the present invention;

[0050] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0051] Fig. 9 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0053] Figure 1 FIG. 1 is a schematic diagram of the main flow of the transaction data processing method according to the first embodiment of the present invention. Figure 1 As shown, the transaction data processing method includes:

[0054] Step S101, receiving a transaction data processing request, obtaining the message field of the request, calling a preset mapping engine to identify the corresponding sub-library key value and partition key value, and then calculating the basket number of the data to be processed through a consistent hashing algorithm.

[0055] In an embodiment, the consistent hashing algorithm maps the entire hash value space into a virtual ring. When removing or adding a server, the mapping relationship between the existing service request and the server processing the request can be changed as little as possible. Database sharding is to solve the problem of reduced database performance due to excessive data volume and to improve the throughput of the entire system at the same time. Certain rules are used to split the original independent database into several database components, and the large data table is split into several data tables, so that the data volume of a single database and a single data table is reduced, and the overall performance and capacity are enhanced. The sharding key is the first-level sharding identifier of the two-level sharding algorithm, through which a group (single) of shards can be located. The partition key is the second-level sharding identifier of the two-level sharding algorithm. The basket number can be calculated by using the consistent hashing algorithm for this field. The basket number is another name for the number of the shard table, which is calculated by the consistent hashing algorithm. It is a node on the hash ring. The baskets and database nodes can be formed by clockwise aggregation on the ring.

[0056] In other embodiments, after obtaining the message field of the request, it can be determined whether the partition key and the sub-library key exist in the message field and neither is empty. If so, the sub-library key value and the partition key value are directly extracted; otherwise, a preset mapping function or a preset index table is called to identify the corresponding sub-library key value and partition key value based on the message field.

[0057] That is to say, the message field includes the partition key identification strategy field and the sub-database key identification strategy field and neither of them is empty. Then, the sub-database key value and partition key value of the message field can be directly configured in the corresponding partition key identification strategy field and sub-database key identification strategy field in the transaction identification strategy table. For example, Table 1 is the transaction identification strategy table. The message with transaction code TX_ACC_INQ carries the ACC_NO field. The word in this field is the partition key, so just fill in the transaction field name in the partition key identification strategy field.

[0058] Table 1

[0059]

[0060] In addition, if the partition key value and sub-library key value are not directly included in the fields of the transaction message, the preset mapping function can be called (for example, the interception function and the splicing function can be used to obtain the sub-library key value and the partition key value from the fields of the transaction message.) or the preset index table (that is, the sub-library key value and the partition key value are found by reading the index table through a single or multiple fields). For example: in Table 1, the transaction code TX_UID_UPD, the first 2 digits of the UID are the branch number, and the last 15 digits are the account, then the UID field can be intercepted to obtain the partition key value and the sub-library key value. When locating the sub-library key for the transaction code TX_ACC_INQ, it is found that the configuration has a comma, the transaction field before the comma, and the index table (such as Table 2) type after the comma, then the system will use the value of the transaction ACC_NO field as the key, and use ACC_BR_MAPPING as the type to read the index table, and finally obtain TargetID=25, then the obtained value 25 is the sub-library key.

[0061] Table 2

[0062] Key Type TargetID 250010010031 ACC_BR_MAPPING 25 15019230014 PHONE_ACC_MAPPING 250010010031

[0063] In addition, some transaction data processing requests may require multiple methods to obtain the sub-library key value and partition key value. In this case, you can configure multiple lines of strategies in the identification strategy field, that is, directly obtain, through the preset mapping function and index table, until the sub-library key value and partition key value are identified.

[0064] It is worth noting that the present invention realizes the mapping relationship of the sub-library partition key in the mapping engine by configuring the SpringEL expression in the transaction rule table. Among them, SpringEL (Sping dynamic expression language) is a powerful dynamic language for querying and operating objects at runtime, with display functions such as explicit method calls and basic string template functions.

[0065] In other embodiments, the basket number of the data to be processed is obtained by calculating the consistent hash algorithm, such as Figure 2 As shown in the figure, according to the consistent hashing algorithm, the customer has implemented a fixed mapping of the 32-bit space on the ring, and then marked N baskets on the ring (the number of baskets is the number of sub-tables). Each customer can be mapped to a unique basket in a clockwise or counterclockwise direction. The number of baskets is fixed (that is, the number of sub-tables is determined). If the number of baskets changes, the customer number needs to be remapped to the basket, causing a large amount of data movement. Consider migrating as little data as possible during subsequent expansion, and introduce nodes above the basket number. The node is the database sub-database. Take the loan system as an example (such as Figure 3As shown in the figure, each branch can have one node, that is, one sub-database. If the business volume of a branch explodes, the basket can be used as a unit to move between sub-databases to split into multiple sub-databases. The relationship between customers and baskets remains unchanged, as little data is migrated as possible, and the application code will not be aware of it.

[0066] Step S102, read the preset deployment configuration table, and obtain the database and deployment unit where the data to be processed is located based on the sub-library key value and the basket number.

[0067] In some embodiments, the core of the two-level sub-library and sub-table strategy is to locate the sub-library group first, and then locate the sub-library and sub-table by the basket number. The function of the sub-library group is to aggregate data with high correlation (in one sub-library or in multiple sub-libraries) to avoid many unnecessary cross-library queries. As shown in Table 3, the deployment configuration table has data in rows 31, 32, and 33 in the sub-library where SPU1 is located. There are 16 sub-tables in the library, and the data in row 44 is in two sub-libraries, which are the sub-libraries corresponding to SPU2 and SPU3 respectively. The sub-library where SPU2 is located has only baskets 1 to 5, and the sub-library where SPU3 is located has data in baskets 6 to 16.

[0068] Table 3

[0069]

[0070]

[0071] Step S103, accessing the database and deployment unit where the data to be processed is located, and executing the transaction data processing request.

[0072] As some embodiments, before executing step S101, it is necessary to establish a system database and store the system data in the database before processing the transaction data. The specific implementation process includes obtaining the system information of the database to be created, calling the corresponding capacity assessment component based on the attribute identifier in the system information, and determining the number of sub-libraries and sub-tables of the database to be created; selecting the sub-library key value and the partition key value according to the business scenario in the system information, so as to put the system data with the same sub-library key value into the same sub-library; and obtaining the corresponding basket number through consistent hashing according to the partition key value, so as to put the system data with the same basket number into the same sub-table.

[0073] It can be seen that in this embodiment of the present invention, multiple hash rings are formed by using the sub-library key, and then the nodes on the ring are located by the consistent hash algorithm using the shard key and the same virtual node is located according to the same direction rule to form a sub-library. There can be one or more sub-libraries on a hash ring. Taking the e-commerce platform as an example, not only the product ID but also the store ID is required as the sub-library key. Assuming there are 1024 sub-tables, the data of the product ID is also in the 6th (6mod 1024=6) sub-table. The difference is that the sub-library where it is located is no longer determined by the product ID alone, but now it is determined by the sub-library key (store ID) together with the product ID. If the store ID is 123 (123 is in sub-library three and there is only one sub-library), the data of the product should eventually be in the 6th sub-table of the third sub-library.

[0074] Preferably, based on the attribute identifier in the system information, it can be determined whether it is a migration system. If so, the migration capacity assessment component is called to determine the number of sub-libraries and sub-tables of the database to be created; otherwise, the new system capacity assessment component is called to determine the number of sub-libraries and sub-tables of the database to be created. It is worth noting that if the migration capacity assessment component is called, the growth rate of the future preset time is determined based on the current system data volume, and the total data volume is calculated, and then the number of sub-libraries and sub-tables are calculated based on the preset single-library data volume threshold, the preset single-table data volume threshold and the preset expansion threshold. If the new system capacity assessment component is called, the number of databases deployed in the new system is determined to obtain the number of sub-libraries, so as to obtain the system's future expected data volume and calculate the single-library data volume, and then the single-library data volume is divided by the preset single-table maximum data volume, rounded up and multiplied by the preset expansion threshold to obtain the number of sub-tables.

[0075] It can be seen that for a system with sharded databases and tables, accurate capacity assessment can reduce or even fundamentally avoid data movement, ensuring the smooth and healthy operation of the system. For a new system, capacity assessment takes many aspects into consideration, generally based on cost and future expectations. For example: Based on cost, the system budget can only deploy up to 4 databases, so the number of sharded databases can only be 4. In addition, the number of system users is expected to reach 1 million data in the future, so a single database basically requires 250,000 data. If a single table is designed to not exceed 100,000 data, the number of sharded tables should be set to 25 / 10 and then rounded up = 3, so the number of sharded databases is 4 and the number of sharded tables is 3. At the same time, in order to better achieve capacity expansion, it is generally recommended to multiply the estimated number of sharded tables by the expansion threshold of 2, so the number of sharded databases is 4 and the number of sharded tables is 6. For the migrated system, the capacity assessment is generally based on the existing business volume and the expected growth in the next few years. For example, if the current system has 10 million data and is expected to grow by 100% in the next five years, the total data volume is 20 million. Assuming that the optimal storage capacity of a single database is 5 million (i.e., the data volume threshold of a single database) and the optimal storage capacity of a single table is 100,000 (i.e., the data volume threshold of a single table), 2000 / 500 = 4 sub-databases are required, and the number of sub-tables is 500 / 10 = 50. At the same time, in order to better achieve capacity expansion, it is generally recommended to multiply the number of sub-tables in the assessment area by 2, so the number of sub-databases is 4 and the number of sub-tables is 100.

[0076] As some other embodiments, the present invention divides the data in two dimensions or even multiple dimensions. In the specific design, the most common aggregate query conditions can be used as the sub-library key. Then, when performing online aggregate query, only the sub-library group where the sub-library key is located needs to be queried. There is no need to query all the sub-libraries concurrently. The implementation is simple and does not increase the amount of network requests. In batch processing, only a single or specified number of sub-library files need to be processed, which makes it simple and controllable.

[0077] In other words, make the data structure compatible with the company's management structure and minimize the aggregate queries caused by management transactions. When selecting the shard key, you should choose a field that is suitable for organizing transactions. For example, in a banking system, branches are generally used as management units. It is recommended to use the branch number as the shard key to aggregate branch data. For e-commerce platforms, the main management objects are stores, and management reports are mostly issued based on stores as the basic unit. It is recommended to use the store ID as the shard key. It is recommended to select a field that can evenly scatter the data for the shard key. In banking systems, it is recommended to use the customer number, and in e-commerce platforms, it is recommended to use the product ID. For example Figure 4 As shown in the figure, taking the bank's loan system as an example, in the table AR_ACC_BASE, select the BRANCH_NO branch number field as the sub-database key, and ACC_NO as the partition key. After the sub-database key and partition key are determined, the application data distribution is as follows Figure 5After the shard key and partition key are specified, the application needs to put the data with the same shard key in the same shard, and put the data with the same basket number obtained by the consistent hashing algorithm in the same shard table.

[0078] In summary, the present invention solves the existing problem that data cannot be split from multiple dimensions, and at the same time, the data is cohesively integrated according to business characteristics to solve the problem that aggregate query is difficult to implement due to data dispersion, that is, the aggregate query condition can be used as a partition key to gather data and implement simple aggregate query; in the process of data positioning, compared with the traditional solution that hard-codes the mapping relationship or requires that transactions must have or contain partition keys, the present invention uses Spring EL expressions to flexibly meet various mapping requirements, and there are no restrictions on transaction input data and usage methods; in the process of data expansion, the present invention can easily migrate data without recalculating the data. In addition, the present invention can also first vertically split the data and divide it into homogeneous microservice center management, and then horizontally split the data to perform sub-library or sub-table processing. Among them, horizontal splitting is to divide the data in a table into different tables or databases according to certain rules. Vertical splitting is to divide the table into different database tables according to functional modules without destroying the database design principles.

[0079] Figure 6 FIG. 1 is a schematic diagram of the main flow of the transaction data processing method according to the second embodiment of the present invention. Figure 6 As shown, the transaction data processing method includes:

[0080] Obtain the system information of the database to be created, and based on the attribute identifier in the system information, call the corresponding capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created; select the sub-library key value and the partition key value according to the business scenario in the system information to put the system data with the same sub-library key value into the same sub-library; obtain the corresponding basket number through consistent hash calculation according to the partition key value, so as to put the system data with the same basket number into the same sub-table. Receive a transaction data processing request, obtain the message field of the request, call the preset mapping engine to identify the corresponding sub-library key value and partition key value, and then calculate the basket number of the data to be processed through the consistent hash algorithm. Then read the preset deployment configuration table, and obtain the database and deployment unit where the data to be processed is located based on the sub-library key value and the basket number. Finally, access the database and deployment unit where the data to be processed is located, and execute the transaction data processing request, where the basket number is used as the table name suffix to uniquely identify the physical table on a single database.

[0081] That is to say, data records with associated relationships are packaged into a data tuple (such as a customer table and a detail table), and the data tuple is placed in the corresponding basket according to the hash value calculated by the partition key. One basket corresponds to a set of databases, that is, it is assigned to the corresponding database virtual node. According to the actual node corresponding to the virtual node (such as the actual node Node B corresponding to the virtual node B2), the node where the data basket is finally stored is determined. Therefore, the present invention simplifies the complexity of data migration after node expansion and reduces the complexity of data migration during node expansion according to the idea of ​​classifying the logical state and deployment state of data distribution, realizes the decoupling of the logical state and deployment state of data distribution, and avoids managing sharded data from the record level. Among them, the logical state is divided into a fixed number (at least greater than 500) of data baskets such as BK1 and BK2, that is, virtual nodes, which are evenly distributed in the 2^32 hash space. Each basket corresponds to a table family, which contains a group of data tables related to business logic. Subsequent migration can directly export and import operations on the table without operating on the data records.

[0082] Figure 7 Schematic diagram of main modules of a transaction data processing device according to an embodiment of the present invention. Figure 7 As shown, the transaction data processing device includes an acquisition module 701 and a processing module 702. The acquisition module 701 receives a transaction data processing request, acquires the message field of the request, calls a preset mapping engine to identify the corresponding sub-library key value and partition key value, and then calculates the basket number of the data to be processed through a consistent hash algorithm; the processing module 702 reads a preset deployment configuration table, acquires the database and deployment unit where the data to be processed is located based on the sub-library key value and the basket number; accesses the database and deployment unit where the data to be processed is located, and executes the transaction data processing request.

[0083] In some embodiments, before the acquisition module 701 receives the transaction data processing request, it includes:

[0084] Obtaining system information of the database to be created, and based on the attribute identifier in the system information, calling the corresponding capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created;

[0085] According to the business scenario in the system information, a sub-library key value and a partition key value are selected to put system data with the same sub-library key value into the same sub-library;

[0086] The corresponding basket number is obtained by consistent hashing according to the partition key value, so that system data with the same basket number can be put into the same partition table.

[0087] In some embodiments, the acquisition module 701 calls the corresponding capacity assessment component based on the attribute identifier in the system information to determine the number of sub-libraries and sub-tables of the database to be created, including:

[0088] According to the attribute identifier in the system information, determine whether it is a migration system. If so, call the migration capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created; otherwise, call the new system capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created.

[0089] In some embodiments, the acquisition module 701 calls the new system capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created, including:

[0090] Determine the number of databases deployed in the new system to obtain the number of sub-databases, obtain the expected future data volume of the system, calculate the single database data volume, and then divide the single database data volume by the preset single table maximum data volume, round up, and multiply by the preset expansion threshold to obtain the number of sub-tables; and

[0091] Call the migration capacity assessment component to determine the number of sub-databases and sub-tables of the database to be created, including:

[0092] According to the current system data volume, determine the growth rate in the future preset time, calculate the total data volume, and then calculate the number of sub-databases and sub-tables based on the preset single-database data volume threshold, the preset single-table data volume threshold and the preset expansion threshold.

[0093] In some embodiments, the acquisition module 701 acquires the message fields of the request, and calls a preset mapping engine to identify the corresponding sub-library key value and partition key value, including:

[0094] Obtain the message field of the request, determine whether the partition key and sub-library key exist in the message field and are not empty, if so, directly extract the sub-library key value and partition key value; otherwise, call a preset mapping function or a preset index table to identify the corresponding sub-library key value and partition key value based on the message field.

[0095] In some embodiments, the acquisition module 701 is further used to:

[0096] The mapping relationship of the partition key of the sub-library in the mapping engine is realized by configuring the SpringEL expression in the transaction rule table.

[0097] It should be noted that the transaction data processing method and the transaction data processing device described in the present invention have a corresponding relationship in terms of specific implementation contents, so the repeated contents will not be described again.

[0098] Figure 8An exemplary system architecture 800 is shown to which the transaction data processing method or transaction data processing apparatus according to the embodiment of the present invention can be applied.

[0099] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. Network 804 is used to provide a medium for communication links between terminal devices 801, 802, 803 and server 805. Network 804 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0100] Users can use terminal devices 801, 802, and 803 to interact with server 805 through network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0101] The terminal devices 801 , 802 , and 803 may be various electronic devices having a transaction data processing screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0102] The server 805 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 801, 802, and 803. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0103] It should be noted that the transaction data processing method provided in the embodiment of the present invention is generally executed by the server 805 , and accordingly, the computing device is generally arranged in the server 805 .

[0104] It should be understood that Figure 8 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0105] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 of a terminal device suitable for implementing an embodiment of the present invention. Fig. 9 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0106] like Fig. 9As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the computer system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0107] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display processor (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0108] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present invention are executed.

[0109] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0110] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor. For example, it may be described as follows: a processor includes an acquisition module and a processing module. The names of these modules do not, in some cases, limit the modules themselves.

[0112] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes receiving a transaction data processing request, obtaining the message field of the request, calling a preset mapping engine to identify the corresponding sub-library key value and partition key value, and then calculating the basket number of the data to be processed through a consistent hash algorithm; reading a preset deployment configuration table, obtaining the database and deployment unit where the data to be processed is located based on the sub-library key value and the basket number; accessing the database and deployment unit where the data to be processed is located, and executing the transaction data processing request.

[0113] According to the technical solution of the embodiment of the present invention, the data is split into two dimensions or multiple dimensions to adapt to the multi-level management mode of the enterprise, simplify the processing logic of aggregation business, and achieve more flexible sub-library and sub-table processing.

[0114] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A transaction data processing method, characterized in that: include: Receive a transaction data processing request, obtain the message field of the request, call the preset mapping engine to identify the corresponding sub-library key value and partition key value, and then calculate the basket number of the data to be processed through the consistent hash algorithm; wherein the transaction data processing request is a transaction data splitting request; the sub-library key is the first-level sub-library identifier of the two-level sub-library and table algorithm, through which a group of libraries can be located; the partition key is the second-level sub-library and table identifier of the two-level sub-library and table algorithm, and the basket number can be calculated by using the consistent hash algorithm for the column; the basket number is another name for the sub-table number, which is calculated by the consistent hash algorithm and is a node on the hash ring. The baskets can be collected clockwise on the ring to form a database node; Read the preset deployment configuration table, and obtain the database and deployment unit where the data to be processed is located based on the sub-library key value and the basket number; wherein the sub-library key value is first used to locate the sub-library group, and then the basket number is used to locate the sub-library and sub-table; the sub-library group is used to aggregate data with a correlation greater than or equal to a preset value; Access the database and deployment unit where the data to be processed is located, and execute the transaction data processing request.

2. The method according to claim 1, characterized in that Before receiving a request for transaction data processing, including: Obtaining system information of the database to be created, and based on the attribute identifier in the system information, calling the corresponding capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created; According to the business scenario in the system information, a sub-library key value and a partition key value are selected to put system data with the same sub-library key value into the same sub-library; The corresponding basket number is obtained by consistent hashing according to the partition key value, so that system data with the same basket number can be put into the same partition table.

3. The method according to claim 2, characterized in that Based on the attribute identifier in the system information, the corresponding capacity assessment component is called to determine the number of sub-libraries and sub-tables of the database to be created, including: According to the attribute identifier in the system information, determine whether it is a migration system. If so, call the migration capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created; otherwise, call the new system capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created.

4. The method according to claim 3, characterized in that Call the new system capacity assessment component to determine the number of sub-databases and sub-tables of the database to be created, including: Determine the number of databases deployed in the new system to obtain the number of sub-databases, obtain the expected future data volume of the system, calculate the single database data volume, and then divide the single database data volume by the preset single table maximum data volume, round up, and multiply by the preset expansion threshold to obtain the number of sub-tables; and Call the migration capacity assessment component to determine the number of sub-databases and sub-tables of the database to be created, including: According to the current system data volume, determine the growth rate in the future preset time, calculate the total data volume, and then calculate the number of sub-databases and sub-tables based on the preset single-database data volume threshold, the preset single-table data volume threshold and the preset expansion threshold.

5. The method according to claim 1, characterized in that Obtain the message fields of the request, and call a preset mapping engine to identify the corresponding sub-library key value and partition key value, including: Obtain the message field of the request, determine whether the partition key and sub-library key exist in the message field and are not empty, if so, directly extract the sub-library key value and partition key value; otherwise, call a preset mapping function or a preset index table to identify the corresponding sub-library key value and partition key value based on the message field.

6. The method according to claim 5, characterized in that The preset mapping functions include interception function and concatenation function; Call the preset interception function or splicing function to obtain the sub-library key value and partition key value from the message field.

7. The method according to claim 5, characterized in that Also includes: Set the identification strategy field to configure a multi-line strategy; wherein the multi-line strategy includes a mapping function, an index table, and direct acquisition.

8. The method according to any one of claims 1 to 7, characterized in that: include: The mapping relationship of the partition key of the sub-library in the mapping engine is realized by configuring the SpringEL expression in the transaction rule table.

9. A transaction data processing device, characterized in that: include: The acquisition module is used to receive a transaction data processing request, obtain the message field of the request, call a preset mapping engine to identify the corresponding sub-library key value and partition key value, and then calculate the basket number of the data to be processed through a consistent hash algorithm; wherein the transaction data processing request is a transaction data splitting request; the sub-library key is the first-level sub-library identifier of the two-level sub-library and table algorithm, through which a group of libraries can be located; the partition key is the second-level sub-library and table identifier of the two-level sub-library and table algorithm, and the basket number can be calculated by using the consistent hash algorithm for the column; the basket number is another name for the sub-table number, which is calculated by the consistent hash algorithm and is a node on the hash ring. The baskets can be collected clockwise on the ring to form a database node; The processing module is used to read a preset deployment configuration table, obtain the database and deployment unit where the data to be processed is located based on the sub-library key value and the basket number; access the database and deployment unit where the data to be processed is located, and execute the transaction data processing request; wherein, the sub-library key value is first used to locate the sub-library group, and then the basket number is used to locate the sub-library and sub-table; the sub-library group is used to aggregate data with a correlation greater than or equal to a preset value.

10. The device according to claim 9, characterized in that Before the acquisition module receives a request for transaction data processing, it includes: Obtaining system information of the database to be created, and based on the attribute identifier in the system information, calling the corresponding capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created; According to the business scenario in the system information, a sub-library key value and a partition key value are selected to put system data with the same sub-library key value into the same sub-library; The corresponding basket number is obtained by consistent hashing according to the partition key value, so that system data with the same basket number can be put into the same partition table.

11. The device according to claim 10, characterized in that The acquisition module calls the corresponding capacity assessment component based on the attribute identifier in the system information to determine the number of sub-libraries and sub-tables of the database to be created, including: According to the attribute identifier in the system information, determine whether it is a migration system. If so, call the migration capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created; otherwise, call the new system capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created.

12. The device according to claim 11, characterized in that The acquisition module calls the new system capacity assessment component to determine the number of sub-libraries and sub-tables of the database to be created, including: Determine the number of databases deployed in the new system to obtain the number of sub-databases, obtain the expected future data volume of the system, calculate the single database data volume, and then divide the single database data volume by the preset single table maximum data volume, round up, and multiply by the preset expansion threshold to obtain the number of sub-tables; and Call the migration capacity assessment component to determine the number of sub-databases and sub-tables of the database to be created, including: According to the current system data volume, determine the growth rate in the future preset time, calculate the total data volume, and then calculate the number of sub-databases and sub-tables based on the preset single-database data volume threshold, the preset single-table data volume threshold and the preset expansion threshold.

13. The device according to claim 9, characterized in that The acquisition module acquires the message fields of the request and calls a preset mapping engine to identify the corresponding sub-library key value and partition key value, including: Obtain the message field of the request, determine whether the partition key and sub-library key exist in the message field and are not empty, if so, directly extract the sub-library key value and partition key value; otherwise, call a preset mapping function or a preset index table to identify the corresponding sub-library key value and partition key value based on the message field.

14. The device according to any one of claims 9 to 13, characterized in that: Get module, also used for: The mapping relationship of the partition key of the sub-library in the mapping engine is realized by configuring the SpringEL expression in the transaction rule table.

15. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

16. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images