A data management system and transaction processing method for highly competitive e-commerce business

Abstract

The present invention proposes a data management system, including an input and output management module, an executor module, a transaction management module, and a log management module. The present invention also proposes a transaction processing method for high-competition e-commerce loads, which binds working threads to data partitions, so that each thread is only responsible for its own partition, filters out invalid operations, and merges similar operations for execution, greatly reducing the competition for data locks between competing transactions.

Description

Technical Field

[0001] The present invention relates to the technical field of database system management, and in particular to a data management system and a transaction processing method for highly competitive e-commerce businesses. Background Art

[0002] With the development of e-commerce business, a large number of offline transactions have been replaced by online transactions. As the key pillar of transaction processing, the database management system is under increasing pressure. Unlike traditional physical store transactions, online transactions break the space limitation, and millions or even tens of millions of customers can initiate purchase requests at the same time. According to [6], Alibaba processed up to 491,000 transactions per second during the shopping festival. This kind of business requires the database to be able to handle high concurrent loads.

[0003] What's even more difficult is that high concurrency often leads to high contention for data. During a promotion, a large number of users will read or write the same inventory at the same time, causing multiple transactions to compete to access the same data items, resulting in high contention. High contention scenarios will increase system response time. When a request does not receive a response, users are likely to send repeated requests, further increasing the amount of invalid load and placing greater pressure on the database management system.

[0004] 1. Concurrency control: The mutual influence between concurrently executed transactions may lead to inconsistent database status, even if each transaction can maintain the correct state and no failure occurs. Therefore, the execution order of each step of different transactions must be standardized in some way. This specification is completed by the scheduler component of the database management system, and the whole process of ensuring that concurrently executed transactions can maintain consistency is called concurrency control.

[0005] 2. Serializable Schedules: If there exists a serial schedule S' such that for each database initial state, schedules S and S' have the same effect, we say that the schedule S is serializable.

[0006] 3. Two-phase locking (2PL): In each transaction, all lock requests precede all unlock requests. The "two phases" are the first phase of acquiring the lock and the second phase of giving up the lock. Under this condition, it can be guaranteed that the legal scheduling of consistent transactions is contention serializable. Two-phase locking, like consistency, is a condition that restricts the order of actions in a transaction. Transactions that obey the 2PL condition are called two-phase locking transactions.

[0007] 4. Optimistic concurrency control protocol: Assume that no non-serializable behavior occurs when scheduling transaction operations, and only recover when violations are obvious. The optimistic approach differs from blocking in that when a problem does occur, the only remedy is to abort and restart the transaction that attempted to participate in the non-serializable behavior.

[0008] In order to better handle high contention loads, the present invention provides the following definitions:

[0009] Definition of homogeneous operations: Operations are homogeneous if they satisfy the following three conditions:

[0010] 1) They are update operations generated by the same SQL template;

[0011] 2) Their updates to properties only involve increasing or decreasing a constant;

[0012] 3) They access the same tuple.

[0013] Definition 2: There exists an attribute a, and due to certain constraints, a must belong to a left closed interval.

[0014] 1) If an update operation O attempts to subtract a positive number from the value of attribute a, then O is a CC-Update;

[0015] 2) If an update operation O attempts to add a positive number to the value of attribute a, then O is a CS-Update;

[0016] In both cases above, a is called a constrained attribute of O.

[0017] Problems with existing technologies:

[0018] In the implementation of the concurrency control module of the current database management system, the optimistic concurrency control protocol rolls back frequently in high-contention scenarios, resulting in poor performance [4]; lock-based concurrency control requires frequent locks and releases, and when the concurrent transaction conflict rate is high, serious lock contention will occur. Even if the concurrency control protocol is optimized [13,18], write-write conflicts on the same data item will not be alleviated, which seriously reduces the parallelism between transactions and thus limits the throughput of the database management system.

[0019] In recent years, a lot of work has been done to improve the performance of database management systems when handling highly competitive workloads. Orthrus

[13] proposed two design principles. One is to assign dedicated threads to manage concurrency control. Each concurrency control thread is responsible for a disjoint subset of database objects to avoid data movement and synchronization overhead. The other is to grant locks to transactions in a consistent order to avoid deadlocks. MOCC

[18] introduced a locking mechanism in OCC. When a transaction accesses high-temperature data, it needs to acquire a lock. This method reduces the rollback rate of transactions accessing hot data. All of these techniques are not optimized for homogeneous operations and invalid operations.

[0020] Multiple Query Optimization (MQO)

[15] executes a subquery only once and does not count the number of times the subquery appears in the OLAP query. This method only supports sharing resources in a single query, provided that there are multiple similar subqueries in the query. MQJoin[7] and CJOIN[2] use pipelines to share resources in connections. However, connections are very rare in OLTP workloads. SharedDB[5] and BatchDB[8] consider OLTP workloads. SharedDB batches queries and compiles them into a large query plan. Different queries share operators. BatchDB separates OLTP replicas from OLAP replicas, and resources can be shared between OLAP queries. The above methods focus on processing OLAP queries. OLTPShare

[12] is closest to the method of the present invention. Statements in OLTP workloads are merged to adapt to high-load scenarios. However, OLTPShare only merges single-point read-only operations, while the focus of the present invention is writes. Moreover, OLTPShare requires that the merged operations must be the only operations in the transaction to which they belong. Summary of the invention

[0021] The resolution of write-write conflicts is complex in the processing of highly competitive e-commerce workloads. In the e-commerce field, fierce competition is mainly due to the need to reduce the inventory of purchased products in the database for each sales transaction. The industry usually intercepts requests at the service layer by building queues to limit traffic

[10] . This solution is closely coupled to the specific business. Once the business changes, the service layer code also needs to be modified, which increases the burden on developers.

[0022] The present invention proposes a data management system for highly competitive e-commerce businesses, including the following modules:

[0023] Input and output management module: This module includes a filter and a distributor; the filter receives the operation request sent by the client, pre-processes it, and filters the invalid operations with reference to the filter table; the distributor distributes the SQL operations that attempt to add, delete, modify, and query the database to the executor module, and distributes the transaction operations including transaction commit, transaction rollback, and transaction start to the transaction management module;

[0024] Executor module: The executor module consists of several worker threads, the specific number of which is user-configurable; each worker thread is only responsible for its own data partition and maintains the data lock table of its own partition; the executor module reads data from the storage layer and records the changes of the operation in the context of the corresponding transaction;

[0025] Transaction management module: creates and deletes context for transactions, notifies the log management module to write logs, and writes the changes made by the worker thread to the transaction context back to the storage;

[0026] Log management module: receives log write requests from the transaction management module and writes transaction logs back to disk.

[0027] In the present invention, for the data management system of highly competitive e-commerce business, the filter first pre-processes the operation and parses the structure of the operation. If a quantity-restricted update is found, the filter table is checked to see whether there is a corresponding table entry, and if so, the update is filtered out; if a quantity-supplementing update is found, the filter table is also checked to see whether there is a corresponding table entry, and if so, the filter item is removed.

[0028] Based on the above system, the present invention proposes a transaction processing method for highly competitive e-commerce business, the method comprising the following steps:

[0029] Step 1: Parsing, filtering and distributing operations. Perform syntax parsing on the operations, determine whether the operations are likely to be successfully executed based on the internal situation of the database, filter out operations that have failed multiple times, and distribute the remaining operations to the corresponding worker threads according to their requested data.

[0030] Step 2: Execute the add, delete, modify and query operations. The executor locks the operations, reads data from the storage, and writes the changes made by the operations to the context of the transaction to which the operations belong. At the same time, similar operations are merged and executed. The merged operations only add a shared write lock.

[0031] Step 3: Execute transaction operations. The transaction management module creates and deletes contexts for transactions, and sends messages to the log manager on behalf of the transaction being submitted to record logs.

[0032] Step 4: Record the log to disk. After the log management module receives the message from the transaction management module requesting to write the log, it flushes the log in the cache to the disk and returns the log submission message to the transaction management module.

[0033] The present invention adds a filter table in the input and output management module to filter out invalid operations that are doomed to fail in advance.

[0034] The present invention identifies and merges similar operations in the executor module and executes the similar operations together.

[0035] The present invention logically binds data and working threads, and each working thread is only responsible for the data of its own partition.

[0036] The input and output management module of the present invention includes two modules: a filter and a distributor. The filter pre-processes the operation, filters out invalid operations, and replies the result to the client; the distributor sends the operation to the corresponding working thread in the executor module according to the data accessed by the operation.

[0037] The transaction processing method for high-competition e-commerce load proposed by the present invention binds the working thread to the data partition, and each thread is only responsible for its own partition, which greatly reduces the competition for data locks between competing transactions. The method of the present invention includes an input-output manager module, whose input is the operation request sent by the client, and the preprocessor in which the operation request is parsed to determine whether the operation is likely to be successfully executed; if not, the operation is directly filtered out to avoid causing unnecessary pressure on the data management system in the future. Then the input-output management module distributes the operation to the corresponding working thread according to the data that the operation attempts to access. The executor module is composed of several working threads, each executor is responsible for the operation access of a specified data partition and maintains the corresponding data lock table; in order to further increase the concurrency between operations, the executor will merge similar operations for execution, and these similar operations share the same data write lock. The input of the transaction manager is various transaction operations, such as transaction start, transaction commit, and transaction rollback. It is also responsible for managing the transaction context and notifying the log manager to log. The input of the log manager is the message notified by the transaction manager to log. After receiving the message, the log is written on the disk and the transaction manager is notified that the log is written successfully.

[0038] References

[0039] [1]Philip A.Bernstein,Vassos Hadzilacos,and NathanGoodman.1987.Concurrency Control and Recovery in Database Systems.Addison-Wesley.

[0040] [2]George Candea,Neoklis Polyzotis,and RadekVingralek.2011.Predictable performance and high query concurrency for dataanalytics.The VLDB Journal 20,2(2011),227–248.

[0041] [3]Giuseppe DeCandia,Deniz Hastorun,Madan Jampani,GunavardhanKakulapati,Avinash Lakshman,Alex Pilchin,Swaminathan Sivasubramanian,PeterVosshall,and Werner Vogels.2007.Dynamo:Amazon’s Highly Available Key-ValueStore.In Proceedings of Twenty-First ACM SIGOPS Symposium on OperatingSystems Principles.205–220.

[0042] [4]Jose M.Faleiro and Daniel J.Abadi.2015.Rethinking SerializableMultiversion Concurrency Control.Proceedings of the VLDB Endowment 8,11(2015),1190–1201.

[0043] [5]Georgios Giannikis,Gustavo Alonso,and DonaldKossmann.2012.SharedDB:killing one thousand queries with onestone.Proceedings of the VLDB Endowment 5,6(2012),526–537.

[0044] [6]Gui Huang,Xuntao Cheng,Jianying Wang,Yujie Wang,Dengcheng He,Tieying Zhang,Feifei Li,Sheng Wang,Wei Cao,and Qiang Li.2019.X-Engine:Anoptimized storage engine for large-scale E-commerce transaction processing.InProceedings of the 2019 International Conference on Management of Data.651–665.

[0045] [7]Darko Makreshanski,Georgios Giannikis,Gustavo Alonso,and DonaldKossmann.2016.MQJoin:efcient shared execution of main-memoryjoins.Proceedings of the VLDB Endowment 9,6(2016),480–491.

[0046] [8]Darko Makreshanski,Jana Giceva,Claude Barthels,and GustavoAlonso.2017.BatchDB:Efcient Isolated Execution of Hybrid OLTP+OLAP Workloadsfor Interactive Applications.In Proceedings of the 2017 ACM InternationalConference on Management of Data.37–50.

[0047] [9]Neha Narula,Cody Cutler,Eddie Kohler,and Robert Morris.2014.Phasereconciliation for contended in-memory transactions.In Proceedings of the11th USENIX conference on Operating Systems Design and Implementation.511–524.

[0048]

[10] Oracle.2015.Oracle Database 12c:Advanced Queuing Whitepaper.

[0049]

[11] Ippokratis Pandis,Ryan Johnson,Nikos Hardavellas,and AnastasiaAilamaki.2010.Data-oriented transaction execution.Proceedings of the VLDBEndowment 3,1-2(2010),928–939.

[0050]

[12] Robin Rehrmann,Carsten Binnig,Alexander Kihong Kim,WolfgangLehner,and Amr Rizk.2018.Oltpshare:the case for sharing in OLTPworkloads.Proceedings of the VLDB Endowment 11,12(2018),1769–1780.

[0051]

[13] Kun Ren,Jose M Faleiro,and Daniel J Abadi.2016.Design principlesfor scaling multi-core oltp under high contention.In Proceedings of the 2016International Conference on Management of Data.1583–1598.

[0052]

[14] Ohad Rodeh. 2008. B-trees, shadowing, and clones. ACM Transactions on Storage 3, 4 (2008), 2.

[0053]

[15] Timos K Sellis. 1988. Multiple-query optimization. ACM Transactions on Database Systems (TODS) 13, 1 (1988), 23–52.

[0054]

[16] Michael Stonebraker, Samuel Madden, Daniel J. Abadi, Stavros Harizopoulos, Nabil Hachem, and Pat Helland. 2007. The End of an Architectural Era: (It’s Time for a Complete Rewrite). In Proceedings of the 33rd International Conference on Very Large Data Bases. 1150–1160.

[0055]

[17] Boyu Tian, Jiamin Huang, Barzan Mozafari, and Grant Schoenebeck. 2018. Contention-aware lock scheduling for transactional databases. Proceedings of the VLDB Endowment 11, 5 (2018), 648–662.

[0056]

[18] Tianzheng Wang and Hideaki Kimura. 2016. Mostly-optimistic concurrency control for highly contended dynamic workloads on a thousand cores. Proceedings of the VLDB Endowment 10, 2 (2016), 49–60. Description of the Drawings

[0057] Figure 1 This is the system architecture of the present invention.

[0058] Figure 2 This is the filtering process of the filter of the present invention.

[0059] Figure 3 This is the merging process of the actuator modules of the present invention. DETAILED DESCRIPTION

[0060] The invention is further described in detail with reference to the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0061] The transaction processing method for highly competitive e-commerce business proposed in the present invention is used to establish, use and maintain a database, and to uniformly manage and control the database to protect the security and integrity of the database; the method manages data partitions for highly competitive e-commerce loads, binds data partitions to work threads, identifies invalid operations as early as possible and filters them, and merges similar operations for execution, and includes the following steps:

[0062] Step 1: The input and output management module records the information of the failed operation, and then based on this information, filters the next operation that arrives at the data management system. Some operations will not be continued to be executed, but directly return failure. The method of the present invention saves system resources consumed by those operations that are destined to fail.

[0063] The I / O manager module consists of two modules: filter and dispatcher.

[0064] The filter first pre-processes the operation and parses the structure of the operation. If a quantity-restricted update is found, the filter table is checked to see if there is a corresponding entry. If so, the update is filtered out. If a quantity-supplementing update is found, the filter table is also checked to see if there is a corresponding entry. If so, the filter item is removed.

[0065] Step 2: The distributor described in step 1 distributes the operation to the working thread specified in the executor module according to the data accessed by the operation request.

[0066] Step 3: The worker thread in the executor module is responsible for the specific execution of the operation, reading data from the memory and recording the changes requested by the operation in the corresponding transaction context. If there are several operations trying to add or subtract constant terms on the same data item, the executor will merge these operations for execution. These operations no longer need to lock the data items separately, but share the same write lock. This allows the transactions to which the operations belong to be executed in parallel, and the modification of the data items is reduced from multiple times to one. The merging of writes greatly reduces transaction contention for the same data.

[0067] Step 4: The transaction management module creates a context for the transaction, deletes the context, and sends a message to the log management module on behalf of the transaction requested to be submitted to record the log; the log management module records the log to the disk. When the log management module receives the message from the transaction management module requesting to write the log, it flushes the log in the cache to the disk and returns the log submission message to the transaction management module, finally notifying the client that the transaction is completed.

[0068] The present invention designs and implements a data management system for highly competitive e-commerce businesses according to the proposed method, including the following modules:

[0069] Input and output management module: This module consists of two parts, the filter and the distributor. The filter receives the operation request sent by the client, pre-processes it, and filters the invalid operations by referring to the filter table. The distributor distributes the SQL operations that attempt to add, delete, modify, and query the database to the executor module, and distributes the transaction operations (transaction commit, transaction rollback, transaction start) to the transaction management module.

[0070] Executor module: The executor module consists of several worker threads, and the specific number of worker threads is user-configurable. Each worker thread is only responsible for its own data partition and maintains the data lock table of its own partition. The executor reads data from the storage layer and records the changes of the operation in the context of the corresponding transaction.

[0071] Transaction management module: creates and deletes context for transactions, notifies the log management module to write logs, and writes the changes made by the worker thread to the transaction context back to the storage.

[0072] Log management module: receives log write requests from the transaction management module and writes transaction logs back to disk.

[0073] Example

[0074] Filtering invalid operations aims to reduce unnecessary use of system resources. To better achieve this goal, filtering should be performed in the input-output manager module (IOM), because the input-output manager module is located at the entrance of the entire system. This will minimize the time that invalid operations reside in the system. In e-commerce applications, update operations are most likely to be invalid operations. For a period of time after the rush purchase, although the goods are sold out, there will still be many operations trying to reduce the inventory of this product. These updates are bound to be invalid. In order to efficiently identify invalid operations and save system overhead, the filtering behavior is only for quantity-limited updates, and its definition is shown in Definition 2. The filter table is used to record invalid updates to determine whether the subsequent quantity-limited updates are valid. The entry in the filter table is a triple: <table name, primary key, column number>. If a quantity-limited update fails, its information will be recorded in the filter table. The table name and primary key specify the tuple that the update attempts to modify, and the column number indicates the specific column modified by the update.

[0075] The entire filtering process is as follows Figure 2 As shown. As mentioned earlier, the request sent by the client first arrives at the input-output manager module. Before the filtering operation, it needs to be preprocessed to obtain the necessary information. The input-output manager module parses the operation and obtains the primary key value of the tuple to be accessed according to the predicate. If the positioning predicate is based on the primary key, the task can be easily completed. However, if it is based on the secondary index, the secondary index index needs to be accessed to obtain the value of the primary key. After preprocessing, the way the input-output manager module handles the operation depends on the type of operation. If it is a quantity-restricted update or a quantity-supplemented update, its <table name, primary key, column number> will be extracted, possibly more than one, for comparison with the entries in the filter table. If they are found to have an intersection, there are two cases:

[0076] 1) For quantity-limited updates, it can be determined that it is an invalid operation. The input-output manager module filters the operation and directly returns the result to the client.

[0077] 2) For quantity supplementary updates, the I / O manager module removes the entries that produce intersections from the filter table and distributes the operations to the executors. This is because quantity supplementary updates may make previously failed quantity restriction updates with the same constraint attributes no longer invalid.

[0078] It is the executor's responsibility to add new entries to the filter table. When a worker thread finds that a limited update has failed due to a violation of the constraint attribute, it adds information about the operation to the filter in preparation for subsequent filtering.

[0079] The merger process is as follows Figure 3As shown. The first step is to filter out operations that may have homogeneous partners. Then, for each operation, its key information is stitched into a string, which is called the mode of the operation. Operations with the same mode are homogeneous, so they will be put in the same list for merging. The correspondence between modes and lists is recorded in the mode operation mapping table (PO-Map). If an operation does not meet the following two conditions, there must be no homogeneous operation with it:

[0080] 1) is an update operation;

[0081] 2) Define condition 2 in 1.

[0082] For such an operation, there is no need to calculate its mode. Just assign it a unique string as its key and put it into the PO-Map together with its key. Operations that meet the above two conditions have the opportunity to be merged. The information that determines which homogeneous list the operation belongs to includes: the table accessed by the operation, the attributes and corresponding arithmetic operators that the operation attempts to modify, the relational operators and attributes in the conditional predicate, and the primary key value of the tuple to be accessed. Based on this information, an operation mode is obtained. The worker thread will merge the expressions after the homogeneous operation set, such as Figure 3 In the example, "set a=a+1" of O1 and "set a=a+3" of O4 are combined into "set a=a+4".

[0083] In the executor module, each worker thread is responsible for data partitioning and maintains a PO-Map locally. The operations that arrive at the distributor are pre-processed, that is, the tuples to be accessed are already known. If an operation has multiple target tuples, the distributor splits it to ensure that the operations distributed to the worker threads do not need to access other data partitions. Each target tuple corresponds to a sub-operation. After splitting, all operations are single-point queries based on the primary key. The distributor can easily calculate the mode of an operation. Then, based on the primary key value, find the corresponding worker thread and put the operation into the list corresponding to the mode in the local PO-Map of the worker thread. If there is no current mode in the PO-Map, the distributor can add a new entry for it.

[0084] The worker thread takes one entry from the PO-Map at a time and checks the number of operations in the list. If there is more than one operation, they can be merged together. Data locks are required before performing the operation. Just like traditional write locks, the write lock requested by the merge operation is incompatible with other write locks and read locks. However, the merge write lock is shared by multiple transactions, and no transaction is allowed to reacquire the lock, even if some transactions already hold the lock. This is to ensure the serializability of transactions. In addition, if there is only one operation in the list, the data is locked and executed directly. At this stage, all data updates are local. The updates are written to the storage after the transaction commits. When the transaction is executed, the locks and updates it requested are recorded in the context. The updates of the merge operation only need to be recorded in the context of one transaction. This approach reduces the number of times the transaction operation is in memory. However, if one of the transactions participating in the merge is rolled back, the entire merge operation will be useless. In order to ensure the atomicity of the transaction, all transactions merged with it must be rolled back. Therefore, the transaction numbers of all operations participating in the merge are recorded in the same merge set (MergeSet). If one transaction in the merged set is rolled back, all other transactions must be rolled back together. In addition, transactions in the same merged set must be committed at the same time to avoid the unrecoverable situation where some transactions in the set have been committed but others have to be rolled back.

[0085] Basic Architecture

[0086] The present invention is a data management prototype system that focuses on the transaction processing module and does not include some functions that exist in a complete database system, such as a query optimizer. Because modern main memory is always large enough to store all the data involved in transaction processing

[16] , the present invention is implemented as an in-memory database. Data is always stored in memory and is not loaded from disk. However, for persistence, a log manager is built to write logs to disk.

[0087] Since optimistic concurrency control protocols frequently roll back under high contention loads, the present invention adopts a pessimistic protocol, two-phase locking (or 2PL) [1], for the concurrency control model. For simplicity, a single version of 2PL is implemented, which is sufficient to demonstrate the effectiveness of the present method.

[0088] Figure 1 The overall architecture and workflow of the present invention are shown. Data is divided into disjoint logical partitions based on primary key partitions. Each partition is managed by a different worker thread, and each worker thread maintains a local lock table for the data it is responsible for.

[0089] The request sent by the client first arrives at the Input Output Manager module (IOM). The Input Output Manager module consists of a filter and a distributor. After doing some pre-processing on the operation in the request, the filter determines whether the operation is valid. If this is an invalid operation, the system will directly return "operation failure" to the client. Otherwise, the operation is handed over to the distributor for further processing. The distributor is responsible for two things. One is to distribute each common operation (i.e., add, delete, modify, and query) to the corresponding worker thread based on the data to be accessed. The other is to pass the transaction operation (i.e., transaction start, transaction commit, or transaction abort) to the transaction executor (TE).

[0090] After the worker thread obtains the operation, it first locks the data to be accessed on the local lock table. The operation that cannot obtain the lock will be placed in the waiting queue and re-executed after the lock is released. It is worth mentioning that when the operation executed by the thread is suspended, the thread will not be blocked, but will directly execute the next operation. If the operation successfully locks the data, the worker process will first read the required data from the storage, and then record the updated results and the used locks in its transaction context. The execution strategy of the present invention ensures that the worker can merge all the same operations that have arrived and execute them together.

[0091] The transaction manager consists of two modules: the transaction executor and the context manager. The transaction executor selects different execution methods for different transaction operations. For a transaction operation, it only needs to create or delete the context of the corresponding transaction. For the transaction commit operation, the transaction executor notifies the log manager to write to the log. After the log is persisted, the log manager sends a commit message to the context manager. The context manager then processes the messages one by one and writes the updates in the transaction context to the storage. At this point, the transaction is actually completed. Finally, the input-output manager module replies to the client that the transaction has been committed.

[0092] All threads in the three main parts of the architecture of the present invention (i.e., the input / output manager module, the transaction manager, and the executor) are non-blocking, thus reducing the waste of resources caused by context switching in the kernel.

[0093] Experimental conclusion

[0094] Experimental environment

[0095] Experimental hardware configuration: This system is deployed on a physical node, which contains 2 CPUs, model Intel Xeon Silver 4110@2.1GHz CPU; 120GB memory; storage is 4 TB, RAID-5, 7200 rpm HDD disk. Gigabit Ethernet is used for communication between physical nodes.

[0096] Performance Evaluation

[0097] Experiment 1: Load a highly competitive order placement workload. Set the number of products to 100,000 and observe how the successful and failed throughput of the workload changes over time before and after the filter switch is turned on.

[0098] Time (seconds) 2 4 6 8 10 12 Successful throughput per second (filter enabled) 14434 17726 13207 5726 0 0 Failure throughput per second (filter enabled) 0 0 0 42450 82883 82208 Successful throughput per second (filter off) 14125 14984 14864 7472 0 0 Failed throughput per second (filter off) 0 0 0 18866 47989 46641

[0099] Experimental data shows that when the product is sold out (that is, after the 8th second), all requests in the load will fail. At this time, turning on the filtering switch will nearly double the system throughput.

[0100] Experiment 2: Load a highly competitive order load. Set the number of products as large as possible. Change the number of worker threads in the executor and observe the changes in load throughput before and after the merge switch is turned on.

[0101] Number of worker threads 1 2 3 4 5 Throughput per second (open and close) 22026.7 27932.9 31645.6 35714.2 36496.3 Throughput per second (closed) 18939.4 21459.2 22935.8 23809.5 25641.0

[0102] Experimental data show that when the merge function is turned on, the system throughput and scalability are significantly improved.

[0103] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A data management system for highly competitive e-commerce businesses, characterized in that: Includes the following modules: Input and output management module: This module includes a filter and a distributor; the filter receives the operation request sent by the client, pre-processes it, and filters the invalid operations with reference to the filter table; the distributor distributes the SQL operations that attempt to add, delete, modify, and query the database to the executor module, and distributes the transaction operations including transaction commit, transaction rollback, and transaction start to the transaction management module; The filter pre-processes the operation first, analyzes the structure of the operation, and if a quantity-restricted update is found, checks whether there is a corresponding entry in the filter table, and if so, filters out the update; if a quantity-supplementing update is found, checks whether there is a corresponding entry in the filter table, and if so, removes the filter entry; Executor module: The executor module consists of several worker threads, the specific number of which is user-configurable; each worker thread is only responsible for its own data partition and maintains the data lock table of its own partition; The executor module reads data from the storage layer and records the changes of the operation into the context of the corresponding transaction; Transaction management module: creates and deletes context for transactions, notifies the log management module to write logs, and writes the changes made by the worker thread to the transaction context back to the storage; Log management module: receives log write requests from the transaction management module and writes transaction logs back to disk.

2. A transaction processing method for highly competitive e-commerce business, characterized in that: Using the data management system as claimed in claim 1, the method comprises the following steps: Step 1: Parsing, filtering and distributing operations. Perform syntax parsing on the operations, determine whether the operations are likely to be successfully executed based on the internal situation of the database, filter out operations that have failed multiple times, and distribute the remaining operations to the corresponding working threads according to their requested data; add a filter table in the input and output management module to filter out invalid operations that are destined to fail in advance; Step 2: Execute the add, delete, modify and query operations. The executor locks the operations, reads data from the storage, and writes the changes made by the operations to the context of the transaction to which the operations belong. At the same time, similar operations are merged and executed. The merged operations only add a shared write lock. The data and worker threads are logically bound, and each worker thread is only responsible for the data of its own partition. Step 3: Execute transaction operations. The transaction management module creates and deletes contexts for transactions, and sends messages to the log manager on behalf of the transaction being submitted to record logs. Step 4: Record the log to disk. After the log management module receives the message from the transaction management module requesting to write the log, it flushes the log in the cache to the disk and returns the log submission message to the transaction management module.

3. The transaction processing method for highly competitive e-commerce business according to claim 2, characterized in that: Similar operations are identified and merged in the executor module, and similar operations are executed together.

4. The transaction processing method for highly competitive e-commerce business according to claim 2, characterized in that: The input and output management module consists of two modules: filter and distributor. The filter pre-processes the operation, filters out invalid operations, and replies the result to the client; the distributor sends the operation to the corresponding working thread in the executor according to the data accessed by the operation.

Images