A multi-process shared database table updating method and device, and database system

Abstract

The present application discloses a method and device for updating a multi-process shared database table, and a database system. When multiple processes share data, the method can effectively improve the database table update speed and ensure the availability and consistency of shared data. The above method is applied to a multi-process shared database table update device and a database system, wherein the method includes: loading the data of the process when the process is started; monitoring whether an abnormal event occurs during the operation of the process; receiving a call instruction for a data update operation in the process through a data update interface; obtaining data to be updated after executing the data update operation according to the call instruction; judging whether the data to be updated meets the conditions for writing into the system database, and limiting the number of update operations for writing into the system database according to the conditions, so that more update operations occur in the process cache; if not, writing the data to be updated into the process cache, which is a distributed cache; if so, writing the data to be updated into the corresponding table of the system database.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a method and device for updating a multi-process shared database table, and a database system. Background Art

[0002] A database is a "warehouse that organizes, stores, and manages data according to data structures." It is a collection of large amounts of data stored in a computer, organized, shareable, and uniformly managed for a long period of time. A database management system allows for the addition, deletion, modification, and query of stored data. Modification operations include both modifying multiple records at once and modifying a single record at once. Traditional database table indexes, stored procedures, and database engine caches enable fast location-based modifications or cached, one-time modifications. However, for large numbers of aggregated single-record modification operations, especially for shared data in multi-process, highly concurrent systems (centralized modifications to database tables, such as record modifications to cash transfer tables in a highly concurrent UnionPay payment system), distance and modification heterogeneity can significantly delay database table modification operations. In the worst case, the aggregation of multiple Type II operations, such as when the database server and application server are far apart (e.g., >1000 km) and require connecting to different databases to modify different fields in different tables (highly heterogeneous modifications), can result in significant modification delays.

[0003] Currently, the speed-up technologies for modifying database tables mainly include: database table index acceleration, stored procedure acceleration, cache acceleration provided by the database engine, etc.

[0004] 1. Database table index acceleration: An index is a structure that sorts the values ​​of one or more columns in a database table. Using an index allows for quick access to specific information within the table. Typically, by creating a binary tree index on commonly queried fields in a database table, the time complexity for locating database records can be reduced from O(N) to O(log2N), thereby accelerating database table modifications.

[0005] 2. Stored Procedure Acceleration: A stored procedure is a set of SQL statements used to perform specific functions in a large database system. It is stored in the database and permanently valid after being compiled once. Users execute it by specifying the name of the stored procedure and providing parameters (if the stored procedure has parameters). Stored procedures are a key object in the database. When dealing with extremely large amounts of data, using stored procedures can achieve a significant increase in efficiency. However, stored procedures have the following bottlenecks:

[0006] 2.a. Usually only applicable to structured databases, such as MySQL, Oracle, SQL Server, etc.

[0007] 2.b. Specific stored procedures need to be written for different database tables and business needs. They are highly customizable and not universal.

[0008] 3. Multi-level cache acceleration: This is an acceleration mechanism that applies Amdahl's Law and is widely used in computer systems, such as system multi-level storage acceleration and database write acceleration. Typical memory cache systems that can be used for database write acceleration include Memcached, Redis, and Etcd. Generally, the data to be modified is first written to the memory cache system, and then the data is aggregated and written. However, processes may crash and multiple processes may share data. In these cases, the multi-level cache mechanism cannot maintain the continuous availability and consistency of data.

[0009] According to the CAP theorem, in a distributed system, consistency, availability, and partition tolerance are mutually exclusive. Consistency (C) refers to whether all data copies in a distributed system have the same value at the same time (equivalent to all nodes accessing the same, up-to-date copy of the data); availability (A) ensures that every request is responded to, regardless of success or failure; and partition tolerance (P) ensures that the loss or failure of any information in the system does not affect its continued operation. The essence of the CAP theorem is that it can be either AP, CP, or AC, but there is no CAP.

[0010] A multi-process system is a typical distributed system. When sharing data, if a unified database is used to store the shared data of multiple processes, data inconsistency will not occur, ensuring the C and P of the distributed system. However, if the system experiences a network partition or crash, some data will inevitably become inaccessible, and the availability condition cannot be met. In this case, the CP system is achieved, but the CAP cannot be met simultaneously. At the same time, because a single database is shared, when updating data, the table needs to be locked and unlocked to ensure that the data update sequence is consistent with the business time series, which leads to a significant decrease in the data update speed.

[0011] In summary, with respect to database table updates, existing technical solutions cannot effectively improve the efficiency of database table updates when multiple processes share data. Summary of the Invention

[0012] The present application provides a multi-process shared database table update method and device, and a database system for solving the defects existing in the database table update operation for multi-process shared data, which can effectively improve the database table update speed and ensure the availability and consistency of shared data.

[0013] In a first aspect, the present application provides a method for updating a multi-process shared database table, the method being applied to a process subsystem, the process subsystem being communicatively connected to a system database, the system database being connected to at least one process subsystem, and at least one process running on the process subsystem, comprising:

[0014] Loading data of the process when the process starts;

[0015] monitoring whether any abnormal events occur during the execution of the process;

[0016] Receiving a call instruction for a data update operation in the process through a data update interface;

[0017] After executing the data update operation according to the calling instruction, the data to be updated is obtained;

[0018] Determining whether the data to be updated meets the conditions for being written into the system database, and limiting the number of update operations written into the system database according to the conditions, so that more update operations occur on the process cache;

[0019] If not, the data to be updated is written into the process cache, which is a distributed cache;

[0020] If so, the data to be updated is written into the corresponding table of the system database.

[0021] Optionally, determining whether the data to be updated meets the conditions for being written into the system database includes:

[0022] Update the write threshold regularly according to the preset time. The write threshold is determined by the prior knowledge of data update operations in the past preset time and the preset sliding window;

[0023] Determine whether the data to be updated is greater than the write threshold.

[0024] Optionally, loading the data of the process when the process is started includes:

[0025] Starting the process and loading the program of the process into the memory of the computer;

[0026] querying from the process cache whether there is corresponding cache data according to the identifier of the process;

[0027] If so, the cache data is loaded into the memory of the process.

[0028] Optionally, after monitoring whether an abnormal event occurs during the running of the process, the method further includes:

[0029] If so, the data of the process is written into the process cache within a preset exit time.

[0030] Optionally, after writing the data to be updated into the table corresponding to the system database, the method further includes:

[0031] The data called by the calling instruction is cleared from the process cache.

[0032] A second aspect of the present application provides a data reading method, which is applied to an upper-layer application of a process subsystem, wherein the process subsystem is communicatively connected to a system database, and includes:

[0033] Receive a request to read target data;

[0034] Reading cache data corresponding to the target data from a process cache according to an identifier of the target data;

[0035] Reading table data corresponding to the target data from a table corresponding to the system database according to the identifier of the target data;

[0036] The cache data and the table data are added to obtain target data.

[0037] A third aspect of the present application provides a multi-process shared database table update device, comprising:

[0038] A data loading unit, configured to load data of the process when the process is started;

[0039] An abnormal event unit is used to monitor whether an abnormal event occurs during the operation of the process, and the abnormal event includes process abnormality and system abnormality;

[0040] A first receiving unit, configured to receive a call instruction for a data update operation in the process through a data update interface;

[0041] An updating unit, configured to obtain data to be updated after executing a data updating operation according to the calling instruction;

[0042] a first determining unit, configured to determine whether the data to be updated meets a condition for being written into the system database, and to limit the number of update operations written into the system database according to the condition, so that more update operations occur in the process cache;

[0043] a first execution unit, configured to write the data to be updated into a process cache, which is a distributed cache, when the first judgment unit determines that the data to be updated does not meet a write threshold for writing into the system database;

[0044] The second execution unit is configured to write the data to be updated into a table corresponding to the system database when the first judgment unit determines that the data to be updated meets a write threshold for writing into the system database.

[0045] Optionally, the first judgment unit further includes:

[0046] A determination module is used to regularly update the database write threshold according to a preset time. The database write threshold is determined by prior knowledge of data update operations within the past preset time and a preset sliding window;

[0047] The first judgment module judges whether the data to be updated is greater than the write threshold.

[0048] Optionally, the data loading unit includes:

[0049] A startup module, used to start the process and load the program of the process into the memory of the computer;

[0050] A second judgment module is used to query whether there is corresponding cache data from the process cache according to the identifier of the process;

[0051] The loading module is configured to load the cache data into the memory of the process when the second judgment module finds that there is corresponding cache data from the process cache according to the identifier of the process.

[0052] Optionally, the device further includes:

[0053] The third execution unit is configured to write the data of the process into the process cache when the abnormal event unit detects that an abnormal event is sent.

[0054] Optionally, the device further includes:

[0055] A clearing unit is used to clear the data called by the calling instruction in the process cache.

[0056] A fourth aspect of the present application provides a data reading device, comprising:

[0057] A receiving unit, configured to receive a request to read target data;

[0058] A first acquiring unit, configured to read cache data corresponding to the target data from a process cache according to an identifier of the target data;

[0059] A second acquiring unit is configured to read table data corresponding to the target data from a table corresponding to the system database according to an identifier of the target data;

[0060] An execution unit is configured to add the cache data to the table data to obtain target data.

[0061] A fifth aspect of the present application provides a database system, comprising a system database and at least one process subsystem, wherein the system database is communicatively connected to the process subsystem, wherein the process subsystem comprises a data processing unit and a process cache unit, wherein the data processing unit comprises the above-mentioned database table updating device, including:

[0062] The system database is used to store data shared by the process subsystems;

[0063] The process cache unit is used to cache and store data running on the process subsystem, and the process cache unit adopts a distributed cache;

[0064] The data processing unit is used to process data updates and abnormal events of the processes running on the process subsystem.

[0065] A sixth aspect of the present application provides a multi-process shared database table update device, comprising:

[0066] processor, memory, input and output units, and buses;

[0067] The processor is connected to the memory, the input and output unit, and the bus;

[0068] The processor specifically performs the following operations:

[0069] Loading data of the process when the process starts;

[0070] monitoring whether any abnormal events occur during the execution of the process;

[0071] Receiving a call instruction for a data update operation in the process through a data update interface;

[0072] After executing the data update operation according to the calling instruction, the data to be updated is obtained;

[0073] Determining whether the data to be updated meets the conditions for being written into the system database, and limiting the number of update operations written into the system database according to the conditions, so that more update operations occur on the process cache;

[0074] If not, the data to be updated is written into the process cache, which is a distributed cache;

[0075] If so, the data to be updated is written into the corresponding table of the system database.

[0076] A seventh aspect of an embodiment of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the aforementioned database table updating method is executed.

[0077] It can be seen from the above technical solution that in the multi-process shared database table update method, the distributed computing principle is adopted. When the process subsystem performs an update operation, since most of the data can be written to the cache, if all of this data is written to the system database, it will take too much time. In this application, it is first determined whether the data to be updated meets the conditions for writing to the system database. If so, it is written to the system database; if not, it is written to the process cache. This condition limits the number of update operations written to the system database, so that more update operations occur on the process cache. In addition, when multiple processes share data, the occupation of the system database by update operations can be reduced, and the update speed of the system database can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is a schematic diagram of the structure of an embodiment of the database system in this application;

[0079] Figure 2 This is a schematic diagram of the structure of an embodiment of the core component unit of the process subsystem in this application;

[0080] Figure 3 This is a flowchart of an embodiment of a method for updating a multi-process shared database table in this application;

[0081] Figure 4 This is a flowchart of an embodiment of data loading when starting a process in this application;

[0082] Figure 5 This is a flowchart of an embodiment of data protection during process abnormality in this application;

[0083] Figure 6 This is a flowchart of another embodiment of the method for updating a multi-process shared database table in this application;

[0084] Figure 7 This is a flow chart of an embodiment of a method for reading data in an upper-layer application in this application;

[0085] Figure 8 This is a schematic diagram of the structure of another embodiment of the database system in this application;

[0086] Figure 9 This is a flowchart of another embodiment of the method for updating a multi-process shared database table in this application;

[0087] Figure 10 This is a flow chart of another embodiment of a method for reading data in an upper-layer application in this application;

[0088] Figure 11 This is a schematic diagram of the structure of an embodiment of a multi-process shared database table update device in this application;

[0089] Figure 12 This is a schematic structural diagram of an embodiment of a data reading device in this application;

[0090] Figure 13 This is a structural diagram of another embodiment of the multi-process shared database table update device in this application. DETAILED DESCRIPTION

[0091] The present application provides a database table update method and device, and a database system, which can make up for the defects in database table update operations for multi-process shared data, effectively improve the update speed of the system database and maintain data availability and consistency.

[0092] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0093] In order to facilitate understanding of this application, the database system in this application is described below. Figure 1 , an embodiment of the database system in this application includes:

[0094] A system database 1 and at least one process subsystem 2, wherein the system database 1 is communicatively connected to the process subsystem 2, wherein the process subsystem 2 includes a data processing unit 21 and a process cache unit 22; the system database 1 is used to store data shared by the process subsystem 2; the process cache unit 22 is used to cache and store data running on the process subsystem, and the process cache unit adopts a distributed cache; the data processing unit 21 is used to handle data updates and abnormal events of processes running on the process subsystem.

[0095] In the embodiment of the present application, the communication between the system database and each process subsystem can be carried out through a local area network, a wide area network or other communication protocols, which are not limited here. A distributed computing network structure is formed between the system database and each process subsystem, thereby enabling each process subsystem to access the system database in parallel. For specific embodiments of the process subsystem, please refer to Figure 2The details are not detailed here. When a process running on the process subsystem needs to perform a data update operation, it does not write directly to the system database. Instead, it first determines whether the updated data meets the conditions for writing to the system database. If so, it writes to the system database. Otherwise, it writes to the process cache unit unique to the process subsystem. This reduces the process's usage of the system database and improves the efficiency of system database updates.

[0096] See also Figure 2 , Figure 2 This is a schematic diagram of an embodiment of the structure of the core component unit of the process subsystem 2 in the present application, which is used to execute the database table update method provided in the embodiment of the present application, including:

[0097] The data processing unit 21 includes:

[0098] Cache data loading module 211, which loads the cache data of a process when the process starts. When an abnormal event occurs, such as a system crash or a process exits due to an error, and the system / process is restarted, the module loads data from the process cache unit 22;

[0099] The data update module 212 is used to provide a data update interface for the upper layer data update operation and issue the data update operation, add the part of the data to the corresponding data in the process cache to obtain the data to be updated, thereby shielding the lower layer operation from the upper layer data update call;

[0100] The judgment module 213 is used to judge whether the data to be updated meets the conditions for updating to the system database 1;

[0101] The execution module 214 is used to execute the data writing operation. When the judgment module determines that the data to be updated needs to be updated in the system database 1, the data to be updated is written into the system database 1 and the updated data in the process cache unit 22 is cleared; otherwise, the data to be updated is updated in the process cache unit 22.

[0102] The exception handling module 215 may cause an exception in the cache data loading module 211, the data updating module 212, the judgment module 213, and the execution module 214, or even the entire process, which may cause an exception in the process (which may cause the process to crash); or the process subsystem may cause a system exception due to power outage or other reasons (which may cause the system to crash); in the case of a system exception, the system has a short exit time to protect the process running site. Before exiting due to an exception and / or crash, the exception handling module 215 writes the memory data that needs to be updated and written into the library into the process cache unit 22 to ensure that after the process is restarted, the cache data loading module 211 can load the last cached data from the process cache unit 22, thereby restoring the last running site of the process;

[0103] The process cache unit 22 is used to buffer and store the data of the process. It can adopt a distributed memory cache system to support high-speed reading and writing of data, and is deployed in the same computer room as the data processing unit 21 as much as possible to reduce network delay.

[0104] In order to more clearly illustrate the embodiments of the present application, the method executed in the above-mentioned database system and process subsystem will be described below.

[0105] See also Figure 3 In this application, an embodiment of a multi-process shared database table update method includes:

[0106] 301. Receive a call instruction for a data update operation in the process through a data update interface;

[0107] During the running of the process, the upper-layer application sends corresponding operation instructions to update the system database. The data update interface of the process receives the data update operation call of the upper-layer data update operation and then executes the corresponding data update operation.

[0108] 302. Execute a data update operation according to the call instruction to obtain data to be updated;

[0109] 303. Determine whether the data to be updated meets the conditions for being written into the system database. If not, execute step 304; if so, execute step 305.

[0110] Since there may be multiple consecutive update requests during the running process of the process, if each update needs to be written to the system database, it will take a lot of time to wait. Therefore, in order to reduce the process's occupancy of the system database, the number of update operations written to the system database is limited by the conditions so that more update operations occur on the process cache. In the embodiment of the present application, the condition for writing to the system database is that the data to be updated is greater than the write database threshold. In actual applications, other judgment conditions can be set according to corresponding needs, such as judging whether the data to be updated is less than the write database threshold or whether it is within a preset range, etc., which are not limited here.

[0111] 304. Writing the data to be updated into the process cache;

[0112] Write the updated data corresponding to the key that needs to be updated into the process cache to reduce the occupation of the system database by the update operation of the current process.

[0113] 305. Writing the data to be updated into a corresponding table in the system database;

[0114] When the data to be updated meets the conditions for writing into the system database, the data to be updated corresponding to the key to be updated is written into the system database table.

[0115] For example, a user table is stored in the system database. The user table has a score field (type is a floating point number, for example, the value is 102.31). Each update increases the score by a certain amount, and a write threshold (for example, the value is 1000) is set to limit the update operation conditions of the system database. The data update interface of the process receives an operation call to increase the score of the key Zhang San by 50 points. The accumulated score of multiple caches corresponding to Zhang San is 960 from the process cache, and the corresponding data update operation is performed to obtain the data to be updated (960+50=1100). Since 1100>1000, step 305 is executed to write the data with the key Zhang San into the score field of the user table of the system database.

[0116] 306. Clear the data called by the call instruction from the process cache.

[0117] Clear the data that has been written to the system database from the process cache.

[0118] Assuming that N records are written at a time, if the system database is updated directly, it is necessary to perform N record search and write operations on the system database, which is relatively time-consuming. In an embodiment of the present application, when performing an update operation, it is first determined whether the data to be updated meets the conditions for writing to the system database, and then the data to be updated is written to the system database or process cache. Based on this, most of the data to be updated can be written to the cache, and only a small part of the data to be updated needs to be written to the system database, thereby improving the update speed of the system database.

[0119] The following is a formal deduction of the database update speed improved by this application:

[0120] Assuming that the data update speed of writing to the system database in the prior art, that is, the number of read and write operations per second (IOPS, Input / Output Operations Per Second) is V1, after adopting the optimization method of writing to the process cache in this application, the speed of updating data to the process cache is V2. Assuming that a proportion P of data needs to be updated to the process cache, the proportion updated to the database is (1-P), and the optimized speedup ratio is:

[0121] S p =[V1*(1-P)+V2*P] / V1

[0122] Among them, take P = 0.9, V1 = 2000IOPS (traditional databases include MySQL or MongoDB), V2 = 100000IOPS (memory cache databases include Redis, etc.), then S p=[2000*0.1+100000*0.9] / 2000=45.1, that is, the speed after optimization is 45.1 times the speed before optimization.

[0123] See also Figure 4 , an embodiment of data loading when starting a process in this application includes:

[0124] 401. Start a process and load the program of the process into the memory of the computer;

[0125] 402. Query the process cache for corresponding cache data based on the process identifier. If so, execute step 403. If not, terminate the data loading process and continue executing other tasks of the process.

[0126] 403. Load the cache data into the memory of the process.

[0127] In the embodiment of the present application, the started process may be a process that is restarted after being interrupted and exited due to an exception. When exiting, the process subsystem has an exception protection mechanism (see Figure 5 , the details will not be described in detail this time) stores the running data of the process in the cache unit, so when starting a process, it is necessary to query whether there is corresponding cache data in the process cache. If so, the cache data is loaded into the memory of the process before continuing to execute other tasks of the process, so that the process can directly restore the data scene when it last exited, thereby ensuring the continuous availability of the process.

[0128] See also Figure 5 , an embodiment of data protection for process abnormality in this application includes:

[0129] 501. Monitor whether an abnormal event occurs during the operation of the process, wherein the abnormal event includes a process abnormality or a system abnormality. If so, execute step 502; if not, execute corresponding steps along with the process until the process ends.

[0130] Abnormal events may occur during the running of a process. Abnormal events include process abnormalities and system abnormalities. If the degree of process abnormality is low, it is an abnormal exit caused by abnormal operation of the process (such as division by 0). It can generally be captured by a statement block adapted to exception handling, so as to perform short-term processing actions; if it is a system abnormality (such as power outage), it will generally be captured by the operating system and passed to the corresponding system process and application process. The application process can also perform corresponding short-term processing actions for the system abnormality.

[0131] 502. Write the memory data of the process into the process cache within a preset exit time.

[0132] Within a short exit time, the memory data of the process is written to the process cache. For example, in Golang, a scene protection program that can adapt to system exceptions can be written in the defer() anonymous function or in the catch substatement block to quickly write the memory data to the process cache, thereby protecting the scene.

[0133] When a process failure or system failure occurs, the embodiment of the present application writes the memory data of the process to the process cache before the process exits to retain the process data and ensure that the process data is not lost.

[0134] See also Figure 6 Another embodiment of the database table updating method in this application includes:

[0135] 601. Start a process and load the program of the process into the memory of the computer;

[0136] 602. Query the process cache for corresponding cache data according to the process identifier. If yes, execute step 603; if no, execute step 604.

[0137] 603. Load the cache data into the memory of the process;

[0138] In the embodiment of the present application, steps 601 to 603 are similar to steps 401 to 403 of the aforementioned embodiment and are not described in detail here.

[0139] 604. Receive a call instruction for a data update operation in the process through a data update interface;

[0140] 605. Execute a data update operation according to the call instruction to obtain data to be updated;

[0141] In the embodiment of the present application, steps 604 to 605 are similar to steps 301 to 302 of the aforementioned embodiment and are not described in detail here.

[0142] 606. Update the database write threshold regularly according to a preset time. The database write threshold is determined by prior knowledge of data update operations within the past preset time and a preset sliding window.

[0143] In an embodiment of the present application, the judgment conditions for writing to the database / writing to the local cache can be determined by prior knowledge of the past period of time and a sliding window algorithm (Sliding Window Algorithm). For example, on average, more than 9 out of 10 operations to update the library are written to the cache of this process (this part is obtained based on historical records and is not limited to this time); the average value of the field to be updated in the recent past period of time * N (for example, N = 9) can be taken as the judgment condition for writing to the library / writing to the local cache.

[0144] For example, there is a user table with a points field. Assuming that the average value of the accumulated 1 million points increases or decreases in the past 10 days is 100, whether it exceeds 900 can be used as the judgment condition for writing to the database / local cache. This ensures that more than 9 out of 10 updates to the database are written to the local process cache. To ensure the adaptability of the prior to changes over time, the prior window can be slid with the current time. That is, the average value of the accumulated points added or subtracted in the past 10 days from the current time can be recalculated every day, thereby continuously refreshing the average value and ensuring that the update operation is likely to fall into the local process cache.

[0145] 607. Determine whether the data to be updated is greater than the write threshold. If not, execute step 608. If yes, execute step 609.

[0146] 608. If not, write the data to be updated into the process cache;

[0147] 609. If yes, write the data to be updated into the corresponding table of the system database;

[0148] 610. Clear the data called by the call instruction from the process cache.

[0149] In the embodiment of the present application, steps 608 to 610 are similar to steps 304 to 306 in the aforementioned embodiment and are not described again here.

[0150] It should be noted that, in the process of running the process, the embodiment of the present application still needs to monitor the system in real time whether any abnormal events occur. If an abnormal event occurs, exception handling is performed to ensure the data availability of the process. Based on this, the embodiment of the present application adopts and optimizes the map / reduce principle in distributed computing. The data update and exception handling of each process are equivalent to the map operation of each process, and the cache of each process is equivalent to the local data of the corresponding map operation; and the reduce is completed by each process (no need for a separate reduce process) using the inc (increase) or dec (decrease) operation (instead of the set operation, because the set operation will refresh all previous data) that retains the last operation on the system database, thereby completing the consistent aggregation of the data of each process (i.e., reduce), so that the present application has the effect of adapting to the data consistency of multiple processes.

[0151] The result of executing the above-mentioned write operation based on the existing technology is: it takes 55 seconds for a 1-core CPU / 2GB memory machine to write 10,000 records directly to the remote database; and the result obtained after optimizing the embodiment of the present application is: it takes 3ms to 2s (3ms is the best case when all records are written to the process cache, and 2s is the average operating condition), which increases the write speed by 27.5 to 18333.3 times the original.

[0152] See also Figure 7 An embodiment of the data reading method in the present application is applied to an upper-layer application of a process subsystem, including:

[0153] 701. Receive a request to read target data;

[0154] 702. Read cache data corresponding to the target data from the process cache according to the identifier of the target data;

[0155] 703. Read the table data corresponding to the target data from the corresponding table of the system database according to the identifier of the target data;

[0156] 704. Add the cache data and the table data to obtain target data.

[0157] In an embodiment of the present application, the upper-layer application sends a call for a data update operation to the underlying module through a data update interface according to the running status of the process, and shields the underlying operation from the upper-layer application.

[0158] In order to further understand the application of the database table updating method in this application, the following examples are provided to illustrate the application of the database system deployment embodiment and the database table updating method.

[0159] See also Figure 8 , an embodiment of the database system deployment in this application:

[0160] The database system consists of multiple process subsystems and system databases. The process subsystem 801 consists of a process server 8011 and a process cache server 8012. The system database is the system database server 802.

[0161] Process Server 8011 is responsible for deploying modules such as cache data loading, data updates, judgment and execution, and exception handling. It communicates with Process Cache Server 8012 and System Database Server 802. When it determines that a portion of data does not need to be updated to System Database Server 802, it directly updates it to Process Cache Server 8012. When it determines that an update to System Database Server 802 is necessary, it accumulates the data and updates it to System Database Server 802, clearing the data from Process Cache Server 8012. It is worth noting that multiple Process Servers 8011 support distributed cluster deployment of one or multiple programs, allowing multiple Process Servers 8011 to perform the same task simultaneously or independently, thereby achieving elastic support for highly concurrent and highly available systems.

[0162] The process cache server 8012 is used to deploy the process cache module and is responsible for data caching of each process subsystem. It usually adopts a mature open source distributed memory cache database system, such as Redis / Memcached / Etcd, etc., which are not limited here. These distributed systems can achieve more than 100,000 IOPS performance by adopting memory acceleration technology. It can be configured in synchronous / asynchronous disk flushing mode to ensure that data is not lost in the event of anomalies such as power outages on the process cache server 8012. Multiple servers can also be used to host the process cache to eliminate single points of failure.

[0163] The system database server 802 is used to deploy data shared by multiple process subsystems, including business data, user data, log data, etc., and can use MySQL, SQL Server, Oracle, MongoDB, etc. to ensure the ultimate consistency of data.

[0164] See also Figure 9 Based on the above embodiment, the process cache server adopts a Redis cluster (3 servers), the system database adopts MongoDB, and a three-copy cluster can be adopted. An embodiment of the database table update method in this application includes:

[0165] 901. Setting a cumulative integral value threshold;

[0166] The cumulative points value threshold is set by configuring parameters. Assume that it is set to Cmax = 1000 here. That is, when the points value of a user updated this time + the points value of the user in the process cache is greater than or equal to 1000, the operation of writing to the system database will be triggered.

[0167] 902. Obtain the points value of the user Ui that needs to be updated;

[0168] The point value Cj required to update the user UI this time is obtained through the data update interface.

[0169] 903. Obtain the current cache points value Ci of user Ui from Redis;

[0170] Read the value of key=Ui from Redis and set it to Ci. If the value of key=Ui cannot be obtained, Ci=0.

[0171] 904. Calculate the accumulated points of user Ui: Ct=Cj+Ci;

[0172] 905. Determine whether the accumulated integral value is greater than a threshold value. If so, execute step 906; if not, execute step 908.

[0173] Determine whether Ct is greater than Cmax. If so, execute step 906 to write the accumulated integral value into the system database. If not, execute step 908 to write the accumulated integral value into the process cache.

[0174] 906. Locate the record corresponding to user Ui in the user table of the system database MongoDB and perform an Inc (increase) operation on its points field;

[0175] 907. Clear the Value record with key=Ui from the process cache;

[0176] 908. Refresh the value of key=Ui in Redis to Ct;

[0177] 909. Determine whether data needs to be updated. If so, execute step 902. If not, end the data update execution process.

[0178] Because this application shields upper-layer application data from underlying operations, and the points value update mechanism in this embodiment of the application is to update the user's points value in the system database only when the points value in the process cache reaches a threshold, when the upper-layer application needs to read a user's current points value, there is a possibility that the points in the process cache may not reach the threshold and may not be updated to the system database in a timely manner. Therefore, when the upper-layer application reads the user's current points value, it needs to read the points value corresponding to the user in the process cache and the system database separately, and add them together to obtain the user's current points value.

[0179] See also Figure 10 In the embodiment of the present application, the database table updating method of the upper layer application corresponds to the above-mentioned database table data reading method, including:

[0180] 1001. Receive a request to read the points value of user Ui;

[0181] 1002. Obtain the current cache points value Ci of user Ui from Redis;

[0182] In the embodiment of the present application, it is assumed that the current cache integral value Ci is 50.

[0183] 1003. Obtain the point value Cd corresponding to user Ui from the user table of the system database MongoDB;

[0184] Assume that the point value Cd corresponding to user Ui obtained in the user table is 1688.

[0185] 1004. Calculate the accumulated points of user Ui: Cr = Ci + Cd.

[0186] The current integral value of the user is calculated according to Cr=Ci+Cd, Cr=50+1688, that is, Cr=1738 is the current integral value of the user.

[0187] It should be noted that the data reading method described in the embodiment of the present application is a case where a single process subsystem accumulates the corresponding field data of the system database. If multiple process subsystems have caches of this field data, the corresponding field data of multiple process subsystems need to be superimposed, which will not be repeated here.

[0188] See also Figure 11 , an embodiment of a multi-process shared database table update device in this application includes:

[0189] The data loading unit 1101 is used to load the data of the process when the process is started;

[0190] The abnormal event unit 1102 is used to monitor whether an abnormal event occurs during the process of running the process, and the abnormal event includes process abnormality and system abnormality;

[0191] The first receiving unit 1103 is configured to receive a call instruction for a data update operation in the process through a data update interface;

[0192] An updating unit 1104 is configured to obtain data to be updated after executing a data update operation according to the calling instruction;

[0193] The first judgment unit 1105 is used to judge whether the data to be updated meets the conditions for being written into the system database, and to limit the number of update operations written into the system database according to the conditions, so that more update operations occur in the process cache;

[0194] A first execution unit 1106 is configured to write the data to be updated into a process cache, which is a distributed cache, when the first judgment unit determines that the data to be updated does not meet the write threshold for writing into the system database;

[0195] The second execution unit 1107 is configured to write the data to be updated into a corresponding table of the system database when the first judgment unit determines that the data to be updated meets the write threshold for writing into the system database.

[0196] The third execution unit 1108 is configured to write the data of the process into the process cache when the abnormal event unit detects that an abnormal event is sent.

[0197] The clearing unit 1109 is configured to clear the data called by the calling instruction in the process cache.

[0198] In the embodiment of the present application, the first determining unit 1105 includes:

[0199] Determination module 11051, configured to periodically update a write threshold according to a preset time. The write threshold is determined by prior knowledge of data update operations within a preset time period and a preset sliding window.

[0200] The first judgment module 11052 judges whether the data to be updated is greater than the write threshold.

[0201] In the embodiment of the present application, the data loading unit 1101 includes:

[0202] A starting module 11011 is used to start the process and load the program of the process into the memory of the computer;

[0203] The second judgment module 11012 is configured to query the process cache for corresponding cache data according to the process identifier;

[0204] The loading module 11013 is configured to load the cache data into the memory of the process when the second judgment module finds corresponding cache data from the process cache according to the identifier of the process.

[0205] See also Figure 12 In this application, an embodiment of a database table updating device for upper-layer application reading data includes:

[0206] Receiving unit 1201, configured to receive a request to read target data;

[0207] A first acquiring unit 1202 is configured to read cache data corresponding to the target data from a process cache according to an identifier of the target data;

[0208] The second acquiring unit 1203 is configured to read the table data corresponding to the target data from the corresponding table of the system database according to the identifier of the target data;

[0209] The execution unit 1204 is configured to add the cache data to the table data to obtain target data.

[0210] See also Figure 13 Another embodiment of the database table updating device in the present application includes:

[0211] Processor 1301, memory 1302, input / output unit 1303, bus 1304;

[0212] The processor 1301 is connected to the memory 1302, the input and output unit 1303 and the bus 1304;

[0213] The processor 1301 specifically performs the following operations:

[0214] Receiving a call instruction for a data update operation in the process through a data update interface;

[0215] After executing the data update operation according to the calling instruction, the data to be updated is obtained;

[0216] Determining whether the data to be updated meets the conditions for being written into the system database;

[0217] If not, writing the data to be updated into the process cache;

[0218] If so, the data to be updated is written into the corresponding table of the system database.

[0219] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0220] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0221] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0222] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0223] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, and other media that can store program code.

Claims

1. A method for updating a multi-process shared database table, characterized in that: The method is applied to a process subsystem, wherein the process subsystem is communicatively connected to a system database, the system database is connected to at least one process subsystem, and at least one process is running on the process subsystem. The method includes: Loading data of the process when the process starts; monitoring whether any abnormal events occur during the execution of the process; Receiving a call instruction for a data update operation in the process through a data update interface; After executing the data update operation according to the calling instruction, the data to be updated is obtained; Determining whether the data to be updated meets the conditions for being written into the system database, and limiting the number of update operations written into the system database according to the conditions, so that more update operations occur on the process cache; If not, the data to be updated is written into the process cache, which is a distributed cache; If yes, then write the data to be updated into the corresponding table of the system database; Determining whether the data to be updated meets the conditions for being written into the system database includes: Update the write threshold regularly according to the preset time. The write threshold is determined by the prior knowledge of data update operations in the past preset time and the preset sliding window; Determine whether the data to be updated is greater than the write threshold; Loading data of the process when the process starts includes: Starting the process and loading the program of the process into the memory of the computer; querying from the process cache whether there is corresponding cache data according to the identifier of the process; If so, the cache data is loaded into the memory of the process.

2. The multi-process shared database table updating method according to claim 1, characterized in that: After monitoring whether an abnormal event occurs during the running of the process, the method further includes: If so, the data of the process is written into the process cache within a preset exit time.

3. The multi-process shared database table updating method according to any one of claims 1 to 2, characterized in that: After writing the data to be updated into the table corresponding to the system database, the method further includes: The data called by the calling instruction is cleared from the process cache.

4. A multi-process shared database table update device, characterized in that: include: A data loading unit, configured to load data of the process when the process is started; An abnormal event unit is used to monitor whether an abnormal event occurs during the operation of the process, and the abnormal event includes process abnormality and system abnormality; A first receiving unit, configured to receive a call instruction for a data update operation in the process through a data update interface; An updating unit, configured to obtain data to be updated after executing a data updating operation according to the calling instruction; a first determining unit, configured to determine whether the data to be updated meets a condition for being written into a system database, and to limit the number of update operations written into the system database according to the condition, so that more update operations occur on the process cache; a first execution unit, configured to write the data to be updated into a process cache, which is a distributed cache, when the first judgment unit determines that the data to be updated does not meet a write threshold for writing into the system database; a second execution unit, configured to write the data to be updated into a corresponding table of the system database when the first judgment unit determines that the data to be updated meets a write threshold for writing into the system database; The first judgment unit further includes: A determination module is used to regularly update the database write threshold according to a preset time. The database write threshold is determined by prior knowledge of data update operations within the past preset time and a preset sliding window; A first judgment module is configured to judge whether the data to be updated is greater than the write threshold; The data loading unit includes: A startup module, used to start the process and load the program of the process into the memory of the computer; A second judgment module is used to query whether there is corresponding cache data from the process cache according to the identifier of the process; The loading module is configured to load the cache data into the memory of the process when the second judgment module finds that there is corresponding cache data from the process cache according to the identifier of the process.

5. The multi-process shared database table updating device according to claim 4, characterized in that: The device further comprises: The third execution unit is configured to write the data of the process into the process cache when the abnormal event unit detects that an abnormal event is sent.

6. The multi-process shared database table updating device according to claim 5, characterized in that: The device further comprises: A clearing unit is used to clear the data called by the calling instruction in the process cache.

7. A database system, characterized in that: The system comprises a system database and at least one process subsystem, wherein the system database is in communication with the process subsystem, wherein the process subsystem comprises a data processing unit and a process cache unit, wherein the data processing unit comprises the database table updating device according to any one of claims 4 to 6, wherein: The system database is used to store data shared by the process subsystems; The process cache unit is used to cache and store data running on the process subsystem, and the process cache unit adopts a distributed cache; The data processing unit is used to process data updates and abnormal events of the processes running on the process subsystem.

Images