Method and system for high-concurrency data storage

Abstract

The present invention discloses a method and system for storing data with high concurrency, the method comprising S10: obtaining a data operation object from a data operation object queue according to a data write request; S20: writing the data to be written into the cache and log file of the data operation object; S30: judging whether the cache size has exceeded the set object cache threshold; S40: if not, putting the data operation object back into the data operation object queue; S50: if yes, persisting the cache data and concurrently executing other steps. The present invention can store data to a storage file or a storage repository with high concurrency by concurrently running the persistent cache data and object cache process (other step processes except step S50), thereby increasing the number of service requests and the storage speed.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a method and system for high-concurrency data storage. Background Art

[0002] With the continuous development of the information age, the need to record data is increasing. However, the services that can generally provide data storage services can only provide a low number of requests and storage efficiency, which brings a lot of trouble to users who store data. The existing technology does not have a good solution to improve the number of requests and storage efficiency of data storage services. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method and system for storing data with high concurrency in view of at least one defect in the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is: constructing a method for storing data with high concurrency, comprising the following steps:

[0005] S10: according to the data write request, obtain a data operation object from the data operation object queue;

[0006] S20: writing the data to be written into the cache and log file of the data operation object;

[0007] S30: Determine whether the cache size exceeds the set object cache threshold;

[0008] S40: If not, putting the data operation object back into the data operation object queue;

[0009] S50: If yes, the cache data is persisted and other steps are executed concurrently.

[0010] Preferably, in the high-concurrency data storage method of the present invention, before step S10, the following steps are further included:

[0011] S01: receiving a write data request;

[0012] S02: Obtain a thread from the data writing thread pool to process the request.

[0013] Preferably, in the high-concurrency data storage method of the present invention, before step S10, the following steps are further included:

[0014] S03: Locking the data operation object queue to prohibit other threads from obtaining data operation objects from the data operation object queue;

[0015] After step S10, the following steps are also included:

[0016] S13: Unlock the data operation object queue.

[0017] Preferably, in the high-concurrency data storage method of the present invention, after step S10, the following steps are further included:

[0018] S11: Determine whether there is a data operation object in the data operation object queue;

[0019] If yes, execute step S13; if no, execute step S12;

[0020] S12: Generate a data operation object and add it to the data operation object queue.

[0021] Preferably, in the high-concurrency data storage method of the present invention, the persistent cache data includes:

[0022] S501: moving the data operation object into a data completion operation object list;

[0023] S502: Acquire cache data of a data operation object from the data completion operation object list;

[0024] S503: Cache the acquired cache data persistently, clear the cache and log files of the data operation object, and return to execute step S502.

[0025] Preferably, in the high-concurrency data storage method of the present invention, the method further comprises:

[0026] S60: When the data writing process is interrupted and then restarted, the object log file is read;

[0027] S70: Generate a data operation object, and write the log file data into the cache of the data operation object;

[0028] S80: Add the data operation object to the data operation object queue, and the startup is completed.

[0029] The present invention also constructs a high-concurrency data storage system, including:

[0030] The object acquisition module is used to obtain a data operation object from the data operation object queue according to a write data request;

[0031] A writing module, used for writing the data to be written into the cache and log file of the data operation object;

[0032] The cache judgment module is used to judge whether the cache size has exceeded the set object cache threshold. If not, the return module is executed; if so, the cache module is executed;

[0033] A return module, used for putting the data operation object back into the data operation object queue;

[0034] The cache module is used to persist cache data; wherein the cache module runs concurrently with other modules.

[0035] Preferably, in the high-concurrency data storage system of the present invention, the system further comprises:

[0036] A receiving module, used for receiving a write data request;

[0037] A thread acquisition module is used to acquire a thread from a data writing thread pool to process the request;

[0038] A locking module is used to lock the data operation object queue to prohibit other threads from obtaining data operation objects from the data operation object queue;

[0039] An unlocking module, used for unlocking the data operation object queue;

[0040] An object determination module, used for determining whether there is a data operation object in the data operation object queue, and if so, executing the unlocking module; if not, executing the first generating module;

[0041] The first generating module is used to generate a data operation object and add it to the data operation object queue.

[0042] Preferably, in the high-concurrency data storage system of the present invention, the cache module includes:

[0043] A moving module, used for moving the data operation object into a data completion operation object list;

[0044] A data acquisition module, used for acquiring cache data of a data operation object from the data completion operation object list;

[0045] The data cache module is used to cache the acquired cache data persistently and clear the cache and log files of the data operation objects.

[0046] Preferably, in the high-concurrency data storage system of the present invention, the system further comprises:

[0047] The log reading module is used to read the object log file when the data writing process is restarted after being interrupted;

[0048] The second generation module is used to generate a data operation object, write the log file data into the cache of the data operation object, add the data operation object to the data operation object queue, and start the completion.

[0049] By implementing the present invention, the following beneficial effects are achieved:

[0050] The present invention can store data in a storage file or a storage repository with high concurrency by concurrently running persistent cache data and object cache processes, thereby increasing the number of service requests and the storage speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0052] Figure 1 This is a schematic diagram of the method flow of high-concurrency data storage of the present invention Figure 1 ;

[0053] Figure 2 This is a schematic diagram of the method flow of high-concurrency data storage of the present invention Figure 2 ;

[0054] Figure 3 It is a flow chart of a method for restarting after interruption in a data writing process of the present invention;

[0055] Figure 4 This is a schematic diagram of the system module for high-concurrency data storage of the present invention. Figure 1 ;

[0056] Figure 5 This is a schematic diagram of the system module for high-concurrency data storage of the present invention. Figure 2 ;

[0057] Figure 6 It is a system module diagram of restarting after interruption of the data writing process of the present invention. DETAILED DESCRIPTION

[0058] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0059] It should be noted that the flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0060] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0061] like Figure 1As shown, the present invention discloses a method for storing data with high concurrency, comprising the following steps:

[0062] Step S10: according to the data write request, obtain a data operation object from the data operation object queue;

[0063] Step S20: writing the data to be written into the cache and log file of the data operation object;

[0064] Step S30: determining whether the cache size has exceeded a set object cache threshold;

[0065] Step S40: If not, the data operation object is put back into the data operation object queue, so that other threads can also write data into the cache and log file of the data operation object;

[0066] Step S50: If yes, the cache data is persisted and other steps are executed concurrently, that is, other steps except step S50.

[0067] Since the process of persisting cached data is relatively long, by running the persistent cache data and object cache processes (other step processes except step S50) concurrently, data can be stored in a storage file or a storage repository with high concurrency, thereby increasing the number of service requests and the storage speed, which can be several times higher than the general data storage method.

[0068] In some embodiments, Figure 2 As shown, before step S10, the following steps are also included:

[0069] Step S01: receiving a write data request;

[0070] Step S02: Obtain a thread processing request from the data writing thread pool. The number of threads in the data thread pool is variable and can be configured according to business needs and computer performance.

[0071] In some embodiments, before step S10, the method further includes:

[0072] Step S03: Lock the data operation object queue to prohibit other threads from obtaining data operation objects from the data operation object queue, thereby preventing multiple threads from obtaining the same object and causing confusion in operation data.

[0073] After step S10, the following steps are also included:

[0074] Step S13: Unlock the data operation object queue.

[0075] Among them, locking is to set a lock flag. When the flag is in the locked state, other threads cannot obtain data operation objects from the data operation object queue and can only wait. After the flag is unlocked, they can obtain data operation objects from the data operation object queue. Locking is to lock the process of obtaining data operation objects from the data operation object queue.

[0076] In some embodiments, in order to ensure that there are always data operation objects in the data operation object queue, step S10 further includes:

[0077] Step S11: determining whether there is a data operation object in the data operation object queue;

[0078] If yes, execute step S13: unlock the data operation object queue; if no, execute step S12;

[0079] Step S12: Generate a data operation object and add it to the data operation object queue.

[0080] In some embodiments, the persistent cache data in step S50 further includes:

[0081] Step S501: moving the data operation object into the data completion operation object list;

[0082] Step S502: acquiring cache data of a data operation object from the data completion operation object list;

[0083] Step S503: cache the acquired cache data persistently, for example, write the data in the computer memory to the computer's hard disk, clear the cache and log files of the data operation object, and return to execute step S502 to continue to obtain the cache data of the next data operation object from the data completion operation object list.

[0084] In some embodiments, Figure 3 As shown, in order to avoid the problem of data loss in the event of power failure or computer crash during data writing, the method further includes:

[0085] Step S60: when the data writing process is restarted after being interrupted, the object log file is read;

[0086] Step S70: Generate a data operation object, and write the log file data into the cache of the data operation object;

[0087] Step S80: Add the data operation object to the data operation object queue, and the startup is completed.

[0088] like Figure 4 As shown, the present invention also discloses a high-concurrency data storage system, including:

[0089] The object acquisition module is used to obtain a data operation object from the data operation object queue according to a write data request;

[0090] The writing module is used to write the data to be written into the cache and log file of the data operation object;

[0091] The cache judgment module is used to judge whether the cache size has exceeded the set object cache threshold. If not, the return module is executed; if so, the cache module is executed;

[0092] The return module is used to put the data operation object back into the data operation object queue, so that other threads can also write data into the cache and log file of the data operation object;

[0093] The cache module is used to persist cache data; wherein the cache module runs concurrently with other modules, that is, other modules except the cache module.

[0094] Since the process of persisting cached data is relatively long, by running the persistent cache data and object cache processes concurrently, data can be stored in a storage file or a repository with high concurrency, increasing the number of service requests and the storage speed, which can be several times higher than the general data storage method.

[0095] In some embodiments, Figure 5 As shown, the system also includes:

[0096] A receiving module, used for receiving a write data request;

[0097] The thread acquisition module is used to obtain a thread processing request from the data writing thread pool. The number of threads in the data thread pool is variable and can be configured according to business needs and computer performance.

[0098] In some embodiments, the system further comprises:

[0099] The locking module is used to lock the data operation object queue to prohibit other threads from obtaining data operation objects from the data operation object queue, thereby preventing multiple threads from obtaining the same object and causing confusion in the operation data;

[0100] The unlocking module is used to unlock the data operation object queue.

[0101] Among them, locking is to set a lock flag. When the flag is in the locked state, other threads cannot obtain data operation objects from the data operation object queue and can only wait. After the flag is unlocked, they can obtain data operation objects from the data operation object queue. Locking is to lock the process of obtaining data operation objects from the data operation object queue.

[0102] In some embodiments, in order to ensure that there are always data operation objects in the data operation object queue, the system further includes:

[0103] The object determination module is used to determine whether there is a data operation object in the data operation object queue, and if so, execute the unlocking module; if not, execute the first generating module;

[0104] The first generating module is used to generate a data operation object and add it to the data operation object queue.

[0105] In some embodiments, the cache module includes:

[0106] A move-in module is used to move the data operation object into the data completion operation object list;

[0107] A data acquisition module, used to acquire cache data of a data operation object from a list of data completion operation objects;

[0108] The data cache module is used to cache the acquired cache data persistently, such as writing the data in the computer memory to the computer's hard disk, and clearing the cache and log files of the data operation objects.

[0109] In some embodiments, Figure 6 As shown, in order to avoid the problem of data loss in the event of power failure or crash during data writing, the system also includes:

[0110] The log reading module is used to read the object log file when the data writing process is restarted after being interrupted;

[0111] The second generation module is used to generate a data operation object, write the log file data into the cache of the data operation object, add the data operation object to the data operation object queue, and start the completion.

[0112] By implementing the present invention, the following beneficial effects are achieved:

[0113] The present invention can store data in a storage file or a storage repository with high concurrency by concurrently running persistent cache data and object cache processes, thereby increasing the number of service requests and the storage speed.

[0114] It can be understood that the above embodiments only express the preferred implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the patent scope of the present invention. It should be pointed out that, for ordinary technicians in this field, the above technical features can be freely combined without departing from the concept of the present invention, and several deformations and improvements can be made, which all belong to the protection scope of the present invention. Therefore, all equivalent changes and modifications made to the scope of the claims of the present invention should belong to the coverage of the claims of the present invention.

Claims

1. A method for storing data with high concurrency, characterized in that: The following steps are involved: S01: receiving a write data request; S02: Obtain a thread from the data writing thread pool to process the request; S03: Locking the data operation object queue to prohibit other threads from obtaining data operation objects from the data operation object queue; S10: acquiring one of the data operation objects from the data operation object queue according to the data write request; S13: unlocking the data operation object queue so that other threads can obtain the data operation object from the data operation object queue; S20: writing the data to be written into the cache and log file of the data operation object; S30: Determine whether the cache size exceeds the set object cache threshold; S40: If not, putting the data operation object back into the data operation object queue, so that other threads can write data into the cache and log file of the data operation object; S50: If yes, the cache data is persisted and other steps except step S50 are executed concurrently.

2. The method for storing data with high concurrency according to claim 1, characterized in that: After step S10, the following steps are also included: S11: Determine whether there is a data operation object in the data operation object queue; If yes, execute step S13; if no, execute step S12; S12: Generate a data operation object and add it to the data operation object queue.

3. The method for storing data with high concurrency according to claim 1, characterized in that: The persistent cache data includes: S501: moving the data operation object into a data completion operation object list; S502: Acquire cache data of a data operation object from the data completion operation object list; S503: Cache the acquired cache data persistently, clear the cache and log files of the data operation object, and return to execute step S502.

4. The method for storing data with high concurrency according to claim 1, characterized in that: The method further comprises: S60: When the data writing process is interrupted and then restarted, the object log file is read; S70: Generate a data operation object, and write the log file data into the cache of the data operation object; S80: Add the data operation object to the data operation object queue, and the startup is completed.

5. A system for storing data with high concurrency, characterized in that: include: A receiving module, used for receiving a write data request; A thread acquisition module is used to acquire a thread from a data writing thread pool to process the request; A locking module, used to lock the data operation object queue to prohibit other threads from obtaining data operation objects from the data operation object queue; An object acquisition module, used for acquiring one of the data operation objects from the data operation object queue according to a write data request; An unlocking module, used for unlocking the data operation object queue so that other threads can obtain the data operation object from the data operation object queue; A writing module, used for writing the data to be written into the cache and log file of the data operation object; The cache judgment module is used to judge whether the cache size has exceeded the set object cache threshold. If not, the return module is executed; If so, the cache module is executed; A return module, used to put the data operation object back into the data operation object queue, so that other threads can write data into the cache and log file of the data operation object; A cache module is used to persist cache data; wherein the cache module runs concurrently with other modules except the cache module.

6. The high-concurrency data storage system according to claim 5, characterized in that: The system further comprises: An object determination module, used for determining whether there is a data operation object in the data operation object queue, and if so, executing the unlocking module; if not, executing the first generating module; The first generating module is used to generate a data operation object and add it to the data operation object queue.

7. The high-concurrency data storage system according to claim 5, characterized in that: The cache module comprises: A moving module, used for moving the data operation object into a data completion operation object list; A data acquisition module, used for acquiring cache data of a data operation object from the data completion operation object list; The data cache module is used to cache the acquired cache data persistently and clear the cache and log files of the data operation objects.

8. The high-concurrency data storage system according to claim 5, characterized in that: The system further comprises: The log reading module is used to read the object log file when the data writing process is restarted after being interrupted; The second generation module is used to generate a data operation object and write the log file data into the data In the cache of the operation object, the data operation object is added to the data operation object queue, Startup complete.

Images