Data storage method and device
By switching data storage between the database and cache, and using temporary data identification lists to achieve data synchronization, the business unavailability problem caused by database and cache failures is solved, the system's high availability and throughput is improved, and the timeliness requirements are met.
Patent Information
- Application Number
- CN202310538943.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-12
AI Technical Summary
The data storage mode where the database and cache cooperate with each other will cause business services to be unavailable when any one fails, and the system throughput is not high, which cannot meet the timeliness requirements.
By detecting storage resource failures, writing data to another storage resource and synchronizing data from the temporary data identification list during failure recovery, ensuring consistency between the two storage resources, achieving high availability and throughput Improvement.
When any database and cache fail, business service availability can still be ensured, the overall throughput of the system can be improved, and the timeliness requirements can be met.
Smart Images

Figure CN118939626A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data storage. Background Art
[0002] At present, the mainstream middleware used in auction bidding on e-commerce platforms is database (e.g. MySQL) and cache (e.g. Redis). The bidding data needs to be written into these two storage resources at the same time, and it is strongly dependent on these two middlewares. The success of the bidding must be ensured when both middlewares are available. In the process of data writing, in order to ensure the security of data storage and the executability of business services, the data will be persistently stored in the database; in order to ensure fast data query, the data will be synchronously stored in the cache. In this way, the data storage mode of database and cache working together can realize the persistent storage and fast query of data.
[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:
[0004] The data storage mode in which the database and cache cooperate with each other. If any failure in the database or cache occurs, the business service will be unavailable. This data access mode is highly dependent on the database and cache, resulting in low overall system throughput and failure to meet timeliness requirements. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a method and device for data storage, which can ensure the availability of business services when any one of the database and the cache fails, thereby achieving high availability of auction bidding services; and there is no need to rely on both the database and the cache at the same time, so that the overall throughput of the system is greatly improved and timeliness requirements can be met.
[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for data storage is provided, comprising:
[0007] In response to detecting a failure of the first storage resource, writing data corresponding to the business operation into a second storage resource, adding the data identifier to a temporary data identifier list, and adding an anti-concurrency lock to the data;
[0008] In response to detecting that the first storage resource is restored, searching, according to a request instruction of the business operation to be processed, from the temporary data identifier list whether there is a data identifier of the target data corresponding to the business operation to be processed;
[0009] If the data identifier of the target data exists in the temporary data identifier list, acquiring the target data from the second storage resource and storing it in the first storage resource, and taking the data identifier of the target data out of the temporary data identifier list;
[0010] When the data identifier of the target data does not exist in the temporary data identifier list, the target data is synchronously stored in the first storage resource and the second storage resource.
[0011] Optionally, before searching the temporary data identifier list for whether there is a data identifier of the target data corresponding to the to-be-processed business operation, the method further includes: determining whether there is a value in the temporary data identifier list.
[0012] Optionally, the first storage resource is a database, the second storage resource is a cache, and the concurrency prevention lock is a distributed lock that depends on the cache.
[0013] Optionally, the first storage resource is a cache, the second storage resource is a database, and the anti-concurrency lock is an exclusive lock of the database; adding the data identifier to the temporary data identifier list includes: adding a temporary data tag to the data corresponding to the business operation in the second storage resource to add the data identifier to the temporary data identifier list.
[0014] Optionally, when the data identifier of the target data exists in the temporary data identifier list, the target data is obtained from the second storage resource and saved in the first storage resource, and the data identifier of the target data is taken out from the temporary data identifier list, including: when the data identifier of the target data exists in the temporary data identifier list, the target data is obtained from the data with a temporary data tag in the second storage resource, and the obtained target data is saved in the first storage resource; and the temporary data tag of the target data is removed.
[0015] Optionally, the method further includes: in response to detecting that the recovery time of the first storage resource reaches a set threshold, synchronizing the data corresponding to the data identifier in the temporary data identifier list to the first storage resource by starting a synchronization task.
[0016] According to another aspect of an embodiment of the present invention, there is provided a data storage device, including:
[0017] A first storage module, configured to, in response to detecting a failure of the first storage resource, write data corresponding to the business operation into a second storage resource, add a data identifier to a temporary data identifier list, and add an anti-concurrency lock to the data;
[0018] an identification search module, configured to, in response to detecting that the first storage resource is restored, search from the temporary data identification list whether there is a data identification of target data corresponding to the business operation to be processed according to a request instruction of the business operation to be processed;
[0019] A second storage module, configured to obtain the target data from the second storage resource and save it to the first storage resource if the data identifier of the target data exists in the temporary data identifier list, and to take the data identifier of the target data out of the temporary data identifier list;
[0020] A third storage module is configured to synchronously store the target data in the first storage resource and the second storage resource when the data identifier of the target data does not exist in the temporary data identifier list.
[0021] Optionally, the identifier search module is further used to: before searching the temporary data identifier list for whether there is a data identifier of the target data corresponding to the to-be-processed business operation, determine whether there is a value in the temporary data identifier list.
[0022] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method provided by an embodiment of the present invention.
[0023] According to another aspect of the embodiments of the present invention, a computer readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the data storage method provided by the embodiments of the present invention is implemented.
[0024] An embodiment of the above invention has the following advantages or beneficial effects: in response to detecting a failure of the first storage resource, the data corresponding to the business operation is written into the second storage resource, the data identifier is added to a temporary data identifier list, and an anti-concurrency lock is added to the data; in response to detecting that the first storage resource is recovered, according to the request instruction of the business operation to be processed, the temporary data identifier list is searched for the data identifier of the target data corresponding to the business operation to be processed; in the case that the data identifier of the target data exists in the temporary data identifier list, the target data is obtained from the second storage resource and saved in the first storage resource, and the data identifier of the target data is taken out from the temporary data identifier list; in the case that the data identifier of the target data does not exist in the temporary data identifier list, the technical solution of synchronously storing the target data in the first storage resource and the second storage resource can still ensure the availability of the business service when any one of the database and the cache fails, thereby achieving high availability of the auction bidding business; and there is no need to rely on the database and the cache at the same time, so that the overall throughput of the system is greatly improved, and the timeliness requirements can be met.
[0025] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific implementation examples. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
[0027] Figure 1 is a schematic diagram of main steps of a method for storing data according to an embodiment of the present invention;
[0028] Figure 2 is a schematic diagram of a data storage process according to an embodiment of the present invention;
[0029] Figure 3 is a schematic diagram of a data storage process according to another embodiment of the present invention;
[0030] Figure 4 is a schematic diagram of main modules of a data storage device according to an embodiment of the present invention;
[0031] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;
[0032] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0033] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0034] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution disclosed in the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.
[0035] In order to solve the technical problems existing in the prior art, the present invention provides a method for data storage, which can realize high-availability auction bidding and provide the design of mysql pipeline, redis pipeline and normal pipeline. When mysql has problems, the bidding can rely on redis, and when redis has problems, the bidding can rely on mysql to achieve real high availability. At the same time, when the switch switches back and forth between the mysql pipeline, the redis pipeline and the normal pipeline, technical solutions such as "transition period", "passive synchronization", "pipeline identification" and "synchronization task" are provided to achieve real high availability. Among them, mysql is a relational database management system, and the relational database stores data in different tables instead of putting all data in a large warehouse, which increases speed and improves flexibility. Redis (Remote Dictionary Server), i.e. remote dictionary service, is an open source log-type, Key-Value database written in ANSI C language, supports network, can be based on memory and can be persistent, and provides APIs in multiple languages.
[0036] Among them, the bid of the normal pipeline is the bid scheme under normal circumstances. When bidding at an auction, the bid is mainly divided into two steps: business verification and data writing, and it depends on the database (for example: based on MySQL) and the cache (for example: based on redis). After the business verification of the database and the cache is passed, the data will be written to the database and the cache. At this time, the data of multiple tables in the database MySQL (for example: real-time table, commodity table, record table, etc.) will be operated, and the database's X lock (Exclusive Lock, exclusive lock, also known as write lock, exclusive lock, is a basic lock type) and optimistic lock will be used to prevent concurrency problems. The bidding in this scenario is strongly dependent on the database MySQL and the cache redis. The unavailability of any one of them will cause problems. The following will be combined with the attached figure to specifically introduce how to bid at an auction when the database and cache fail.
[0037] Figure 1 FIG. 1 is a schematic diagram of the main steps of the method for data storage according to an embodiment of the present invention. Figure 1 As shown, the data storage method of the embodiment of the present invention mainly includes the following steps S101 to S104.
[0038] Step S101: In response to detecting a failure of the first storage resource, write data corresponding to the business operation into the second storage resource, add the data identifier to a temporary data identifier list, and add an anti-concurrency lock to the data. In an embodiment of the present invention, the first storage resource and the second storage resource refer to a database and a cache. Furthermore, the first storage resource can be a database or a cache; correspondingly, the second storage resource can be a cache or a database.
[0039] In an embodiment of the present invention, when it is detected that any storage resource fails or the storage resource recovers after a failure, a corresponding switch can be triggered to switch the data storage scheme of the bidding operation back and forth between MySQL pipeline bidding, Redis pipeline bidding and normal pipeline bidding, so that it can remain available in multiple situations.
[0040] Step S102: In response to detecting that the first storage resource has been recovered, according to the request instruction of the business operation to be processed, searching from the temporary data identification list whether there is a data identification of the target data corresponding to the business operation to be processed. When it is detected that the first storage resource that failed has been recovered, the synchronization of the data corresponding to the data identification in the temporary data identification list will be triggered according to the business operation. Specifically, first, it is necessary to search from the temporary data identification list whether there is a corresponding data identification according to the data identification of the target data corresponding to the business operation to be processed. If so, it means that during the period of the failure of the first storage resource, the target data has been processed according to the business operation; if not, it means that during the period of the failure of the first storage resource, the target data has not been processed according to the business operation.
[0041] Step S103: When the data identifier of the target data exists in the temporary data identifier list, the target data is obtained from the second storage resource and saved in the first storage resource, and the data identifier of the target data is taken out from the temporary data identifier list. This scenario is: during the failure of the first storage resource, the target data has been processed according to the business operation. At this time, the corresponding target data will be found from the second storage resource according to the data identifier of the target data, and then the target data will be synchronously stored in the first storage resource to achieve the purpose of data consistency in the first storage resource and the second storage resource. In addition, after data synchronization, the data identifier of the target data should also be taken out from the temporary data identifier list to avoid repeating the data synchronization operation when the business operation still corresponds to the target data, resulting in a waste of system resources and processing time, thereby ensuring the high availability of the system and improving data storage efficiency.
[0042] Step S104: If the data identifier of the target data does not exist in the temporary data identifier list, the target data is synchronously stored in the first storage resource and the second storage resource. This scenario is that during the failure of the first storage resource, the target data has not been processed according to the business operation. At this time, the target data can be directly synchronously stored in the database and cache according to the data storage process during normal bidding.
[0043] According to an embodiment of the present invention, before searching the temporary data identification list for data identification of the target data corresponding to the business operation to be processed, the method further includes: determining whether there is a value in the temporary data identification list. By confirming in advance that there is a value in the temporary data identification list, it is possible to avoid matching the temporary data identification list when there is no value in the temporary data identification list, thereby saving data matching time.
[0044] According to an embodiment of the present invention, the first storage resource is a database, the second storage resource is a cache, and the concurrency prevention lock is a distributed lock that depends on the cache.
[0045] According to another embodiment of the present invention, the method may further include: in response to detecting that the recovery time of the first storage resource reaches a set threshold, synchronizing the data corresponding to the data identifier in the temporary data identifier list to the first storage resource by starting a synchronization task. In this way, it can be ensured that the data in the two storage resources are eventually consistent.
[0046] Figure 2 FIG. 1 is a schematic diagram of a data storage process of an embodiment of the present invention, which shows the data storage process of the auction bid when a database failure occurs. Figure 2 As shown, after a database failure occurs, for the received business operations, first, the data corresponding to the business operations are written into the cache, and a redis distributed lock is added to the data; then the data identifier is added to the temporary data identifier list. Specifically, for example, a redis distributed lock with (SKU_LOCK+ID) as the key name key is set, where ID is the data identifier and SKU refers to the specific auction product identifier. By adding a redis distributed lock to the data, the distributed lock only relies on redis and the concurrency problem is solved. In this process, both the bid verification and the bid writing rely on redis. After the bid is successful, the data corresponding to the bidding business operation is written into redis, and the data identifier id of the data is written into the temporary data identifier list (for example: "redis pipeline id set") for subsequent "transition period" and "synchronization tasks".
[0047] After being processed by operation and maintenance staff, the MySQL database may have returned to normal. At this time, it is necessary to switch from "redis pipeline bidding" back to "normal pipeline bidding". First, the "transition period" processing is introduced, that is, after receiving the request instruction of the pending business operation, first determine whether there is a value in the temporary data identification list; if there is no value, directly enter the normal bidding, and save the target data synchronously to the database MySQL and cache redis; if there is a value, the data storage of the auction bidding business operation needs to go through the "transition period", and the auction bidding needs to compete for the distributed lock (SKU_LOCK+ID). This distributed lock is to prevent the "redis pipeline bidding" from still being executed when the switch is switched back to "normal pipeline bidding", so as to prevent problems caused by this concurrency. Then, determine whether there is a data identifier of the target data corresponding to the pending business operation in the temporary data identifier list. If so, perform "passive synchronization" and use the "passive synchronization" of the auction bidding business operation to refresh the data, that is, find the target data from the cache redis according to the data identifier of the target data, synchronize the target data to the database MySQL, and then remove the data identifier of the target data from the temporary data identifier list; if not, directly enter normal bidding and synchronize the target data to the database MySQL and cache redis.
[0048] In addition, for items that have no bidding behavior, it is necessary to start the "synchronization task" to perform data synchronization. Specifically, when the database MySQL has returned to normal and reached the preset threshold (for example: 3 minutes), there are still values in the temporary data identification list, which triggers the execution of the synchronization task. According to the data identifier in the temporary data identification list, the corresponding data is found from the cache redis, and the corresponding data is synchronized to the database MySQL, and then the data identifier is removed from the temporary data identification list until there is no value in the temporary data identification list. In this way, the data in the database MySQL and the cache redis can be consistent.
[0049] According to another embodiment of the present invention, the first storage resource is a cache, the second storage resource is a database, and the anti-concurrency lock is an exclusive lock of the database. And, adding the data identifier to the temporary data identifier list may specifically include: adding a temporary data tag to the data corresponding to the business operation in the second storage resource to add the data identifier to the temporary data identifier list. And, in the case where the data identifier of the target data exists in the temporary data identifier list, obtaining the target data from the second storage resource and saving it to the first storage resource, and taking the data identifier of the target data out of the temporary data identifier list may specifically include: in the case where the data identifier of the target data exists in the temporary data identifier list, obtaining the target data from the data with the temporary data tag in the second storage resource, and saving the obtained target data to the first storage resource; removing the temporary data tag of the target data.
[0050] According to another embodiment of the present invention, the method may further include: in response to detecting that the recovery time of the first storage resource reaches a set threshold, synchronizing the data corresponding to the data identifier in the temporary data identifier list to the first storage resource by starting a synchronization task. In this way, it can be ensured that the data in the two storage resources are eventually consistent.
[0051] Figure 3 FIG. 1 is a schematic diagram of a data storage process of another embodiment of the present invention, which shows the data storage process of the auction bid when the cache redis fails. Figure 3 As shown in the figure, after the cache fails, for the received business operation, first, the data corresponding to the business operation is written into the database, and the database X lock is added to the data; then a temporary data tag is added to the data corresponding to the business operation to add the data identifier to the temporary data identifier list. By adding the database X lock to the data, the concurrency problem is solved. In this process, the bid verification and bid writing both rely on the database MySQL to ensure the availability of the auction bid.
[0052] After being processed by operation and maintenance staff, cache redis may have returned to normal. At this time, it is necessary to switch from "mysql pipeline bidding" back to "normal pipeline bidding". First, the "transition period" processing is introduced, that is, after receiving the request instruction of the pending business operation, first determine whether there is a value in the temporary data identification list; if there is no value, directly enter the normal bidding, and synchronize and save the target data to the database mysql and cache redis; if there is a value, the data storage of the auction bidding business operation needs to go through the "transition period", and the auction bidding uses the database X lock to prevent problems caused by concurrency. Then determine whether there is a data identifier of the target data corresponding to the pending business operation in the temporary data identification list. If so, perform "passive synchronization" and use the "passive synchronization" of the auction bidding business operation to refresh the data, that is, find the target data from the database mysql according to the data identifier of the target data, synchronize the target data to the cache redis, and then remove the temporary data tag of the target data; if not, directly enter the normal bidding, and synchronize and save the target data to the database mysql and cache redis.
[0053] In addition, for items that have not been bid, it is also necessary to start the "synchronization task" to perform data synchronization. Specifically, when the cache redis has been restored to normal for a preset threshold (for example: 3 minutes), and there are still values in the temporary data identification list, the synchronization task is triggered, that is: according to the data identifier in the temporary data identification list, the corresponding data is found from the database MySQL, and the corresponding data is synchronized to the cache redis, and then the temporary data tag of the target data is removed to remove the data identifier from the temporary data identification list, until there is no value in the temporary data identification list (that is: all data in the database MySQL have no temporary data tags). In this way, the data in the database MySQL and the cache redis can be consistent.
[0054] exist Figure 2 and Figure 3 In the illustrated embodiment, the switch is switched smoothly, switching back and forth between "redis pipeline bidding", "mysql pipeline bidding" and "normal pipeline bidding", the bids remain available, and the data also meets the "eventual consistency" requirements of the "BASE" theory.
[0055] Figure 4 Schematic diagram of the main modules of the data storage device according to an embodiment of the present invention. Figure 4 As shown, the data storage device 400 of the embodiment of the present invention mainly includes: a first storage module 401, an identification search module 402, a second storage module 403 and a third storage module 404.
[0056] The first storage module 401 is used for writing data corresponding to the business operation into the second storage resource in response to detecting a failure of the first storage resource, adding the data identifier to a temporary data identifier list, and adding a concurrency prevention lock to the data;
[0057] An identifier search module 402 is used for searching, in response to detecting that the first storage resource is restored, whether there is a data identifier of the target data corresponding to the business operation to be processed from the temporary data identifier list according to a request instruction of the business operation to be processed;
[0058] The second storage module 403 is used to obtain the target data from the second storage resource and save it to the first storage resource if the data identifier of the target data exists in the temporary data identifier list, and to take the data identifier of the target data out of the temporary data identifier list;
[0059] The third storage module 404 is configured to synchronously store the target data in the first storage resource and the second storage resource when the data identifier of the target data does not exist in the temporary data identifier list.
[0060] According to an embodiment of the present invention, the identifier search module 402 may also be used to: determine whether there is a value in the temporary data identifier list before searching the temporary data identifier list for the data identifier of the target data corresponding to the to-be-processed business operation.
[0061] According to another embodiment of the present invention, the first storage resource is a database, the second storage resource is a cache, and the concurrency prevention lock is a distributed lock that depends on the cache.
[0062] According to another embodiment of the present invention, the first storage resource is a cache, the second storage resource is a database, and the anti-concurrency lock is an exclusive lock of the database; the first storage module 401 can also be used to: add a temporary data tag to the data corresponding to the business operation in the second storage resource to add the data identifier to the temporary data identifier list.
[0063] According to another embodiment of the present invention, the second storage module 403 can also be used for: when the data identifier of the target data exists in the temporary data identifier list, obtaining the target data from the data with the temporary data tag in the second storage resource, and saving the obtained target data in the first storage resource; removing the temporary data tag of the target data.
[0064] According to another embodiment of the present invention, the data storage device 400 may further include a fourth storage module (not shown in the figure), which is used to: in response to detecting that the recovery time of the first storage resource reaches a set threshold, synchronize the data corresponding to the data identifier in the temporary data identifier list to the first storage resource by starting a synchronization task.
[0065] According to the technical solution of the embodiment of the present invention, in response to detecting a failure of the first storage resource, the data corresponding to the business operation is written into the second storage resource, the data identifier is added to a temporary data identifier list, and an anti-concurrency lock is added to the data; in response to detecting that the first storage resource is recovered, according to the request instruction of the business operation to be processed, the temporary data identifier list is searched for the data identifier of the target data corresponding to the business operation to be processed; in the case that the data identifier of the target data exists in the temporary data identifier list, the target data is obtained from the second storage resource and saved to the first storage resource, and the data identifier of the target data is taken out from the temporary data identifier list; in the case that the data identifier of the target data does not exist in the temporary data identifier list, the target data is synchronously stored in the first storage resource and the second storage resource. This technical solution can ensure the availability of business services when any one of the database and the cache fails, thereby achieving high availability of the auction bidding business; and there is no need to rely on the database and the cache at the same time, so that the overall throughput of the system is greatly improved, and the timeliness requirements can be met.
[0066] Figure 5 An exemplary system architecture 500 is shown to which the data storage method or data storage apparatus according to the embodiment of the present invention may be applied.
[0067] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0068] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).
[0069] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0070] The server 505 may be a server providing various services, such as a backend management server (for example only) that supports shopping websites browsed by users using terminal devices 501, 502, and 503. The backend management server may respond to the received bidding request and other data by detecting the failure of the first storage resource, writing the data corresponding to the business operation into the second storage resource, adding the data identifier to the temporary data identifier list, and adding a concurrent lock to the data; in response to detecting the recovery of the first storage resource, searching from the temporary data identifier list whether there is a data identifier of the target data corresponding to the business operation to be processed according to the request instruction of the business operation to be processed; if the data identifier of the target data exists in the temporary data identifier list, obtaining the target data from the second storage resource and saving it to the first storage resource, taking out the data identifier of the target data from the temporary data identifier list; if the data identifier of the target data does not exist in the temporary data identifier list, synchronously storing the target data in the first storage resource and the second storage resource, etc., and feeding back the processing result (for example, data storage result, data synchronization result - for example only) to the terminal device.
[0071] It should be noted that the data storage method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data storage device is generally set in the server 505 .
[0072] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0073] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 6 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0074] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0075] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.
[0076] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.
[0077] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0078] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0079] The units or modules involved in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be arranged in a processor. For example, they may be described as: a processor includes a first storage module, an identification search module, a second storage module, and a third storage module. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves. For example, the first storage module may also be described as "a module for responding to detecting a failure of the first storage resource, writing data corresponding to the business operation into the second storage resource, adding the data identification to a temporary data identification list, and adding a module for preventing concurrent locks for the data".
[0080] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: in response to detecting a failure of the first storage resource, writing the data corresponding to the business operation into the second storage resource, adding the data identifier to the temporary data identifier list, and adding a concurrent prevention lock to the data; in response to detecting the recovery of the first storage resource, according to the request instruction of the business operation to be processed, searching from the temporary data identifier list whether there is a data identifier of the target data corresponding to the business operation to be processed; if the data identifier of the target data exists in the temporary data identifier list, obtaining the target data from the second storage resource and saving it to the first storage resource, taking the data identifier of the target data from the temporary data identifier list; if the data identifier of the target data does not exist in the temporary data identifier list, storing the target data synchronously in the first storage resource and the second storage resource.
[0081] According to the technical solution of the embodiment of the present invention, in response to detecting a failure of the first storage resource, the data corresponding to the business operation is written into the second storage resource, the data identifier is added to a temporary data identifier list, and an anti-concurrency lock is added to the data; in response to detecting that the first storage resource is recovered, according to the request instruction of the business operation to be processed, the temporary data identifier list is searched for the data identifier of the target data corresponding to the business operation to be processed; in the case that the data identifier of the target data exists in the temporary data identifier list, the target data is obtained from the second storage resource and saved to the first storage resource, and the data identifier of the target data is taken out from the temporary data identifier list; in the case that the data identifier of the target data does not exist in the temporary data identifier list, the target data is synchronously stored in the first storage resource and the second storage resource. This technical solution can ensure the availability of business services when any one of the database and the cache fails, thereby achieving high availability of the auction bidding business; and there is no need to rely on the database and the cache at the same time, so that the overall throughput of the system is greatly improved, and the timeliness requirements can be met.
[0082] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for data storage, characterized in that: include: In response to detecting a failure of the first storage resource, writing data corresponding to the business operation into a second storage resource, adding the data identifier to a temporary data identifier list, and adding an anti-concurrency lock to the data; In response to detecting that the first storage resource is restored, searching, according to a request instruction of the business operation to be processed, from the temporary data identifier list whether there is a data identifier of the target data corresponding to the business operation to be processed; If the data identifier of the target data exists in the temporary data identifier list, acquiring the target data from the second storage resource and storing it in the first storage resource, and taking the data identifier of the target data out of the temporary data identifier list; When the data identifier of the target data does not exist in the temporary data identifier list, the target data is synchronously stored in the first storage resource and the second storage resource.
2. The method according to claim 1, characterized in that Before searching the temporary data identifier list for a data identifier of the target data corresponding to the to-be-processed business operation, the method further includes: It is determined that there is a value in the temporary data identification list.
3. The method according to claim 1 or 2, characterized in that: The first storage resource is a database, the second storage resource is a cache, and the concurrency prevention lock is a distributed lock that depends on the cache.
4. The method according to claim 1 or 2, characterized in that: The first storage resource is a cache, the second storage resource is a database, and the concurrency prevention lock is an exclusive lock of the database; Add data identifiers to the temporary data identifier list, including: The data identifier is added to the temporary data identifier list by adding a temporary data tag to the data corresponding to the business operation in the second storage resource.
5. The method according to claim 4, characterized in that When the data identifier of the target data exists in the temporary data identifier list, acquiring the target data from the second storage resource and storing it in the first storage resource, and taking the data identifier of the target data from the temporary data identifier list comprises: If the data identifier of the target data exists in the temporary data identifier list, acquiring the target data from the data with the temporary data tag in the second storage resource, and saving the acquired target data to the first storage resource; The temporary data tag of the target data is removed.
6. The method according to claim 1, characterized in that The method further comprises: In response to detecting that the recovery time of the first storage resource reaches a set threshold, the data corresponding to the data identifier in the temporary data identifier list is synchronized to the first storage resource by starting a synchronization task.
7. A data storage device, characterized in that: include: A first storage module, configured to, in response to detecting a failure of the first storage resource, write data corresponding to the business operation into a second storage resource, add a data identifier to a temporary data identifier list, and add an anti-concurrency lock to the data; an identification search module, configured to, in response to detecting that the first storage resource is restored, search from the temporary data identification list whether there is a data identification of target data corresponding to the business operation to be processed according to a request instruction of the business operation to be processed; A second storage module, configured to obtain the target data from the second storage resource and save it to the first storage resource if the data identifier of the target data exists in the temporary data identifier list, and to take the data identifier of the target data out of the temporary data identifier list; A third storage module is configured to synchronously store the target data in the first storage resource and the second storage resource when the data identifier of the target data does not exist in the temporary data identifier list.
8. The device according to claim 7, characterized in that The identification search module is also used for: Before searching the temporary data identification list for whether there is a data identification of the target data corresponding to the to-be-processed business operation, it is determined that there is a value in the temporary data identification list.
9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.
10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Centralized bidding system data storage system and method
CN103605753A
Method and apparatus for data storage
CN109144992A
Data reading method and device
CN113760860A
Data storage method and apparatus
US20200285654A1
