Data storage method, device, electronic device and computer readable medium

Abstract

The embodiments of the present application provide a data storage method, device, electronic device and computer-readable medium, which relate to the field of data storage. The method includes: when it is determined that the access data of the received write request exists in the memory, executing the operation of the write request on the access data; recording the operation of the write request on the access data in an incremental log file; applying the incremental log file to the storage engine of the external storage device, so that the storage engine, based on the incremental log file, persistently stores the data after the write request operates on the access data in the external storage device. Through the embodiments of the present application, the storage capacity of data is greatly expanded, and the storage cost of data is greatly reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data storage, and more particularly to a data storage method, device, electronic device, and computer-readable medium. Background Art

[0002] Redis is an open source, network-supported, memory-based, key-value storage database written in ANSI C. It also supports data persistence and rich data structures. Alibaba Cloud provides a memory storage service based on the open source Redis. However, this memory storage service requires that all data be stored in memory. In a computer system, the storage capacity of memory is relatively limited, ranging from tens of GB to hundreds of GB, and the cost of memory is very high, usually tens to hundreds of times that of external memory. The main advantage of memory is high access performance, and the read and write performance is about hundreds to thousands of times that of external memory. It can be seen that how to solve the problem of limited data storage capacity and high data storage cost caused by using memory to store data in the existing technology has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the present application is to propose a data storage method, device, electronic device and computer-readable medium, which are used to solve the technical problems existing in the prior art of limited data storage capacity and high data storage cost caused by using memory to store data.

[0004] According to the first aspect of the embodiment of the present application, a data storage method is provided. The method includes: when it is determined that the access data of the received write request exists in the memory, executing the operation of the write request on the access data; recording the operation of the write request on the access data in an incremental log file; applying the incremental log file to the storage engine of the external storage device, so that the storage engine persistently stores the data after the write request operates on the access data in the external storage device based on the incremental log file.

[0005] According to the second aspect of the embodiment of the present application, a data storage device is provided. The device includes: a first execution module, which is used to execute the operation of the write request on the access data when it is determined that the access data of the received write request exists in the memory; a recording module, which is used to record the operation of the write request on the access data in an incremental log file; and an application module, which is used to apply the incremental log file to the storage engine of the external storage device, so that the storage engine, based on the incremental log file, persistently stores the data after the write request operates on the access data in the external storage device.

[0006] According to the third aspect of the embodiment of the present application, a data storage system is provided. The system includes: a database based on memory, and a storage engine based on an external storage device that interacts with the database, the database is used to perform the operation of the write request on the access data when it is determined that the access data of the received write request exists in the memory, and record the operation of the write request on the access data in an incremental log file, and apply the incremental log file to the storage engine; the storage engine is used to persistently store the data after the write request operates on the access data in the external storage device based on the incremental log file.

[0007] According to a fourth aspect of an embodiment of the present application, there is provided an electronic device, comprising: one or more processors; and a computer-readable medium configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method as described in the first aspect of the above embodiment.

[0008] According to a fifth aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the data storage method as described in the first aspect of the above embodiment is implemented.

[0009] According to the technical solution provided in the embodiment of the present application, when it is determined that the access data of the received write request exists in the memory, the operation of the write request on the access data is executed, and the operation of the write request on the access data is recorded in an incremental log file, and then the incremental log file is applied to the storage engine of the external storage device, so that the storage engine, based on the incremental log file, persistently stores the data after the write request operation on the access data in the external storage device. Compared with other existing methods, by applying the incremental log file recording the operation of the write request on the access data to the storage engine of the external storage device, the storage engine, based on the incremental log file, persistently stores the data after the write request operation on the access data in the external storage device, which greatly expands the data storage capacity and significantly reduces the data storage cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0011] Figure 1 This is a flowchart of the steps of the data storage method in Embodiment 1 of the present application;

[0012] Figure 2A This is a flowchart of the steps of the data storage method in Embodiment 2 of the present application;

[0013] Figure 2B A schematic diagram of a mapping relationship between a data structure of a memory-based database and a data structure of a storage engine provided according to Embodiment 2 of the present application;

[0014] Figure 3A This is a schematic diagram of the structure of the data storage system in the third embodiment of the present application;

[0015] Figure 3B This is a schematic diagram of the architecture of a data storage system provided according to Embodiment 3 of the present application;

[0016] Figure 4 This is a schematic diagram of the structure of the data storage device in the fourth embodiment of the present application;

[0017] Figure 5 This is a schematic diagram of the structure of the data storage device in the fifth embodiment of the present application;

[0018] Figure 6 This is a schematic diagram of the structure of an electronic device in Embodiment 6 of the present application;

[0019] Figure 7 This is the hardware structure of the electronic device in Example 7 of the present application. DETAILED DESCRIPTION

[0020] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only configured to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0021] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] Reference Figure 1 , showing a step flow chart of the data storage method of embodiment 1 of the present application.

[0023] Specifically, the data storage method of this embodiment includes the following steps:

[0024] In step S101, when it is determined that the access data of the received write request exists in the memory, an operation of the write request on the access data is executed.

[0025] In this embodiment, the write request can be understood as a request to change the access data. Any request for access data operation that causes a change in access data can be called a write request of this embodiment. Specifically, the write request may include: data creation request, data deletion request, data modification request, data update request, data addition request, etc. When the database based on the memory is a redis database, the access data of the write request may be a key-value pair. The operation of the write request on the access data may be a creation operation, a deletion operation, a modification operation, an update operation, an addition operation, etc. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0026] In a specific example, when a write request is received from a user through a client of a terminal device, a parsing operation is performed on the write request based on the redis database in the memory to determine the access data of the write request. Then, it is determined whether the access data of the write request exists in the memory. When it is determined that the access data of the write request exists in the memory, the operation of the write request on the access data is executed. In this embodiment, the memory only stores hot data, that is, data that is frequently accessed by users, and the external storage device stores the full amount of data including hot data and cold data. Therefore, before executing the operation of the write request on the access data, it is necessary to determine in advance whether the access data of the write request exists in the memory. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0027] In step S102, the operation of the write request on the access data is recorded in an incremental log file.

[0028] In this embodiment, the operation of the write request for accessing the data is written to an incremental log file in the form of a log. Specifically, when the database based on the memory is a redis database, the Redis command corresponding to the received write request is written to an AOF (Append Only File) file in the form of a log. Among them, the incremental log file can be understood as a log file for recording the operation of the write request for accessing the data in an incremental form. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0029] In step S103, the incremental log file is applied to a storage engine of an external storage device, so that the storage engine persistently stores the data after the write request is operated on the access data in the external storage device based on the incremental log file.

[0030] In this embodiment, the external storage device may be a solid state drive or a mechanical hard disk. The performance of a solid state drive is significantly better than that of a mechanical hard disk, but much lower than that of a memory. The performance of a mechanical hard disk is greatly limited by the seek time of a mechanical head arm. The storage engine is used to store data persistently in an external storage device. The storage engine may be an embedded storage engine, such as Rocks DB. It is to be understood that the above description is merely exemplary and the embodiments of the present application do not impose any limitation thereto.

[0031] In a specific example, when the memory-based database is a redis database, an independent thread is used to asynchronously apply the incremental log file to the storage engine Rocks DB of the external storage device, and the storage engine Rocks DB persistently stores the key-value pair data after the write request for the access data operation in the external storage device based on the incremental log file. It can be understood that the above description is only exemplary and the embodiments of the present application do not impose any limitation on this.

[0032] Through the data storage method provided by the embodiment of the present application, when it is determined that the access data of the received write request exists in the memory, the operation of the write request on the access data is executed, and the operation of the write request on the access data is recorded in an incremental log file, and then the incremental log file is applied to the storage engine of the external storage device, so that the storage engine, based on the incremental log file, persistently stores the data after the write request operation on the access data in the external storage device. Compared with other existing methods, by applying the incremental log file recording the operation of the write request on the access data to the storage engine of the external storage device, the storage engine, based on the incremental log file, persistently stores the data after the write request operation on the access data in the external storage device, which greatly expands the data storage capacity and significantly reduces the data storage cost.

[0033] The data storage method of this embodiment can be executed by any appropriate device with data processing capabilities, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, vehicle-mounted devices, entertainment equipment, advertising equipment, personal digital assistants (PDAs), tablet computers, laptop computers, handheld game consoles, smart glasses, smart watches, wearable devices, virtual display devices or display enhancement devices (such as Google Glass, Oculus Rift, Hololens, Gear VR), etc.

[0034] Reference Figure 2A , showing a step flow chart of the data storage method of embodiment 2 of the present application.

[0035] Specifically, the data storage method of this embodiment includes the following steps:

[0036] In step S201, when it is determined that the access data does not exist in the memory, the access data read from the external storage device is replaced in the memory.

[0037] In this embodiment, the memory only stores hot data, that is, data that is frequently accessed by users, and the external storage device stores the full amount of data including hot data and cold data. Therefore, when it is determined that the access data does not exist in the memory, the access data is read from the external storage device, and the access data read from the external storage device is replaced in the memory. It can be understood that the above description is only exemplary, and the embodiment of the present application does not impose any limitation on this.

[0038] In some optional embodiments, before replacing the access data read from the external storage device into the internal memory, the method further includes: if it is determined that the access frequency of the access data exceeds a preset access frequency threshold, or the idle time of the access data does not exceed a preset time threshold, then the step of replacing the access data read from the external storage device into the internal memory is performed. Specifically, when it is determined that the access frequency of the access data exceeds the preset access frequency threshold, or the idle time of the access data does not exceed the preset time threshold, it can be determined that the access data of the write request is hot data, and therefore, the access data read from the external storage device can be replaced into the internal memory to facilitate the write request to perform related operations on the access data. Among them, the preset access frequency threshold and the preset time threshold can be set by those skilled in the art according to actual needs, and the embodiments of the present application do not impose any restrictions on this. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any restrictions on this.

[0039] In some optional embodiments, the method further includes: if it is determined that the access frequency of the access data does not exceed the preset access frequency threshold, and the idle time of the access data exceeds the preset time threshold, then the access data is read from the external storage device, and the step of executing the operation of the write request on the access data is performed. Specifically, when it is determined that the access frequency of the access data does not exceed the preset access frequency threshold, and the idle time of the access data exceeds the preset time threshold, it can be determined that the access data of the write request is cold data. Therefore, it is not necessary to replace the access data read from the external storage device into the memory, but directly read the access data from the external storage device, and execute the operation of the write request on the access data. Among them, the preset access frequency threshold and the preset time threshold can be set by those skilled in the art according to actual needs, and the embodiments of the present application do not make any restrictions on this. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any restrictions on this.

[0040] In some optional embodiments, when the access data read from the external storage device is replaced into the internal memory, the access data is read from the external storage device based on the mapping relationship between the data structure of the database in the internal memory and the data structure of the storage engine; and the access data read from the external storage device is replaced into the internal memory. Thus, by means of the mapping relationship between the data structure of the database in the internal memory and the data structure of the storage engine, fine-grained access to the data stored in the external storage device can be achieved, thereby greatly reducing the performance jitter caused by reading the access data from the external storage device. It will be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0041] In a specific example, before reading the access data from the external storage device, the method further includes: splitting the data belonging to the data structure of the database to obtain multiple key-value pairs corresponding to the data; establishing a mapping relationship between the multiple key-value pairs corresponding to the data and the key-value pairs of the storage engine, respectively, to obtain a mapping relationship between the data structure of the database and the data structure of the storage engine. Thus, by obtaining the mapping relationship between the data structure of the database and the data structure of the storage engine, fine-grained access to the data stored in the external storage device can be achieved, thereby greatly reducing the performance jitter caused by reading data from the external storage device. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0042] In a specific example, the database may be a redis database based on the memory, and the storage engine may be Rocks DB. When data in a complex data structure (non-String type data structures supported by the Redis database, such as Hash, List, Sorted Set, Set, Geo, etc.) in a redis database is persistently stored in an external storage device, the data belonging to the complex data structure may be split to obtain multiple key-value pairs corresponding to the data, and the multiple key-value pairs corresponding to the data are stored, supporting fine-grained access to the data stored in the external storage device. Figure 2B As shown, a data belonging to a Hash structure can be split into 2 fields; a data belonging to a Set structure can be split into 2 Members; a data belonging to a Zset structure can be split into 2 Scores. If the data belonging to a complex data structure is not split, then when the data read from the external storage device is replaced into the memory, the entire data belonging to the complex data structure needs to be replaced into the memory, which will cause performance jitter. If the data belonging to a complex data structure is split, then when the data read from the external storage device is replaced into the memory, a certain key-value pair of the data needs to be replaced into the memory, and there is no need to replace the entire data belonging to the complex data structure into the memory, thereby making the data access performance of the external storage device smoother. It will be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0043] In step S202, an operation of the write request on the access data is executed.

[0044] Since step S202 is similar to the above step S101, it will not be described again here.

[0045] In step S203, the operation of the write request on the access data is recorded in an incremental log file.

[0046] Since step S203 is similar to the above step S102, it will not be described again here.

[0047] In step S204, the incremental log file is applied to a storage engine of an external storage device, so that the storage engine persistently stores the data after the write request is operated on the access data in the external storage device based on the incremental log file.

[0048] Since step S204 is similar to the above step S103, it will not be described again here.

[0049] In some optional embodiments, the method further includes: when the storage space of the memory is full of data, or the storage space of the memory is cleaned up regularly, determining the access frequency or idle time of the data stored in the storage space of the memory; based on the access frequency or the idle time, cleaning up the data stored in the storage space of the memory to release the storage space of the memory. Thereby, through the access frequency or idle time of the data stored in the storage space of the memory, hot data frequently accessed by users can be selected and stored in the memory, so that the hot data frequently accessed by users has higher access performance (read and write performance), thereby ensuring the access performance of most read and write requests. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0050] In a specific example, regularly cleaning up the storage space of the memory can be understood as cleaning up the storage space of the memory every preset time period. When determining the access frequency or idle time of the data stored in the storage space of the memory, the access frequency or idle time of the data stored in the storage space of the memory can be obtained in real time through the redis database based on the memory. When cleaning up the data stored in the storage space of the memory based on the access frequency or the idle time, the cleaning operation can be performed on the data whose access frequency does not exceed the pre-configured frequency threshold to release the storage space of the memory; or, the cleaning operation can be performed on the data whose idle time exceeds the pre-configured time threshold to release the storage space of the memory. Among them, the pre-configured frequency threshold and the pre-configured time threshold can be set by those skilled in the art according to actual needs, and the embodiment of the present application does not make any limitation on this. The idle time can be understood as the length of time that the data stored in the memory is not accessed. It can be understood that the above description is only exemplary, and the embodiment of the present application does not make any limitation on this.

[0051] In a specific example, when a read request is received from a user through a client of a terminal device, a parsing operation is performed on the read request based on the redis database of the memory to determine the access data of the read request. Then, it is determined whether the access data of the read request exists in the memory. When it is determined that the access data of the read request exists in the memory, the read operation of the read request for the access data is performed. When it is determined that the access data of the read request does not exist in the memory, the access data read from the external storage device is replaced in the memory, and the read operation of the read request for the access data is performed. Optionally, when it is determined that the access data of the read request does not exist in the memory, the access data can be directly read from the external storage device according to the read request. Among them, the specific implementation method of replacing the access data read from the external storage device into the memory is similar to the specific implementation method of replacing the access data read from the external storage device into the memory mentioned above, and the specific implementation method of reading the access data from the external storage device is similar to the specific implementation method of reading the access data from the external storage device mentioned above, which will not be repeated here. In this embodiment, the memory stores only hot data, that is, data that is frequently accessed by users, and the external storage device stores the full amount of data including hot data and cold data. Therefore, before executing the read operation of the read request for the access data, it is necessary to determine in advance whether the access data of the read request exists in the memory. It can be understood that the above description is only exemplary and the embodiment of the present application does not impose any limitation on this.

[0052] Through the data storage method provided in the embodiment of the present application, when it is determined that the access data does not exist in the memory, the access data read from the external storage device is replaced into the memory, and the operation of the write request on the access data is executed, and then the operation of the write request on the access data is recorded in an incremental log file, and then the incremental log file is applied to the storage engine of the external storage device, so that the storage engine, based on the incremental log file, persistently stores the data after the write request operation on the access data in the external storage device. Compared with other existing methods, by applying the incremental log file recording the operation of the write request on the access data to the storage engine of the external storage device, the storage engine, based on the incremental log file, persistently stores the data after the write request operation on the access data in the external storage device, which greatly expands the data storage capacity and significantly reduces the data storage cost.

[0053] The data storage method of this embodiment can be executed by any appropriate device with data processing capabilities, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, vehicle-mounted devices, entertainment equipment, advertising equipment, personal digital assistants (PDAs), tablet computers, laptop computers, handheld game consoles, smart glasses, smart watches, wearable devices, virtual display devices or display enhancement devices (such as Google Glass, Oculus Rift, Hololens, Gear VR), etc.

[0054] Reference Figure 3A , showing a schematic diagram of the structure of the data storage system in Example 3 of the present application.

[0055] The data storage system provided in this embodiment includes: a memory-based database 301, and a storage engine 302 based on an external storage device that interacts with the database 301, wherein the database 301 is used to execute the operation of the write request on the access data when it is determined that the access data of the received write request exists in the memory, and record the operation of the write request on the access data in an incremental log file, and apply the incremental log file to the storage engine; the storage engine 302 is used to persistently store the data after the write request operates on the access data in the external storage device based on the incremental log file. Among them, the memory-based database 301 can be a Redis database, and the storage engine 302 based on an external storage device can be a Rocks DB. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0056] In a specific example, Figure 3BAs shown, the topmost Redis protocol layer is implemented based on the open source Redis, and its main purpose is command compatibility processing, including network interaction, parsing and execution of requests, etc. The middle memory stores hot data, and the exchange module is responsible for swapping in and out of the data in the memory. The bottommost incremental log generation module is responsible for the asynchronous application of incremental logs, and the storage engine based on the solid-state drive is responsible for the persistent storage of the full amount of data, and the storage engine is implemented based on the open source software RockDB. Among them, the full amount of data that the storage engine is responsible for persistent storage includes not only hot data and cold data, but also incremental log files applied to the storage engine. In addition, the data storage system can be described in detail as follows in combination with the data storage method provided in the above embodiment: 1. The full amount of data is stored in an external storage device. When the user's write request is sent to the Redis protocol layer, the write request will be parsed and executed. After the execution of the write request is completed, the incremental log generation module writes the Redis command corresponding to the write request to the AOF file in the form of a log, and uses an independent thread to asynchronously apply the incremental log to the storage engine Rocks DB. The storage engine based on the solid-state drive is implemented based on the open source software RocksDB. Based on the incremental log, the data after the write request is operated on the access data is persistently stored in the external storage device. 2. Hot data is stored in the memory. When the memory space is full of data, the exchange module is responsible for selecting the least frequently accessed or least frequently accessed data, eliminating it from the memory, and freeing up memory space. When the data to be accessed by the write request is not in the memory, the exchange module is responsible for asynchronously swapping the access data from the storage engine RocksDB into the memory. After the swap is completed, the write request is re-executed. 3. Smooth data access realizes the mapping of all data structures of the Redis database to the key-value pairs of the Rocks DB storage engine, such as Figure 2B As shown. When the data to be accessed by the write request is not in the memory, and the access frequency of the data does not reach the threshold, or the idle time exceeds the threshold, the Redis protocol layer will directly read the data from the storage engine based on the solid-state drive without swapping the access data into the memory. It can be understood that the above description is only exemplary and the embodiments of the present application do not impose any limitation on this.

[0057] The data storage system of this embodiment is used to implement the corresponding data storage methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0058] Reference Figure 4 , showing a schematic diagram of the structure of the data storage device in the fourth embodiment of the present application.

[0059] The data storage device of this embodiment includes: a first execution module 401, which is used to execute the operation of the write request on the access data when it is determined that the access data of the received write request exists in the memory; a recording module 402, which is used to record the operation of the write request on the access data in an incremental log file; and an application module 403, which is used to apply the incremental log file to the storage engine of the external storage device, so that the storage engine persistently stores the data after the write request operates on the access data in the external storage device based on the incremental log file.

[0060] The data storage device of this embodiment is used to implement the corresponding data storage methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0061] Reference Figure 5 , shows a schematic diagram of the structure of the data storage device in Example 5 of the present application.

[0062] The data storage device of this embodiment includes: a first execution module 503, which is used to execute the operation of the write request on the access data when it is determined that the access data of the received write request exists in the memory; a recording module 504, which is used to record the operation of the write request on the access data in an incremental log file; and an application module 505, which is used to apply the incremental log file to the storage engine of the external storage device, so that the storage engine persistently stores the data after the operation of the write request on the access data in the external storage device based on the incremental log file.

[0063] Optionally, the apparatus further includes: a second execution module 502, for replacing the access data read from the external storage device into the memory when it is determined that the access data does not exist in the memory, and executing the step of executing the write request operation on the access data.

[0064] Optionally, before the second execution module 502, the device also includes: a third execution module 501, which is used to execute the step of replacing the access data read from the external storage device into the memory if it is determined that the access frequency of the access data exceeds a preset access frequency threshold, or the idle time of the access data does not exceed a preset time threshold.

[0065] Optionally, the device also includes: a fourth execution device 508, which is used to read the access data from the external storage device and execute the step of executing the write request for the access data if it is determined that the access frequency of the access data does not exceed a preset access frequency threshold and the idle time of the access data exceeds a preset time threshold.

[0066] Optionally, the first execution module 503 includes: a reading sub-module 5033, which is used to read the access data from the external storage device based on the mapping relationship between the data structure of the database in the memory and the data structure of the storage engine; and a replacement sub-module 5034, which is used to replace the access data read from the external storage device into the memory.

[0067] Optionally, before the reading submodule 5033, the first execution module 503 also includes: a splitting submodule 5031, used to split the data belonging to the data structure of the database to obtain multiple key-value pairs corresponding to the data; an establishing submodule 5032, used to establish a mapping relationship between the multiple key-value pairs corresponding to the data and the key-value pairs of the storage engine, so as to obtain a mapping relationship between the data structure of the database and the data structure of the storage engine.

[0068] Optionally, the device also includes: a determination module 506, which is used to determine the access frequency or idle time of the data stored in the storage space of the memory when the storage space of the memory is full of data or the storage space of the memory is cleaned up regularly; and a cleaning module 507, which is used to clean up the data stored in the storage space of the memory based on the access frequency or the idle time to release the storage space of the memory.

[0069] The data storage device of this embodiment is used to implement the corresponding data storage methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0070] Figure 6 This is a schematic diagram of the structure of an electronic device in Embodiment 6 of the present application; the electronic device may include:

[0071] One or more processors 601;

[0072] The computer readable medium 602 may be configured to store one or more programs,

[0073] When the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method as described in the first or second embodiment above.

[0074] Figure 7 The hardware structure of the electronic device in the seventh embodiment of the present application; Figure 7 As shown, the hardware structure of the electronic device may include: a processor 701, a communication interface 702, a computer readable medium 703 and a communication bus 704;

[0075] The processor 701, the communication interface 702, and the computer readable medium 703 communicate with each other via a communication bus 704;

[0076] Optionally, the communication interface 702 may be an interface of a communication module, such as an interface of a GSM module;

[0077] Among them, the processor 701 can be specifically configured as: when it is determined that the access data of the received write request exists in the memory, execute the operation of the write request on the access data; record the operation of the write request on the access data in an incremental log file; apply the incremental log file to the storage engine of the external storage device, so that the storage engine, based on the incremental log file, persistently stores the data after the write request operates on the access data in the external storage device.

[0078] The processor 701 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0079] The computer readable medium 703 can be, but is not limited to, a random access memory medium (RAM), a read-only memory medium (ROM), a programmable read-only memory medium (PROM), an erasable programmable read-only memory medium (EPROM), an electrically erasable read-only memory medium (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0080] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code configured to execute the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program configured to be used by or in combination with an instruction execution system, an apparatus or a device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0081] Computer program code configured to perform the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0082] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions configured to implement the specified logical function. There are specific sequential relationships in the above-mentioned specific embodiments, but these sequential relationships are only exemplary. When the specific implementation is implemented, these steps may be less, more, or the execution order may be adjusted. That is, in some implementations as replacements, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0083] The modules involved in the embodiments of the present application may be implemented in software or hardware. The modules described may also be set in a processor, for example, may be described as: a processor includes a first execution module, a recording module and an application module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the first execution module may also be described as "a module that executes the operation of the write request on the access data when it is determined that the access data of the received write request exists in the memory".

[0084] As another aspect, the present application further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the data storage method described in the first or second embodiment above is implemented.

[0085] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the device: when it is determined that the access data of the received write request exists in the memory, executes the operation of the write request on the access data; records the operation of the write request on the access data in an incremental log file; applies the incremental log file to the storage engine of the external storage device, so that the storage engine, based on the incremental log file, persistently stores the data after the write request operates on the access data in the external storage device.

[0086] The expressions "first", "second", "the first" or "the second" used in various embodiments of the present disclosure may modify various components regardless of order and / or importance, but these expressions do not limit the corresponding components. The above expressions are only configured for the purpose of distinguishing an element from other elements. For example, a first user device and a second user device represent different user devices, although both are user devices. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present disclosure.

[0087] When one element (e.g., a first element) is referred to as being "(operably or communicatively) coupled" or "(operably or communicatively) coupled to" or "connected to" another element (e.g., a second element), it is understood that the one element is directly connected to the other element or the one element is indirectly connected to the other element via yet another element (e.g., a third element). Conversely, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (the second element), no element (e.g., a third element) is interposed between the two.

[0088] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A data storage method, characterized in that: The method comprises: When it is determined that the access data of the received write request exists in the memory, executing the operation of the write request on the access data; Recording the operation of the write request on the access data in an incremental log file; Applying the incremental log file to a storage engine of an external storage device, so that the storage engine persistently stores the data after the write request is operated on the access data in the external storage device based on the incremental log file; The method further comprises: When it is determined that the access data does not exist in the memory, replacing the access data read from the external storage device into the memory, and executing the step of executing the operation of the write request on the access data; Before replacing the access data read from the external storage device into the memory, the method further includes: If it is determined that the access frequency of the access data exceeds a preset access frequency threshold, or the idle time of the access data does not exceed a preset time threshold, the step of replacing the access data read from the external storage device into the internal memory is performed.

2. The method according to claim 1, characterized in that: The method further comprises: If it is determined that the access frequency of the access data does not exceed the preset access frequency threshold, and the idle time of the access data exceeds the preset time threshold, the access data is read from the external storage device, and the step of executing the write request operation on the access data is performed.

3. The method according to claim 1, characterized in that The step of replacing the access data read from the external storage device into the memory includes: Based on the mapping relationship between the data structure of the database in the memory and the data structure of the storage engine, reading the access data from the external storage device; The access data read from the external storage device is replaced in the internal memory.

4. The method according to claim 3, characterized in that Before reading the access data from the external storage device, the method further includes: Splitting data belonging to the data structure of the database to obtain multiple key-value pairs corresponding to the data; A mapping relationship is established between a plurality of key-value pairs corresponding to the data and the key-value pairs of the storage engine, so as to obtain a mapping relationship between the data structure of the database and the data structure of the storage engine.

5. The method according to claim 1, characterized in that The method further comprises: When the storage space of the memory is full of data, or the storage space of the memory is cleaned up regularly, determining the access frequency or idle time of the data stored in the storage space of the memory; Based on the access frequency or the idle time, data stored in the storage space of the memory is cleared to release the storage space of the memory.

6. A data storage device, characterized in that: The device comprises: A first execution module, configured to, when determining that the access data of the received write request exists in the memory, execute the operation of the write request on the access data; A recording module, used for recording the operation of the write request on the access data in an incremental log file; An application module, configured to apply the incremental log file to a storage engine of an external storage device, so that the storage engine persistently stores the data after the write request is operated on the access data in the external storage device based on the incremental log file; The device also includes: A second execution module is used for replacing the access data read from the external storage device into the memory when it is determined that the access data does not exist in the memory, and executing the step of executing the operation of the write request on the access data; Before the second execution module, the device further includes: The third execution module is used to execute the step of replacing the access data read from the external storage device into the internal memory if it is determined that the access frequency of the access data exceeds a preset access frequency threshold, or the idle time of the access data does not exceed a preset time threshold.

7. The device according to claim 6, characterized in that The device also includes: A fourth execution device is used to read the access data from the external storage device and execute the step of executing the write request for the access data if it is determined that the access frequency of the access data does not exceed a preset access frequency threshold and the idle time of the access data exceeds a preset time threshold.

8. The device according to claim 6, characterized in that The first execution module includes: A reading submodule, configured to read the access data from the external storage device based on a mapping relationship between a data structure of the database in the memory and a data structure of the storage engine; A replacement submodule is used to replace the access data read from the external storage device into the internal memory.

9. The device according to claim 8, characterized in that Before the reading submodule, the first execution module further includes: A splitting submodule, used for splitting the data belonging to the data structure of the database to obtain a plurality of key-value pairs corresponding to the data; A submodule is established to establish mapping relationships between the multiple key-value pairs corresponding to the data and the key-value pairs of the storage engine, so as to obtain a mapping relationship between the data structure of the database and the data structure of the storage engine.

10. The device according to claim 6, characterized in that The device also includes: A determination module, configured to determine the access frequency or idle time of the data stored in the storage space of the memory when the storage space of the memory is full of data or the storage space of the memory is cleaned up regularly; A cleaning module is used to clean up the data stored in the storage space of the memory based on the access frequency or the idle time to release the storage space of the memory.

11. A data storage system, characterized in that: The system comprises: A memory-based database and an external storage device-based storage engine that interacts with the database. The database is used to, when determining that the access data of the received write request exists in the memory, execute the operation of the write request on the access data, record the operation of the write request on the access data in an incremental log file, and apply the incremental log file to the storage engine; The storage engine is used to persistently store the data after the write request is operated on the access data in the external storage device based on the incremental log file; The database is further used for replacing the access data read from the external storage device into the memory when it is determined that the access data does not exist in the memory, and performing the step of executing the operation of the write request on the access data; Before replacing the access data read from the external storage device into the internal memory, the database is further used to: If it is determined that the access frequency of the access data exceeds a preset access frequency threshold, or the idle time of the access data does not exceed a preset time threshold, the step of replacing the access data read from the external storage device into the internal memory is performed.

12. An electronic device comprising: one or more processors; A computer readable medium configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method according to any one of claims 1 to 5.

13. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data storage method according to any one of claims 1 to 5 is implemented.

Images