Data storage method, device, non-relational database system and storage medium

By obtaining the data to be stored in a big data environment and determining the storage space index identifier, selecting the unused storage space to perform storage operations, the problem of low data reading performance in the prior art is solved, and high efficiency, atomicity and consistency are achieved data storage.

CN111414356BActive Publication Date: 2024-11-29BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN201910013131.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-01-07
Publication Date
2024-11-29
Estimated Expiration
2039-01-07

AI Technical Summary

Technical Problem

The prior art is difficult to effectively improve data reading performance in a big data environment, especially in high concurrency and high frequency usage scenarios, and it is difficult to ensure the atomicity and consistency of data scattered storage.

Method used

By obtaining the data to be stored, determining the currently used storage space index identifier, and selecting the unused storage space to perform storage operations, completing asynchronous data storage updates, ensuring that the distributed storage of data is balanced between high efficiency and consistency.

Benefits of technology

It realizes efficient reading performance of data, meets the needs of high concurrency and high frequency usage, and ensures the atomicity and consistency of data storage, improving overall storage efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111414356B_ABST
    Figure CN111414356B_ABST
Patent Text Reader

Abstract

An embodiment of the present invention provides a data storage method, device, non-relational database system and storage medium, the method comprising: acquiring data to be stored; determining a first index identifier corresponding to a first storage space currently in use; selecting a second storage space currently unused pointed to by a second index identifier to perform a storage operation on the data to be stored, and when determining that the storage operation on the second storage space is completed, updating the second storage space to the storage space currently in use.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to data storage technology, and in particular to a data storage method, device, non-relational database system and storage medium. Background Art

[0002] In the era of big data, various cloud storage systems for big data ensure the high reliability of the services provided through the management and operation mechanisms of the cloud platform; on the other hand, by pooling the storage resources contributed by all participating users, low-cost and huge storage space is provided to effectively meet the needs of rapidly developing Internet applications in terms of large scale, efficiency, high reliability, high scalability and high cost performance.

[0003] In order to make better use of big data, various database systems are needed to be able to regularly pull updated data information from the cloud storage system to local storage. Local storage can facilitate users to quickly search big data and other usage needs. For example, in an online shopping mall platform system, the business system regularly pulls updated product data information from the product center to local storage, so that users can directly access the business system through the client and meet the client's data query and other usage needs. Summary of the invention

[0004] Embodiments of the present invention provide a data storage method, device, non-relational database system and storage medium, which can improve data reading performance.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a data storage method, including: acquiring data to be stored; determining a first index identifier corresponding to a first storage space currently in use; selecting a second storage space that is not currently in use and pointed to by a second index identifier to perform a storage operation on the data to be stored, and when determining that the storage operation on the second storage space is completed, updating the second storage space to the storage space currently in use.

[0007] In a second aspect, an embodiment of the present invention provides a data storage device, comprising an acquisition module for acquiring data to be stored; a routing judgment module for determining a first index identifier corresponding to a first storage space currently in use; and a routing switching module for selecting a second storage space currently unused pointed to by a second index identifier to perform a storage operation on the data to be stored, and when it is determined that the storage operation on the second storage space is completed, updating the second storage space to the storage space currently in use.

[0008] In a third aspect, an embodiment of the present invention provides a non-relational database system, comprising a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, it executes the data storage method provided by the embodiment of the present invention.

[0009] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data storage method provided by the embodiment of the present invention is implemented.

[0010] The embodiments of the present invention have the following beneficial effects:

[0011] Acquire the data to be stored, determine the first index identifier corresponding to the currently used first storage space, select the currently unused second storage space pointed to by the second index identifier to perform the storage operation of the data to be stored, and when it is determined that the storage operation of the second storage space is completed, update the second storage space to the currently used storage space. In this way, asynchronous operations of current use of data and storage update of data are realized, so that the requirements of high-efficiency, high-concurrency and high-frequency use can be met through the distributed storage of data, and the atomicity and consistency of data can be ensured when the distributed storage of data is used to store and update the overall data, thereby improving data reading performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of an application scenario of a data storage method provided by an embodiment of the present invention;

[0013] Figure 2 A schematic diagram of a data storage method according to an embodiment of the present invention.

[0014] Figure 3 A flowchart of a method for implementing overlay storage based on the Hash data type of Redis provided in one embodiment of the present invention;

[0015] Figure 4 A flowchart of a search method for storing big data using a String data type based on Redis provided in one embodiment of the present invention;

[0016] Figure 5 A flow chart of a data storage method after data structure conversion after data synchronization between a business system A and a commodity center provided by an embodiment of the present invention;

[0017] Figure 6 A flowchart of a data storage method provided by another embodiment of the present invention;

[0018] Figure 7 A schematic diagram of the hardware structure of a non-relational database system provided by an embodiment of the present invention;

[0019] Figure 8 A schematic diagram of the structure of a data storage device implementing a data storage solution and a data reading solution provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0022] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are subject to the following explanations.

[0023] 1) Atomicity, one of the characteristics of a database, means that all operations contained in a transaction are either successful or fail and roll back.

[0024] 2) Consistency, one of the characteristics of a database, means that a transaction must transform the database from one consistent state to another, that is, the database must be in a consistent state before and after a transaction is executed.

[0025] 3) Redis is a very fast non-relational database that can store mappings between keys and five different types of values. It can persist key-value pairs stored in memory to disk, use replication to scale read performance, and use client sharding 1 to scale write performance.

[0026] 4) Non-relational database, also known as NoSQL, means not just SQL (Structured Query Language), mainly refers to a database design pattern that is non-relational, distributed, and does not provide the four basic elements of database transaction processing: atomicity, consistency, isolation, and durability.

[0027] like Figure 1 As shown, a schematic diagram of an optional application scenario for implementing the data storage method provided by an embodiment of the present invention. The online shopping mall platform system generally includes a product center, a business system, and a client. The business data of online products used by users through the client is stored in the product center. The business system periodically pulls updated business data from the product center for local storage. The local storage can quickly respond to the user's search, access, and other usage needs through the client. An optional process for the online product platform system to implement client query operations includes:

[0028] Step S11, the business system schedules product synchronization tasks and sends business requests to the product center according to the business;

[0029] Step S12, the product center processes the business request, queries the local, and assembles and returns the general product data structure based on the current business request; here, the general product data structure sets the data structure used to store business data in the product center so as to provide comprehensive data attributes.

[0030] Step S13, the business system converts the received general commodity data structure into a business commodity data structure and stores it locally; here, the business commodity data structure refers to the data structure used by the business center to store business data, so as to redefine data attribute fields to quickly respond to different search requests.

[0031] Step S14, the client obtains the search conditions input by the user and sends them to the business system;

[0032] Step S15: the business system queries the local database based on the search conditions, obtains the corresponding product data information and returns it.

[0033] Step S16: The client receives the returned product data information result from the business system and displays it to the user.

[0034] Among them, the business system's scheduled commodity synchronization task, the method of obtaining data from the commodity center and storing it locally can adopt the data storage method provided in the embodiment of this application. Secondly, in the online commodity platform system, the data stored in the commodity center also needs to be obtained from other data systems such as the price center and stored locally. It can be understood that the method for the commodity center to obtain data from other data systems and store it locally can also adopt the data storage method provided in the embodiment of this application.

[0035] See also Figure 2 , which is a flow chart of the data storage method provided in the embodiment of the present application, can be applied to a database system that needs to store data locally, such as Figure 1The business system or product center shown in the figure, the method comprises the following steps:

[0036] Step 101, obtaining data to be stored;

[0037] Here, the acquisition of data to be stored may refer to data to be updated for local storage that is pulled from other data systems at set intervals. Figure 1 Taking the online commodity platform system shown in as an example, the data to be stored may refer to the data result of the general commodity data structure returned by the commodity center after the business system requests the commodity center through the scheduled commodity synchronization task.

[0038] Step 103, determining a first index identifier corresponding to the first storage space currently in use;

[0039] Here, the currently used first storage space refers to the storage space that responds to the current use demand, and the current use demand refers to the request from the client used by the user, such as the data search request issued by the user through the client. The database system includes a first storage space and a second storage space, and a first index identifier corresponding to the first storage space and a second index identifier corresponding to the second storage space are established respectively. The first storage space and the second storage space are switched according to the set rules to determine that one of them is the currently used storage space, and the other is the currently unused storage space.

[0040] Step 105: Select a currently unused second storage space pointed to by a second index identifier to perform a storage operation on the data to be stored, and when it is determined that the storage operation on the second storage space is completed, update the second storage space to a currently used storage space.

[0041] By determining the first index identifier corresponding to the currently used first storage space, the storage space corresponding to another index identifier, that is, the currently unused second storage space pointed to by the second index identifier, can be determined to perform the storage operation of the data to be stored. At this time, the first storage space is used as the currently used storage space to respond to the client's usage requirements, and the second storage space is used as the currently unused storage space to perform the storage operation of the data to be stored, thereby realizing the asynchronous operation of the current use of data and the storage update of data.

[0042] In the above embodiment of the present application, the database system obtains the data to be stored, determines the first index identifier corresponding to the currently used first storage space, selects the currently unused second storage space pointed to by the second index identifier to perform the storage operation of the data to be stored, and when it is determined that the storage operation of the second storage space is completed, updates the second storage space to the currently used storage space. In this way, asynchronous operations of current use of data and storage update of data are realized, so that high-efficiency, high-concurrency and high-frequency usage requirements can be met through distributed storage of data, and the atomicity and consistency of data can be ensured when the distributed storage of data is used to store and update the overall data, thereby improving data reading performance.

[0043] In some embodiments, the selecting the currently unused second storage space pointed to by the second index identifier to perform the storage operation of the data to be stored includes:

[0044] Select the currently unused second storage space pointed to by the second index identifier, delete the entire historical data of the second storage space, and store the data to be stored in the second storage space.

[0045] Here, selecting the second storage space to perform the storage operation of the data to be stored refers to a data storage operation through an overwrite operation. Taking the database system as a non-relational database system, such as the Redis database system, when using the unordered hash table Hash data type provided by Redis to store data, if you want to overwrite the entire key, Redis itself does not provide a corresponding supported application programming interface (API). Please refer to Figure 3 For the Redis database system, as an optional solution, the HMSET command can be used to perform batch append overwrite to achieve overwrite storage. However, the HMSET operation is only a batch append overwrite method, and will not delete the deletion function that does not contain the key in the input parameter set. The method of implementing batch append overwrite with the HMSET operation includes:

[0046] Step S22, the client uses hmset operation to issue a batch append overwrite operation request; Figure 3 As shown in the figure, the client makes a batch append overwrite operation request for a:1 and c:3 in the map according to the value;

[0047] Step S23, the server responds to the hmset operation of the client and forms the operation results of map modification and addition stored locally accordingly; Figure 3As specifically shown in the figure, the server performs a batch append and overwrite operation request on a:1; c:3 in the map according to the value sent by the client, forming the operation results of a:1; b:2; c:3 in the map, where a:1 is the modification operation result, b:2 is the non-deletion operation result, and c:3 is the append operation result.

[0048] Alternatively, as another optional solution, you can use Redis to start a transaction and perform a del+set combination operation to implement the overwrite action in a different direction. However, Redis only implements the atomicity of the transaction and cannot guarantee the consistency of the combined execution results of the operations in the transaction. In addition, Redis is a single-process, single-threaded operation, and query blocking may occur when executing complex transactions. The method of implementing the overwrite action in a different direction by performing the del+set combination operation includes:

[0049] Step S24, the client sends a start transaction multi command to the server;

[0050] Step S25, the client sends a del deletion operation request to the server;

[0051] Step S26, the client sends a hmset operation request to the server;

[0052] Step S27, the client sends an exec command to the server;

[0053] Step S28, the server executes the multiple Redis commands from the client at one time to form an execution result.

[0054] The del+set combination operation is based on the multi / exec transaction command, which executes multiple Redis commands at once. When the transaction is complex, the execution results will be inconsistent, such as Figure 3 Specifically, the server fails to execute the batch append overwrite operation request for a:1; c:3 in the map according to value sent by the client, and the server successfully executes the deletion operation request del:test sent by the client, resulting in an operation result in which value is nil (empty) and the data in the map is deleted.

[0055] In the above embodiment of the present application, a second storage space that is not currently in use is selected to implement a data storage operation of an overwrite operation, and the first storage space that is currently in use is maintained to respond to the client's usage requirements. When the storage operation of the second storage space is completed, the second storage space is updated to the storage space corresponding to the next usage requirement response. In this way, asynchronous operations of the current use of data and the storage update of data are implemented, so that the storage space for the data storage operation of the overwrite operation is independent of the storage space that supports the current usage requirements. The storage of the stored data can be an overall overwrite after recalculating the full amount of data, and the consistency of the execution results can be ensured.

[0056] In some embodiments, storing the data to be stored in the second storage space includes:

[0057] The data to be stored is stored in the second storage space in the form of an unordered hash table containing key-value pairs.

[0058] Storing the data to be stored in the second storage space in the form of an unordered hash table Hash data type containing key-value pairs can avoid the need to transmit the entire data structure every time a query is made. By storing the Hash data structure rules, the data can be sufficiently dispersed, which can greatly reduce the bandwidth occupied by the response data.

[0059] Take the Redis database system as an example. When using the string data type provided by Redis to store data, refer to Figure 4 Since String is binary safe, it can store serialized objects of any data type, and the desired data structure can be constructed for serialized storage. The advantage of such storage is that the operation of the overall data structure is more flexible. However, when the overall data volume of the constructed data structure is relatively large (for example: 5M), although Redis supports String space of 512M, when it is used frequently in high-concurrency scenarios, it will cause a rapid decline in performance, because network bandwidth and other issues will also cause network congestion and other conditions. Exemplary search methods for storing big data based on Redis's String data type may include:

[0060] Step S31, the client sends an operation request to the server to search for an object in the map according to the value;

[0061] Step S32, the server returns the corresponding map search result according to the operation request;

[0062] Step S33, the client sends an operation request to the server to search for objects in the list according to the value;

[0063] Step S34, the server returns the corresponding list search results according to the operation request;

[0064] Step S35, the client sends a data deletion operation request to the server, and the server executes the deletion based on the operation.

[0065] Due to the serialized storage characteristics of the String data type, when the overall data volume of the constructed data structure is relatively large, the search performance will drop rapidly. In the above embodiment of the present application, by using the Hash data type of Redis to replace the String data type, it is possible to avoid the need to transmit the entire data structure in the network every time a query is made. The storage method can directly obtain the index value locally based on the fixed key and the key stored in the Hash when searching, which greatly reduces the amount of data in the network transmission process, solves the problem of decreased search performance when Redis uses the String data type to store large data under high concurrency, improves search performance, and improves user experience.

[0066] In some embodiments, before selecting the currently unused second storage space pointed to by the second index identifier to perform the storage operation of the data to be stored, the method includes:

[0067] The data structure of the data to be stored is converted, and the converted data to be stored is filtered according to set conditions to obtain updated data to be stored.

[0068] Here, the database system pulls the data to be stored locally for local storage. The main purpose is to facilitate the conversion of the data structure, so that it can be converted into a data structure that can quickly respond to the client's data usage needs and then stored locally, thereby improving the efficiency of data usage. Figure 1 Taking the online commodity platform system shown in as an example, the business system regularly triggers tasks to query the commodity center data according to the business (full volume); the commodity center locally filters all commodities corresponding to the business, assembles a general commodity data structure and returns it to the business system; the business system converts the general commodity data structure into a business commodity data structure, and constructs a commodity Hash data structure that matches various query conditions of the client as indexes. The purpose of converting the general commodity data structure into a business commodity data structure is to facilitate the construction of a commodity data structure that can match various query conditions of the client as indexes, so as to achieve the purpose of quickly responding to the client's data usage needs.

[0069] The filtering of the converted data to be stored according to the set conditions refers to deleting duplicate or erroneous data according to the set filtering conditions to avoid problems such as data duplication. Figure 5Taking the example of business system A periodically pulling all the commodity data from the commodity center, the data storage after data structure conversion after data synchronization between business system A and the commodity center includes the following steps:

[0070] Step S41, the business system periodically pulls tasks, pull A; Figure 5 As specifically shown in the figure, the commodity data set pulled from the commodity center by business system A at 10:00 is {A business system commodities: a_were1, a_were2, a_were3; B business system commodities: b_were1, b_were2, b_were3};

[0071] Step S42, business system A converts the pulled commodity center full commodity data into a business commodity data structure, and constructs multiple sets of data storage structures based on search conditions according to the converted data set; Figure 5 As specifically shown in the figure, the business system A converts the business commodity data structure into:

[0072] a_were1->product1(bj_ky)

[0073] a_were2->product2(bj_jp)

[0074] a_were3->product3(sh_yd)

[0075] The constructed data storage structure is: Hash

[0076] <"bj",[product1,product2]>

[0077] <"sh", [product3]>

[0078] Step S43, the product center modifies the business product;

[0079] Step S44, the business system periodically pulls tasks, pull B; Figure 5 As specifically shown in the figure, the commodity data set pulled from the commodity center by business system A at 10:05 is {A business system commodities: a_were1, a_were2, a_were3, a_were4; B business system commodities: b_were1, b_were2, b_were3};

[0080] Step S45, the business system A converts the pulled commodity center full commodity data into a business commodity data structure, and constructs multiple sets of data storage structures based on search conditions according to the converted data set; Figure 5 As specifically shown in the figure, the business system A converts the business commodity data structure into:

[0081] a_were1->product1(bj_ky)

[0082] a_were2->product2(bj_jp)

[0083] a_were3->product3(sh_yd)

[0084] a_were4->product4(bj_ky)

[0085] The constructed data storage structure is: Hash

[0086] <"bj", [product2]>

[0087] <"sh", [product3]>

[0088] Because the product data set pulled by business system A at 10:05 contains more product product4 than the product data set pulled at 10:00, and because product product4 is the same as product product1 in the product data set, product product1 is deleted from the data storage structure instead of adding product product4 to the data storage structure when the data storage structure is reconstructed. Therefore, operations on the data storage structure require filtering operations on similar data, and all data must be recalculated for overall coverage storage operations.

[0089] In some embodiments, after the second storage space is updated to the currently used storage space, the following steps may be performed:

[0090] Obtain a search request, determine the second index identifier corresponding to the second storage space currently in use according to the search request, and select the second storage space pointed to by the second index identifier to perform a search operation in response to the search request.

[0091] The first storage space and the second storage space are switched to perform asynchronous operations of current use of data and storage update of data respectively. For example, when the first storage space is used as the currently used storage space to respond to the use demand of the client, the second storage space is selected as the currently unused storage space to perform the data storage operation. When the storage operation of the second storage space is completed, the second storage space is switched to the currently used storage space to respond to the real-time use demand of the client; when the storage operation needs to be performed again, the first storage space is selected as the currently unused storage space to perform the data storage operation. When the storage operation of the first storage space is completed, the first storage space is switched to the currently used storage space again to respond to the real-time use demand of the client; this cycle is repeated, and at any time, the first storage space and the second storage space are kept. The storage space that has recently completed the storage operation will be used as the currently used storage space to respond to the real-time use demand, and the other will be used as the currently unused storage space to perform the next storage operation. In the process of performing asynchronous storage operations through one of the storage spaces, performing full deletion and new storage operations will not affect the other storage space to respond to the high concurrent use demand of the client, while ensuring the data reading and retrieval performance of the database system. Among them, usage demand usually refers to the search request sent by the user to the business system through the client.

[0092] In some embodiments, determining the first index identifier corresponding to the currently used first storage space includes:

[0093] It is determined that the storage space corresponding to the current search route is the first storage space currently in use, and a first index identifier corresponding to the first storage space is determined according to a correspondence between the index identifier and the storage space.

[0094] The current search route refers to the route that the database system determines to respond to the current search request to perform data query operations based on the current search request. Here, by establishing a first index identifier and a second index identifier corresponding to the first storage space and the second storage space, the current search route is identified by the first index identifier and the second index identifier. For example, when it is determined that the storage space currently in use is the first storage space, the first index identifier is used as the identifier of the current search route. When a search request from a client is received, the search operation is performed according to the first storage space pointed to by the identifier of the current search route, and the second storage space pointed to by the second index identifier that is not used as the current search route is used as the storage space for performing data storage operations; when it is determined that the storage space currently in use is the second storage space, the second index identifier is switched to be the identifier of the current search route. When a search request from a client is received, the search operation is performed according to the second storage space pointed to by the identifier of the current search route, and the first storage space pointed to by the first index identifier that is not used as the current search route is switched to the storage space for performing data storage operations.

[0095] In the above embodiments of the present application, the first storage space and the second storage space are dynamically switched according to the logical index of reading and writing to realize asynchronous construction of storage space, that is, the storage space data for executing data storage operations has no effect on the synchronous use of storage space, that is, the storage space for executing search request responses. When the construction is completed, the storage space index is switched back to the storage space after the construction is completed, which solves the problem of the database system being able to take into account reading efficiency while performing local storage and use of data after synchronization.

[0096] In order to better understand the data storage method provided in the embodiment of the present application, please refer to Figure 6 , the following specifically takes the database system as Figure 1 Taking the business system described in as an example, the data storage method provided by the embodiment of the present invention is described as follows:

[0097] Step S61, periodically pull all data from the product center and obtain it locally;

[0098] Step S62, performing data structure conversion on the pulled full amount of data and then performing full amount filtering calculation;

[0099] Step S63, determining the index identifier corresponding to the current search route; when the corresponding index identifier is route identifier 1, executing step S64; when the corresponding index identifier is route identifier 2, executing step S65;

[0100] Step S64, deleting the historical full data of storage space 2, adding the latest data to be stored to storage space 2, and switching the index identifier corresponding to the current search route to route identifier 2;

[0101] Step S65 , delete the historical full data of storage space 1 , add the latest data to be stored to storage space 1 , and switch the index identifier corresponding to the current search route to route identifier 1 .

[0102] In the above embodiment, by judging the index identifier corresponding to the current search route, operating the storage space pointed to by another index identifier, performing full deletion and new addition operations, when the operation is successful, switching the index identifier corresponding to the current search route to the other index identifier, and synchronizing the query operation by dynamically switching the route index, the pre-constructed commodity Hash data structure can be directly stored in the non-relational database system, and when searching, the index value can be directly obtained locally based on the fixed key and the key stored in the Hash, which greatly reduces the amount of data in the network transmission process, avoids network congestion in high-concurrency search scenarios, and improves search performance; secondly, asynchronous execution of data storage operations can ensure the final consistency of transaction operations.

[0103] The data storage method provided in the embodiment of the present invention can be applied to non-relational database systems, such as non-relational database systems based on Redis, ES, MongoDB and other storage data sources, see Figure 7 , which is an optional hardware structure diagram of the non-relational database system provided in an embodiment of the present invention, and various servers or service clusters used to implement the data storage method of the embodiment of the present application can be non-relational database systems in a broad sense, including: a processor 101, a memory 102 and a network interface 104, and each component is coupled together through a bus system 105. It can be understood that the bus system 105 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 105 can also include a power bus, a control bus and a status signal bus.

[0104] The memory 102 is used to store computer programs that can run on the processor 101, and can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0105] The memory 102 in the embodiment of the present invention is used to store various types of data to support the operation of the data storage device that implements the data storage method. Examples of these data include: any executable program used to operate on the data storage device, such as an operating system and an application; data to be stored that is periodically pulled from other data centers and used for local storage; wherein the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., to implement various basic services and process hardware-based tasks. The data storage device that implements the data storage method provided by the embodiment of the present invention can be included in the application.

[0106] The data storage method disclosed in the above embodiment of the present invention can be implemented by the processor 101 running a computer program. The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 101 or the instruction in the form of software. The above processor 101 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 101 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 101 can be a microprocessor or any conventional processor, etc. In combination with the steps of the data storage method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to perform, or it can be performed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0107] In an exemplary embodiment, the data storage device may be used by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0108] In an exemplary embodiment, it is a structural schematic diagram of a data storage device provided by an embodiment of the present invention, and the data storage device includes: a data pulling module 11, used to obtain data to be stored; a routing judgment module 13, used to determine a first index identifier corresponding to a first storage space currently in use; a routing switching module 15, used to select a currently unused second storage space pointed to by a second index identifier to perform a storage operation of the data to be stored, and when it is determined that the storage operation of the second storage space is completed, the second storage space is updated to the currently used storage space.

[0109] In some embodiments, the routing switching module 15 is specifically configured to select a currently unused second storage space pointed to by a second index identifier, delete the entire historical data of the second storage space, and store the data to be stored in the second storage space.

[0110] In some embodiments, the routing switching module 15 is specifically configured to store the data to be stored in the second storage space in the form of an unordered hash table containing key-value pairs.

[0111] In some embodiments, the device further includes a data construction module 14, which is used to convert the data structure of the data to be stored, and filter the converted data to be stored according to set conditions to obtain updated data to be stored.

[0112] In some embodiments, the device also includes a data reading module 12, which is used to obtain a search request; the routing judgment module 13 is also used to determine the second index identifier corresponding to the second storage space currently in use based on the search request; the routing switching module 15 is also used to select the second storage space pointed to by the second index identifier to perform a search operation in response to the search request.

[0113] In some embodiments, the route determination module 13 is used to determine that the storage space corresponding to the current search route is the first storage space currently in use, and determine the first index identifier corresponding to the first storage space according to the correspondence between the index identifier and the storage space.

[0114] The data storage device provided in the above embodiment is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the data storage device provided in the above embodiment and the data storage method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0115] For example, the first storage space is redis storage space and the second storage space is redis storage space (copy). Figure 8 , the data storage device provided in the above embodiment of the present application implements the data storage scheme and the data reading scheme by asynchronously performing storage operations based on the idea of ​​replicating storage space, and synchronously performing query operations by dynamically switching routing indexes, wherein the dynamic index routing module is used to implement the steps performed by the above routing judgment module 13 and the routing switching module 15. In the data storage scheme, when the data pulling module receives the data to be stored, after the data structure conversion and filtering by the data construction module, the current search routing identifier is identified by the dynamic index reason module. When the current search routing identifier is routing identifier 1, the Redis storage space (copy) corresponding to routing identifier 2 is used to perform full data storage deletion and new storage operations; when the current search routing identifier is routing identifier 2, the Redis storage space corresponding to routing identifier 1 is used to perform full data storage deletion and new storage operations. In the data reading scheme, when the data reading module receives a search request, it identifies the current search routing identifier through the dynamic index reason module. When the current search routing identifier is routing identifier 1, the Redis storage space corresponding to routing identifier 1 responds to the search operation; when the current search routing identifier is routing identifier 2, the Redis storage space (replication) corresponding to routing identifier 2 responds to the search operation.

[0116] In an exemplary embodiment, the present application also provides a readable storage medium, such as a memory including an executable program, and the above executable program can be executed by a processor to complete the steps of the above method. The readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM; it can also be various devices including one or any combination of the above memories, such as mobile phones, computer devices, tablet devices, personal digital assistants, medical devices, etc.

[0117] The above description is only an embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement and improvement made within the spirit and scope of the present invention are included in the protection scope of the present invention.

Claims

1. A data storage method, characterized in that: include: Get the data to be stored; Determine a first index identifier corresponding to a first storage space currently in use; Selecting a currently unused second storage space pointed to by a second index identifier to perform a storage operation on the data to be stored, and when determining that the storage operation on the second storage space is completed, updating the second storage space to a currently used storage space, and updating the first storage space to a currently unused storage space to perform a data storage operation; Obtain a search request, determine the second index identifier corresponding to the second storage space currently in use according to the search request, and select the second storage space pointed to by the second index identifier to perform a search operation in response to the search request.

2. The method according to claim 1, characterized in that The selecting the currently unused second storage space pointed to by the second index identifier to perform the storage operation of the data to be stored includes: Select the currently unused second storage space pointed to by the second index identifier, delete the entire historical data of the second storage space, and store the data to be stored in the second storage space.

3. The method according to claim 2, characterized in that The storing the data to be stored in the second storage space includes: The data to be stored is stored in the second storage space in the form of an unordered hash table containing key-value pairs.

4. The method according to claim 1, characterized in that Before the currently unused second storage space pointed to by the second index identifier is selected to perform the storage operation of the data to be stored, the method includes: The data structure of the data to be stored is converted, and the converted data to be stored is filtered according to set conditions to obtain updated data to be stored.

5. The method according to claim 1, characterized in that The determining the first index identifier corresponding to the currently used first storage space includes: It is determined that the storage space corresponding to the current search route is the first storage space currently in use, and a first index identifier corresponding to the first storage space is determined according to a correspondence between the index identifier and the storage space.

6. A data storage device, characterized in that: include: A data pulling module is used to obtain data to be stored; A routing determination module, used to determine a first index identifier corresponding to a first storage space currently in use; a routing switching module, configured to select a currently unused second storage space pointed to by a second index identifier to perform a storage operation on the data to be stored, and when determining that the storage operation on the second storage space is completed, update the second storage space to a currently used storage space, and update the first storage space to a currently unused storage space to perform a data storage operation; Search module, used to obtain search requests; The routing determination module is further used to determine the second index identifier corresponding to the second storage space currently in use according to the search request; The routing switching module is further used to select the second storage space pointed to by the second index identifier to perform a search operation in response to the search request.

7. The device according to claim 6, characterized in that The routing switching module is specifically used to select the currently unused second storage space pointed to by the second index identifier, delete the historical full data of the second storage space, and store the data to be stored in the second storage space.

8. The device according to claim 7, characterized in that The routing switching module is specifically used to store the data to be stored in the second storage space in the form of an unordered hash table containing key-value pairs.

9. The device according to claim 6, characterized in that It also includes a data construction module, which is used to convert the data structure of the data to be stored, filter the converted data to be stored according to set conditions, and obtain updated data to be stored.

10. The device according to claim 6, characterized in that The route determination module is used to determine that the storage space corresponding to the current search route is the first storage space currently in use, and determine the first index identifier corresponding to the first storage space according to the corresponding relationship between the index identifier and the storage space.

11. A non-relational database system comprising a processor and a memory for storing a computer program that can be run on the processor; wherein: When the processor is used to run the computer program, it executes the data storage method according to any one of claims 1 to 5.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data storage method according to any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Method and device for processing data

    CN102968498A