Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment, and relates to the technical field of data processing.The method comprises the steps that firstly, in response to a data processing instruction, incremental data corresponding to a data operation request is obtained; then storing the incremental data into an incremental cache database; and when the data volume of the incremental data in the incremental cache database reaches a preset threshold value, flashing the incremental data in the incremental cache database into the column storage database, merging the incremental data with the stock data in the column storage database, and transferring the incremental data in the incremental cache database into the query cache database. Compared with the prior art, the incremental data generated by the data processing operation is synchronously stored through the cache database and the column storage database, the cache database is utilized to provide the rapid read-write performance of row storage, the advantages of the column storage database can be exerted, the data processing efficiency is improved, and the user experience is improved. And the real-time consistency of the business data and the analysis data is ensured.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, storage medium and electronic device. Background Art

[0002] Database storage format is a crucial aspect in database management systems, which directly affects the organization and retrieval efficiency of data. In databases, there are two main storage formats, namely row storage and column storage. In row storage, all column data of a single row of a table are stored continuously on the disk, and in column storage, each column data of a table is stored continuously on the disk.

[0003] At present, column storage is usually used for online analytical processing (OLAP) databases that focus on large amounts of data and complex analytical queries.

[0004] However, enterprises usually have dual needs for online transaction processing (OLTP) and online analytical processing. Online transaction processing requires frequent transactional operations such as updates and insertions. The columnar storage method has low data processing efficiency and cannot guarantee data timeliness. Summary of the invention

[0005] In view of this, the present application provides a data processing method, device, storage medium and electronic device, the main purpose of which is to improve the technical problem that the data processing efficiency in the current data storage method is relatively low and the timeliness of the data in the database cannot be guaranteed.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] In response to the data processing instruction, obtaining incremental data corresponding to the data operation request;

[0008] Storing the incremental data in an incremental cache database;

[0009] When the amount of incremental data in the incremental cache database reaches a preset threshold, the incremental data in the incremental cache database is flushed to the column storage database and merged with the stock data in the column storage database;

[0010] The incremental data in the incremental cache database is transferred to the query cache database.

[0011] In a second aspect, the present application provides a data processing device, including:

[0012] An acquisition module is configured to acquire incremental data corresponding to a data insertion, update and / or deletion request in response to a data processing instruction;

[0013] A storage module, configured to store the incremental data in an incremental cache database;

[0014] A processing module is configured to, when the amount of incremental data in the incremental cache database reaches a preset threshold, flush the incremental data in the incremental cache database to the column storage database and merge it with the stock data in the column storage database;

[0015] The transfer module is configured to transfer the incremental data in the incremental cache database to the query cache database.

[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0017] In a fourth aspect, the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0018] In a fifth aspect, the present application provides a computer program product having a computer program stored thereon, wherein the computer program implements the method described in the first aspect when executed by a processor.

[0019] By means of the above technical scheme, the present application provides a data processing method, device, storage medium and electronic device, which first respond to data processing instructions to obtain incremental data corresponding to data operation requests; then store the incremental data in an incremental cache database; when the amount of incremental data in the incremental cache database reaches a preset threshold, flush the incremental data in the incremental cache database to the column storage database, and merge it with the stock data in the column storage database, and then transfer the incremental data in the incremental cache database to the query cache database. Compared with the current prior art, the present application synchronously stores the incremental data generated by data processing operations through a cache database and a column storage database, which not only utilizes the fast read and write performance of row storage provided by the cache database, but also gives play to the advantages of a column storage database, and not only has transaction processing functions but also can support the analysis needs of large amounts of data, improves data processing efficiency, and ensures the real-time consistency of business data and analysis data.

[0020] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 A schematic diagram of the structure of a data storage system provided by an embodiment of the present application is shown;

[0024] Figure 2 A schematic diagram of an example process provided by an embodiment of the present application is shown;

[0025] Figure 3 A schematic diagram of a data processing method provided in an embodiment of the present application is shown;

[0026] Figure 4 A schematic diagram of an example process provided by an embodiment of the present application is shown;

[0027] Figure 5 A schematic diagram showing a flow chart of another data processing method provided in an embodiment of the present application is shown;

[0028] Figure 6 A schematic diagram of an example process provided by an embodiment of the present application is shown;

[0029] Figure 7 A schematic diagram of an example process provided by an embodiment of the present application is shown;

[0030] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown;

[0031] Figure 1 middle:

[0032] 1-Incremental cache module, 2-Query cache module, 3-Column storage module. DETAILED DESCRIPTION

[0033] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0034] Combine the following Figure 1 A data storage system according to some embodiments of the present application is described.

[0035] This embodiment provides a data storage system, such as Figure 1 As shown, it includes: an incremental cache module 1, a query cache module 2 and a column storage module 3; the incremental cache module 1 is used to store incremental data generated by data processing operations; the query cache module 2 is used to store incremental data transferred after the incremental cache module 1 performs a flush operation; the column storage module 3 is used to store the stock data in a column format on a disk and / or a shared storage database. The query cache module 2 is also used to store the data queried in the column storage module 3 based on a point query request for data.

[0036] In some examples, such as Figure 2 As shown, the data storage system provided in this embodiment is an integrated data storage system based on a cache module (Cache System) and a column storage module (Data Storage), wherein the cache module is divided into two parts: an incremental cache (Delta Cache) module and a query cache (Query Cache) module, and the data format is consistent and distinguished by marking. Delta Cache undertakes the incremental data of the business system, and the addition, deletion and modification of business data are direct operations on the cache. The cached data is stored in the form of key-value, wherein the key is generally a characteristic value that can uniquely identify a row of data, similar to the identifier rowid, a key column or a combination of key columns, etc., and the value stores the entire row of data, which can quickly add, delete and modify the entire row of data, providing a fast row write performance similar to a row storage database. The stock data is stored in the column storage database, and the incremental data in the cache will be flushed to the column storage database (Database, DB) after reaching a certain threshold, and merged with the stock data, and the data is finally persisted in the column storage database.

[0037] Exemplarily, the incremental cache module is a module that directly operates on requests such as insert, update, and delete of data, that is, where incremental data is stored; the query cache module is used to optimize point queries, and point queries are specifically tuned using cache characteristics. The cache module is a temporary storage area for storing data, designed to speed up data access and improve system performance. It saves copies of frequently accessed or computationally expensive data so that the data can be quickly obtained without having to load it from the original source (such as a database, file system, or remote service) every time.

[0038] Exemplarily, the column storage module is used to store the stock data in a column format on a disk or shared storage, and the column storage and indexing of the columns are suitable for OLAP queries. Column storage only requires operations such as query or aggregation calculation on certain columns in the table, which is more efficient than row storage because the database can avoid reading redundant columns and reduce the amount of data read from the disk. Column storage can usually achieve better data compression rate and storage efficiency due to the consistency of data types.

[0039] Compared with the current existing technology, this embodiment provides a data storage system, wherein the data storage system includes an incremental cache module, a query cache module and a column storage module. Specifically, the incremental cache module can be used to store incremental data generated by data processing operations; the query cache module can be used to store incremental data transferred after the incremental cache module performs a flush operation; the column storage module can be used to store the existing data in a column format on a disk and / or a shared storage database; the query cache module can also be used to store the data obtained by querying the column storage module based on a point query request for the data. The data storage system in this embodiment not only uses the cache module to provide the performance of fast read and write of row storage, but also can give play to the advantages of the column storage database. The combination of the two directly solves the problem of data fragmentation between the OLTP and OLAP databases, reduces development costs and improves resource availability.

[0040] In order to improve the current data storage method, the data processing efficiency is relatively low, and thus the technical problem of being unable to ensure the timeliness of data in the database is solved. This embodiment provides a data processing method, such as Figure 3 As shown, the method includes:

[0041] Step 201: In response to a data processing instruction, obtain incremental data corresponding to a data operation request.

[0042] Exemplarily, data operation requests may include data insertion, update and / or deletion requests, etc. Incremental data refers to the newly added or changed part in a series of data, rather than the sum of all data. When processing a large amount of data, if all data is processed each time, it is not only inefficient, but also may cause inaccurate results due to errors in the data processing process. By only processing incremental data, the speed and accuracy of data processing can be greatly improved. In database synchronization and backup, incremental data can be used to achieve incremental backup and synchronization of data. Synchronizing or backing up only incremental data can save a lot of storage space and network bandwidth.

[0043] For example, when inserting / adding new data, you can directly write the data to the Delta Cache, and then return to the business system request to end. When updating data, you need to write the updated data to the Delta Cache and mark the latest version number. If the row data corresponding to the key exists in the cache, it is directly updated. If the row data does not exist in the cache, a new data is written, that is, the cache always retains only the last update operation for the updated data. When deleting data, if the row data corresponding to the key exists in the Delta Cache, it is deleted, and the key of the deleted data is recorded.

[0044] Step 202: Store the incremental data in the incremental cache database.

[0045] Exemplarily, the cache database includes an incremental cache database and a query cache database, and processing incremental data is more efficient than processing all data. Especially when the amount of data is huge, processing incremental data can greatly save time and resources, and the integrity, consistency and accuracy of the data need to be guaranteed when processing incremental data.

[0046] In some examples, by applying the data storage system provided in this embodiment, the captured incremental data can be cleaned and converted to meet the storage format and requirements of the cache database. The addition, deletion and modification of business data can directly operate the Delta Cache module in the cache module, providing fast row write performance similar to that of a row-based database.

[0047] Step 203: When the amount of incremental data in the incremental cache database reaches a preset threshold, the incremental data in the incremental cache database is flushed to the column storage database and merged with the stock data in the column storage database.

[0048] For example, when the amount of newly added or updated data reaches a preset amount, the incremental data is written to the cache database in a timely manner, and the corresponding cache items are updated. When the incremental data in the cache database is flushed to the column storage database, the inserted / newly added data can be directly written to the column storage database, and the updated data can be merged to overwrite the old version of the column storage database. For deletion operations, the corresponding cache items need to be deleted in the database or marked as invalid. The incremental data corresponding to the deleted data can also be merged to delete the corresponding data in the column storage database.

[0049] In some examples, the incremental data stored in the Delta Cache is flushed to the column-based database after reaching a preset threshold. Figure 4 As shown, during the flush process, the row format will first be changed to the column format (Row change to column), and then merged with the data in the column storage database. When the data is written to the column storage database, it is already updated in the cache. Compared with the Write-Ahead Logging (WAL) of the traditional column storage database, a lot of WAL data has been reduced. For example, the data with key="aaaaa" has been updated 6 times in the cache before flushing, so only the V6 version data of the last update is retained in the cache. During the merge, only the V6 version data is merged with the V0 version data in the column storage database. Compared with the WAL log operation of the traditional column storage database, 5 data compaction operations in the middle are reduced, which greatly improves efficiency. This embodiment achieves a reduction in the amount of data by merging with the cache data, and there is at most one version of data in the cache, and the corresponding read amplification problem is also solved.

[0050] In summary, by storing existing data in a column-based database, the incremental data in the Delta Cache will be flushed to the column-based database when it reaches a certain threshold, and merged with the existing data. The existence of the intermediate cache system reduces the number of compactions of the column-based database, further improving the read and write efficiency of the column-based database.

[0051] Step 204: Transfer the incremental data in the incremental cache database to the query cache database.

[0052] Exemplarily, the incremental data in the Delta Cache is transferred to the Query Cache after being flushed to the column-stored database. By first storing the incremental data in the Delta Cache and then flushing it to the column-stored database in batches after reaching a preset threshold, the pressure on the column-stored database caused by frequent single record update operations is reduced. This allows the column-stored database to focus on large-scale analytical queries rather than processing a large number of transactional updates. When the incremental data is flushed to the column-stored database, the data is also loaded into the Query Cache. For point queries, the Query Cache is equivalent to a cache layer that allows the system to quickly locate and return the required data without accessing the slower column-stored database every time. If the requested data already exists in the Query Cache, the results can be obtained directly from here, greatly improving the response speed.

[0053] Optionally, the method of this embodiment may further include: analyzing data access patterns and using an adaptive replacement cache (ARC) algorithm to determine a cache elimination strategy for the incremental cache database and the query cache database according to the duplication of input data.

[0054] For example, the Delta Cache or Query Cache space can be reasonably allocated, and only the data in the Query Cache can be eliminated. The data in the Delta Cache can only be transferred to the Query Cache after flushing. At the same time, a scenario-aware cache elimination strategy is set for the Query Cache module. By analyzing data access patterns, such as random access, sequential access, and high-frequency access, the characteristics of the current scenario are identified. The ARC algorithm is used, combined with the Least Recently Used (LRU) and Least Frequently Used (LFU) calculation methods. According to the repetition of input data, the LRU area and LFU area size are dynamically allocated, and the cache data is managed in a targeted manner to ensure efficient and reasonable use of cache resources.

[0055] Optionally, the method of this embodiment may also include: determining the cache expiration time corresponding to the data in the incremental cache database and the query cache database based on the data operation frequency and operation heat; and managing the data in the incremental cache database and the query cache database based on the cache expiration time and cache elimination strategy.

[0056] For example, in addition to cache preheating and other means, the reasonable setting of cache expiration time and cache elimination strategy can also further improve the cache hit rate of Query Cache.

[0057] For example, the data in the Delta Cache is important business data, so the cache expiration time of the Delta Cache can be set to never expire. It can only be transferred to the Query Cache module after the flush operation is completed, and then the expiration time can be set in a targeted manner based on the frequency and heat of data operations.

[0058] Compared with the current existing technology, by applying the technical solution of this embodiment, first respond to the data processing instruction to obtain the incremental data corresponding to the data operation request; then store the incremental data in the incremental cache database; when the amount of incremental data in the incremental cache database reaches the preset threshold, the incremental data in the incremental cache database is flushed to the column storage database and merged with the stock data in the column storage database, and then the incremental data in the incremental cache database is transferred to the query cache database. The incremental data generated by the data processing operation is synchronously stored by the cache database and the column storage database, which not only utilizes the fast read and write performance of the row storage provided by the cache database, but also gives play to the advantages of the column storage database. It not only has transaction processing functions but also can support the analysis needs of large amounts of data, improves data processing efficiency, and ensures the real-time consistency of business data and analysis data.

[0059] Furthermore, this embodiment provides a specific method for performing data query in a data storage system, such as Figure 5 As shown, the method includes:

[0060] Step 301: In response to a data query instruction, obtain the type of application request, where the application request includes a point query request and a batch data query request.

[0061] In some examples, point query requests and batch data query requests are two basic types of database query operations. Point query requests refer to query operations for a single record or a small number of records in a database, usually used to obtain specific data under specific conditions; while batch data query requests refer to query operations that retrieve multiple records or a large amount of data from a database at one time, usually used to obtain a set of data that meets specific conditions. In terms of query efficiency, point query requests have clear query targets and high query efficiency, and can use indexes in the data storage system to quickly locate target records; the efficiency of batch data query requests depends on factors such as the amount of data and index usage, and may take longer to complete.

[0062] Step 302: query data in the cache database and / or the column storage database according to the type of application request.

[0063] Exemplarily, when querying data in a database, various factors need to be comprehensively considered, including query type, database characteristics, query efficiency, resource consumption, etc. By selecting an appropriate database and query method, query efficiency and performance can be maximally improved.

[0064] Optionally, the cache database includes an incremental cache database and a query cache database; correspondingly, step 302 may specifically include: if the application request is a point query request, query whether there is corresponding data in the query cache database; if there is data corresponding to the point query request in the query cache database, return the query result; if there is no data corresponding to the point query request in the query cache database, query in the columnar database and store the queried data in the query cache database.

[0065] In some examples, by adding a Query Cache module, optimization of point query requests can be achieved. As Figure 6 shown, if the application request is determined by the query optimizer to be a point query request, first query whether there is data in the Query Cache. If there is, directly return the query result; if not, load the queried data from the columnar database into the cache database.

[0066] Although the point query operation in this embodiment has an additional step of retrieving from the cache in the case of cache miss compared to a single point query operation, overall, as shown in the following formula (1), for the cache hit rate f, the data reading time from the cache is t1, and the data reading and return time from the database in case of cache miss is t2. For the average access time of point query data, when t1 < t2 is known, by increasing the cache hit rate, the point query performance can be further optimized.

[0067] t = f * t1 + (1 - f) * t2 (Formula 1)

[0068] In some examples, the Delta Cache is responsible for receiving incremental data of business operations and, after performing a flush operation, passing this data to the Query Cache module. This embodiment stores data in the cache module based on the principle of data locality, that is, the currently accessed data items are very likely to be accessed again in the near future. Given that the Delta Cache stores the most recently operated data, which has a high probability of being queried again, after the flush operation is completed, this data is transferred to the Query Cache to continue to play its role, thereby effectively increasing the cache hit rate of the Query Cache. Correspondingly, the point query performance is also improved.

[0069] Optionally, step 302 may further specifically include: if the application request is a batch data query request, querying the incremental cache database and the column storage database at the same time, and merging the data obtained from the query.

[0070] For example, during the batch data analysis query process, it is necessary to request to query the data in the Delta Cache and the column storage database at the same time, and then merge the obtained data, that is, merge the added, deleted and modified data corresponding to the data insertion, update and / or deletion operations respectively.

[0071] Compared with the current existing technology, by applying the technical solution of this embodiment, this embodiment first responds to the data query instruction and obtains the type of application request, which includes point query request and batch data query request; then, according to the type of application request, the data is queried in the cache database and / or the column storage database. By directly operating the user's common additions, deletions, and point queries in the cache module, the data is finally stored in the column storage database in the form of column storage, which not only fully utilizes the write efficiency of the cache module, but also gives full play to the large data volume OLAP analysis performance of the column storage database, thereby improving the resource utilization and data processing efficiency of the cluster.

[0072] Further, as Figure 3 and Figure 5 The specific implementation of the method shown in this embodiment provides a data processing device, such as Figure 8 As shown, the device includes: an acquisition module 41, a storage module 42, a processing module 43, and a transfer module 44.

[0073] An acquisition module 41 is configured to acquire incremental data corresponding to a data operation request in response to a data processing instruction;

[0074] A storage module 42, configured to store the incremental data in an incremental cache database;

[0075] The processing module 43 is configured to, when the amount of incremental data in the incremental cache database reaches a preset threshold, flush the incremental data in the incremental cache database to the column storage database and merge it with the stock data in the column storage database;

[0076] The transfer module 44 is configured to transfer the incremental data in the incremental cache database to the query cache database.

[0077] In some examples of this embodiment, the processing module 43 is further configured to analyze data access patterns and utilize an adaptive replacement cache ARC algorithm to determine cache elimination strategies for the incremental cache database and the query cache database according to the duplication of input data.

[0078] In some examples of this embodiment, the processing module 43 is further specifically configured to determine the cache expiration time corresponding to the data in the incremental cache database and the query cache database according to the data operation frequency and the operation heat; and manage the data in the incremental cache database and the query cache database based on the cache expiration time and the cache elimination policy.

[0079] In a specific application scenario, the device further includes: a query module;

[0080] The query module is configured to obtain the type of application request in response to a data query instruction, wherein the application request includes a point query request and a batch data query request; and query data in the cache database and / or the column storage database according to the type of the application request.

[0081] In some examples of this embodiment, the cache database includes an incremental cache database and a query cache database; accordingly, the query module is further configured to query whether there is corresponding data in the query cache database if the application request is a point query request; if the query cache database has data corresponding to the point query request, return the query result; if the query cache database does not have data corresponding to the point query request, perform a query in the column storage database, and store the data obtained from the query in the query cache database.

[0082] In some examples of this embodiment, the query module is further specifically configured to perform queries in the incremental cache database and the column storage database at the same time if the application request is a batch data query request, and merge the data obtained from the query.

[0083] It should be noted that the data processing device provided in this embodiment and other corresponding descriptions of the functional units involved can be referred to in Figure 3 and Figure 5 The corresponding description in will not be repeated here.

[0084] Based on the above Figure 3 and Figure 5 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figure 3 and Figure 5 The method shown.

[0085] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0086] Based on the above Figure 3 and Figure 5 The method shown, and Figure 8 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, such as a personal computer, a server, a laptop computer, an intelligent robot or other intelligent terminal, which includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 3 and Figure 5 The method shown.

[0087] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0088] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0089] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0090] Through the description of the above implementation methods, the technical personnel in this field can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware. By applying the scheme of this embodiment, compared with the current existing technology, this embodiment is based on the cache database and the column storage database in the data storage system to synchronously store the incremental data generated by the data processing operation, which not only uses the cache module to provide the performance of fast reading and writing of row storage, but also can give play to the advantages of the column storage database, not only has the transaction processing function but also can support the analysis requirements of large amounts of data, ensures the real-time consistency of business data and analysis data, reduces costs and improves resource utilization. And the data storage system provided by this embodiment not only uses the cache module to provide the performance of fast reading and writing of row storage, but also uses the column storage module to give play to the advantages of the column storage database. The combination of the two directly solves the problem of data fragmentation of the two sets of OLTP and OLAP databases, reduces development costs and improves resource availability. The scheme of this embodiment is a low-cost, feasible and easy-to-promote technology, with extremely high practical value, and has a very broad market prospect and a wide range of market applications.

[0091] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0092] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. A data processing method, characterized in that: include: In response to the data processing instruction, obtaining incremental data corresponding to the data operation request; Storing the incremental data in an incremental cache database; When the amount of incremental data in the incremental cache database reaches a preset threshold, the incremental data in the incremental cache database is flushed to the column storage database and merged with the stock data in the column storage database; The incremental data in the incremental cache database is transferred to the query cache database.

2. The method according to claim 1, characterized in that The method further comprises: By analyzing the data access pattern and using the adaptive replacement cache ARC algorithm, the cache elimination strategy for the incremental cache database and the query cache database is determined according to the duplication of the input data.

3. The method according to claim 2, characterized in that The method further comprises: Determine cache expiration time corresponding to the data in the incremental cache database and the query cache database according to the data operation frequency and operation heat; Based on the cache expiration time and the cache elimination policy, the data in the incremental cache database and the query cache database are managed.

4. The method according to claim 1, characterized in that: The method further comprises: In response to the data query instruction, obtaining a type of application request, wherein the application request includes a point query request and a batch data query request; According to the type of the application request, data is queried in the cache database and / or the column storage database.

5. The method according to claim 4, characterized in that The querying of data in the cache database and / or the column storage database according to the type of the application request includes: If the application request is a point query request, querying whether there is corresponding data in the query cache database; If the query cache database has data corresponding to the point query request, the query result is returned; If the query cache database does not have data corresponding to the point query request, a query is performed in the column storage database, and the data obtained by the query is stored in the query cache database.

6. The method according to claim 5, characterized in that The querying of data in the cache database and / or the column storage database according to the type of the application request includes: If the application request is a batch data query request, a query is performed in the incremental cache database and the column storage database at the same time, and the data obtained from the query is merged.

7. A data processing device, characterized in that: include: An acquisition module is configured to acquire incremental data corresponding to a data insertion, update and / or deletion request in response to a data processing instruction; A storage module, configured to store the incremental data in an incremental cache database; A processing module is configured to, when the amount of incremental data in the incremental cache database reaches a preset threshold, flush the incremental data in the incremental cache database to the column storage database and merge it with the stock data in the column storage database; The transfer module is configured to transfer the incremental data in the incremental cache database to the query cache database.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product having a computer program stored thereon, characterized in that: When the computer program product is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images