Data processing method, device, medium and electronic equipment

Abstract

The present application belongs to the field of computer technology, and specifically relates to a data processing method, a data processing device, a computer-readable medium, an electronic device, and a computer program product. The method includes: splitting a data query request into a first sub-request corresponding to a first storage system and a second sub-request corresponding to a second storage system; the first storage system and the second storage system have different system configurations, and the system configuration includes the distribution ratio of computing tasks between the computing layer and the storage layer; executing the first sub-request on the first storage system to obtain a first query result, and executing the second sub-request on the second storage system to obtain a second query result; aggregating the first query result and the second query result to obtain the query result of the data query request. The present application can improve the resource allocation balance of the database system.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a data processing method, a data processing device, a computer-readable medium, an electronic device, and a computer program product. Background Art

[0002] In traditional database systems, in order to improve data processing efficiency, complete data operation functions such as query, calculation, and writing are usually integrated on a single computer node. However, for different data processing businesses, the requirements for computing resources and storage resources in various operation links often vary greatly, making it difficult for database systems to evenly configure system resources.

[0003] For example, some data processing businesses have a large demand for computing resources but a small demand for storage resources. When the computing resources are working at saturation, the storage resources may be idle and wasted. For another example, some data processing businesses have a small demand for computing resources but a large demand for storage resources. When the storage resources are working at saturation, the computing resources may be idle and wasted.

[0004] Therefore, how to evenly configure the system resources of the database system is an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present application is to provide a data processing method, a data processing device, a computer-readable medium, an electronic device, and a computer program product, aiming to improve the resource allocation balance of a database system.

[0006] According to one aspect of an embodiment of the present application, a data processing method is provided, the method comprising: splitting a data query request into a first sub-request corresponding to a first storage system and a second sub-request corresponding to a second storage system; the first storage system and the second storage system have different system configurations, the system configurations including the distribution ratio of computing tasks between a computing layer and a storage layer; executing the first sub-request on the first storage system to obtain a first query result, and executing the second sub-request on the second storage system to obtain a second query result; aggregating the first query result and the second query result to obtain a query result of the data query request.

[0007] According to one aspect of an embodiment of the present application, a data processing device is provided, the device comprising:

[0008] a splitting module configured to split a data query request into a first sub-request corresponding to a first storage system and a second sub-request corresponding to a second storage system; the first storage system and the second storage system have different system configurations, and the system configurations include a distribution ratio of computing tasks between a computing layer and a storage layer;

[0009] an execution module, configured to execute the first sub-request on the first storage system to obtain a first query result, and execute the second sub-request on the second storage system to obtain a second query result;

[0010] The aggregation module is configured to aggregate the first query result and the second query result to obtain a query result of the data query request.

[0011] In some embodiments of the present application, based on the above technical solution, the splitting module may further include:

[0012] A request parsing module is configured to parse the data query request to obtain a data time series range carried in the data query request, where the data time series range is used to represent a distribution range of the data to be queried in a time series;

[0013] a time series splitting module, configured to split the data time series range into a first time interval corresponding to the first storage system and a second time interval corresponding to the second storage system according to the time series capacity of the first storage system, wherein the time series capacity is used to represent a maximum time range of the time series data stored on the first storage system;

[0014] The request splitting module is configured to split the data query request into a first sub-request with the first time interval as the data timing range and a second sub-request with the second time interval as the data timing range.

[0015] In some embodiments of the present application, based on the above technical solution, in the system configuration of the first storage system, the distribution proportion of computing tasks in the computing layer is smaller than that in the storage layer; the execution module can be further configured as: in the computing layer of the first storage system, splitting the first sub-request into multiple computing tasks; in the storage layer of the first storage system, multiple storage nodes respectively query the original data corresponding to each of the computing tasks, and perform computational processing on the original data according to the computing tasks to obtain query result shards; in the computing layer of the first storage system, a plurality of the query result shards are aggregated to obtain a first query result.

[0016] In some embodiments of the present application, based on the above technical solution, the storage layer of the first storage system includes multiple logical shards for storing different data, the logical shards include multiple storage nodes, and the multiple storage nodes located in the same logical shard are used to store the same data; the execution module can be further configured to: in the storage layer of the first storage system, send the multiple computing tasks to the multiple logical shards respectively; within the logical shards, select the target storage node according to the operating status of each of the storage nodes, and query the original data corresponding to the computing task on the target storage node.

[0017] In some embodiments of the present application, based on the above technical solution, in the system configuration of the second storage system, the distribution ratio of computing tasks on the computing layer is greater than that on the storage layer; the execution module can be further configured to: in the storage layer of the second storage system, query the original data corresponding to the second sub-request; in the computing layer of the second storage system, calculate and process the original data according to the second sub-request to obtain a second query result.

[0018] In some embodiments of the present application, based on the above technical solution, the computing layer of the second storage system includes multiple query pools corresponding to different query types, the query pool includes multiple computing nodes, and the multiple computing nodes located in the same query pool are used to process data query requests with the same query type; the execution module can be further configured as: in the computing layer of the second storage system, the original data is sent to the target query pool according to the query type of the second sub-request; in the target query pool, the target computing node is selected according to the operating status of each of the computing nodes, and the original data is calculated and processed on the target computing node to obtain the second query result.

[0019] In some embodiments of the present application, based on the above technical solution, the storage layer of the first storage system includes multiple logical slices for storing different data, the logical slices include multiple storage nodes, and the multiple storage nodes in the same logical slice are used to store the same data; the device also includes:

[0020] A creation module, configured to create, in the computing layer of the first storage system, a management process corresponding to each of the logical slices, the management process being used to synchronize data between the first storage system and the second storage system;

[0021] an acquisition module, configured to acquire a write-ahead log corresponding to the storage node from the logical shard according to the management process, wherein the write-ahead log is used to store an operation record for the storage node;

[0022] A generating module, configured to generate a data update file according to the write-ahead log, wherein the data update file includes data update content on the storage node;

[0023] The saving module is configured to transmit the data update file to the second storage system, and update the data object saved in the second storage system according to the data update file.

[0024] In some embodiments of the present application, based on the above technical solution, the computing layer of the first storage system includes multiple access nodes for assigning tasks to the logical shards; the creation module is further configured to: obtain the node type of each of the access nodes, the node type including a read-write node or a management node; the read-write node is used to assign read-write tasks for performing data reading operations or data writing operations to the logical shards, and the management node is used to assign management tasks for performing data synchronization operations to the logical shards; select a management node from the multiple access nodes according to the node type, and create a management process corresponding to each of the logical shards in the management node.

[0025] In some embodiments of the present application, based on the above technical solution, the computing layer of the first storage system includes multiple access nodes for assigning tasks to the logical slices; the creation module is further configured to: create management processes corresponding to each of the logical slices in the access nodes; obtain the resource occupancy of all management processes in the access nodes, and the resource occupancy is used to indicate the usage ratio of the computing resources and storage resources of the first storage system by the management process; when the resource occupancy is greater than a preset ratio threshold, configure some management processes in the access nodes to a waiting state.

[0026] In some embodiments of the present application, based on the above technical solution, the generation module is further configured to: decode the write-ahead log to obtain the operation type of each log entry in the write-ahead log, where the operation type includes a write operation or a non-write operation; and generate a data update file for the log entry of the write operation according to the operation type in the write-ahead log.

[0027] In some embodiments of the present application, based on the above technical solution, the device also includes an update module, which is configured to: obtain an identification interval of the data update file, the identification interval is used to represent the identification numbers of all log entries corresponding to the data update file; update a database directory of the second storage system according to the identification interval, the database directory is used to store metadata of data objects in the second storage system; when the data objects in the second storage system and the database directory are updated successfully, mark the identification interval as a committed state; and remove log entries in the write-ahead log for which data updates have been completed according to the identification interval in the committed state.

[0028] In some embodiments of the present application, based on the above technical solution, the computing layer of the first storage system includes multiple access nodes for running the management process; the update module is also configured to: identify the running status of the management process in each of the access nodes; when the running status of the management process is a failure, transfer the management process from the access node where it is located to another access node in the computing layer; extract log entries of incomplete data updates from the write-ahead log according to the identification interval in the committed state; and update the data objects and the database directory in the second storage system according to the log entries of the incomplete data updates.

[0029] In some embodiments of the present application, based on the above technical solution, the device also includes an aggregation module, which is configured to: obtain metadata of the data object from the database directory, and identify the identification interval corresponding to each of the data update files based on the metadata; mark multiple data update files with consecutive identification numbers as files to be aggregated according to the identification interval; download the files to be aggregated from the storage layer of the second storage system according to a preset merging period, and aggregate the files to be aggregated to obtain aggregated files; upload the aggregated files to the storage layer of the second storage system, and delete the files to be aggregated from the storage layer of the second storage system after updating the database directory.

[0030] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data processing method in the above technical solution is implemented.

[0031] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement a data processing method as in the above technical solution.

[0032] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the data processing method in the above technical solution is implemented.

[0033] In the technical solution provided in the embodiment of the present application, by splitting the data query request into a first sub-request corresponding to the first storage system and a second sub-request corresponding to the second storage system, the first sub-request can be executed on the first storage system to obtain the first query result, and the second sub-request can be executed on the second storage system to obtain the second query result; then the first query result and the second query result are aggregated to obtain the query result of the data query request. Among them, the first storage system and the second storage system have different system configurations, and the system configuration includes the distribution ratio of computing tasks between the computing layer and the storage layer. The embodiment of the present application concentrates computing tasks on the computing layer in one storage system and concentrates computing tasks on the storage layer in another storage system, thereby forming a complementary relationship by utilizing the first storage system and the second storage system with different system configurations, realizing a balanced configuration of computing resources and storage resources, and improving the resource utilization of the database storage system.

[0034] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 The exemplary system architecture block diagram applying the technical solution of the present application is schematically shown.

[0037] Figure 2 The figure shows an architectural block diagram of a storage system in the related technology of the present application.

[0038] Figure 3 A flow chart of a data processing method for performing a data query operation in one embodiment of the present application is shown.

[0039] Figure 4 A flowchart of splitting a data query request in one embodiment of the present application is shown.

[0040] Figure 5 A flowchart of executing a first sub-request on a first storage system in one embodiment of the present application is shown.

[0041] Figure 6 A flowchart of executing a second sub-request on a second storage system in one embodiment of the present application is shown.

[0042] Figure 7 A flow chart of a data processing method for performing storage system data synchronization in one embodiment of the present application is shown.

[0043] Figure 8 A schematic diagram of a first distribution method of a management process on a first storage system in an application scenario according to an embodiment of the present application is shown.

[0044] Fig. 9 A schematic diagram of a second distribution method of management processes on a first storage system in an application scenario according to an embodiment of the present application is shown.

[0045] Fig.10 A schematic diagram showing the principle of performing data synchronization in an application scenario according to an embodiment of the present application is shown.

[0046] Fig.11 The following is a block diagram showing the architecture of a storage system in an application scenario according to an embodiment of the present application.

[0047] Fig.12 The structural block diagram of the data processing device provided in the embodiment of the present application is schematically shown.

[0048] Fig.13 The structure block diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0049] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0050] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0051] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0052] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0053] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0054] Figure 1 The exemplary system architecture block diagram applying the technical solution of the present application is schematically shown.

[0055] like Figure 1 As shown, the system architecture may include a terminal device 110 , a first storage system 120 , and a second storage system 130 .

[0056] The terminal device 110 may include various electronic devices such as smart phones, tablet computers, laptops, desktop computers, smart speakers, smart wearable devices, smart car devices, smart payment terminals, etc.

[0057] The first storage system 120 and the second storage system 130 may be servers for providing data storage services, for example, they may be independent physical servers, or they may be server clusters or distributed systems composed of multiple physical servers, or they may be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0058] The terminal device 110, the first storage system 120 and the second storage system 130 perform data communication with each other through a network, and the network may include communication media of various connection types, such as a wired communication link or a wireless communication link.

[0059] According to the implementation requirements, the system architecture in the embodiment of the present application can have any number of terminal devices, networks and servers. In addition, the technical solution provided in the embodiment of the present application can be applied to the terminal device, can also be applied to the server, or can be implemented by the terminal device and the server together, and the present application does not make any special restrictions on this.

[0060] In the technical solution provided in the embodiment of the present application, the first storage system 120 and the second storage system 130 have different system configurations, and the system configurations include the distribution ratio of computing tasks between the computing layer and the storage layer. For example, in the embodiment of the present application, computing tasks can be concentrated on the storage layer in the first storage system, and computing tasks can be concentrated on the computing layer in the second storage system, so that the first storage system and the second storage system can form a complementary relationship, realize the balanced configuration of computing resources and storage resources, and improve the resource utilization of the database storage system.

[0061] To facilitate the explanation of the research and development ideas of the technical solution of this application, the following briefly introduces the storage technology related to this application.

[0062] Figure 2 The figure shows an architectural block diagram of a storage system in the related technology of the present application.

[0063] like Figure 2 As shown in Figure 1, the basic architecture of the storage system is generally divided into two layers, namely the computing layer and the storage layer. The computing layer includes multiple access nodes, while the storage layer includes multiple storage nodes. According to different functions, storage nodes can be divided into the master node leader and the slave node follower.

[0064] The master node is responsible for handling all read and write requests to maintain data consistency and integrity. In a distributed database, the master node is usually responsible for accepting write operations from clients and ensuring that these changes are correctly replicated to other nodes.

[0065] The slave node maintains data consistency by replicating the data of the master node. The slave node usually only handles read requests in order to share the load of the master node and provide an additional data access point. In some configurations, the slave node can also handle write requests, but these write requests ultimately need to be synchronized through the master node.

[0066] By storing data copies on multiple slave nodes, data reliability and fault tolerance can be improved. If the master node fails, failover can be achieved by electing a slave node to become the new master node. In addition, the performance and scalability of the database can be improved by separating read and write operations to different nodes.

[0067] In the operation link of the storage system performing data write operations, the user write request is sent to any access node through load balancing, and is written to the corresponding storage node through the routing strategy. The data is actually stored on the local disk.

[0068] In the operation link of the storage system performing data reading operations, the access node splits the query plan and distributes the subtasks to the storage nodes for execution. Multiple storage nodes execute queries simultaneously, consuming the computing resources and storage resources of the storage nodes. When the subtask is completed, the access node is replied to, and the access node performs secondary aggregation on the data and returns it to the user.

[0069] The storage system uses query pushdown to move part of the settlement process down to the storage node for execution, using more nodes to participate in the calculation, so as to quickly return the results. When the number of storage nodes is sufficient, a single query can be calculated by hundreds of nodes at the same time, and the access node performs small-scale data aggregation tasks, making the single query performance extremely high.

[0070] However, such a system architecture makes the storage node have the function of computing. The storage node carries the operation process from raw data to aggregated results, resulting in the integration of storage and computing in a single node. Therefore, the storage node is not only responsible for the storage of physical data, but also for the main data computing. Different businesses have different requirements for storage resources and computing resources. Such an architecture makes it impossible to expand computing / storage independently, which in turn leads to low resource utilization and waste of resources.

[0071] In addition, such a system architecture concentrates the consumption of query operations and write operations on the storage layer. When large-scale queries occur, storage layer resources will be occupied, and in extreme cases, data writing will be affected.

[0072] At present, some cloud time series databases have begun to use the original storage and computing separation architecture to solve the problem of low resource utilization. The specific idea is to introduce external storage systems for the storage layer, such as object storage, file storage and other storage services. Computing nodes are divided into different functions, responsible for writing, querying, computing, etc., which naturally achieves storage and computing separation, and can achieve the characteristics of querying without affecting writing, and separation of large queries and small queries. However, because the storage module of the key link is too dependent on the external system, the upper limit of its own products is strongly bound to the external storage product, the reliability depends on the peripheral module, and the throughput of object storage / file storage is limited compared with the physical solid-state hard disk, which makes the performance of such an architecture not as strong as the traditional architecture.

[0073] In response to the defects of the solutions in the above application scenarios, the inventors of the present application discovered during the research process that in time series data queries, queries with higher freshness have the characteristics of large number and high real-time requirements; while historical queries have the characteristics of small number and low real-time requirements. Utilizing the above query characteristics, the embodiment of the present application designs a reliable and high-performance cloud-native time series database computing separation solution. The present application innovatively proposes a storage architecture in which a first storage system and a second storage system complement each other's resources, and uses local storage and near real-time hot backup to provide storage and computing separation, high performance, and reliable time series database services.

[0074] For example, a local cloud time series database is used as the first storage system, and an external storage system is used as the second storage system.

[0075] In the local cloud time series database part, user-written data is stored in real time and provides real-time query capabilities, combining storage and computing. Computing nodes support custom query media based on time division, such as querying data cache time series database within the past day, which is responsible for providing high-performance queries with high freshness, and the fault domain is isolated from the second storage system. When the second storage system fails, it will not affect the user's full query, greatly improving reliability.

[0076] In the external storage system part, the time series database computing node aggregates data files in near real time and uploads them to the external storage system. The computing node allows to obtain raw data from the external storage medium and perform calculations locally, without relying on the time series database storage node. This not only achieves storage and computing separation, but also separates data read requests from write requests. In addition, when the external storage medium fails, the query can be switched to the local cloud time series database without affecting user availability.

[0077] The following describes in detail the technical solutions such as the data processing method, data processing device, computer-readable medium, electronic device, and computer program product provided by the present application in conjunction with specific implementation methods.

[0078] Figure 3 A flow chart of a data processing method for performing a data query operation in an embodiment of the present application is shown. The data processing method can be executed by a terminal device or a server, or can be executed by a terminal device and a server together. The present application embodiment takes the data processing method executed by a server as an example for explanation.

[0079] like Figure 3 As shown, the data processing method for executing a data query operation may include the following steps S310 to S330.

[0080] S310: Split the data query request into a first sub-request corresponding to the first storage system and a second sub-request corresponding to the second storage system; the first storage system and the second storage system have different system configurations, and the system configurations include a distribution ratio of computing tasks between the computing layer and the storage layer;

[0081] S320: Execute the first sub-request on the first storage system to obtain a first query result, and execute the second sub-request on the second storage system to obtain a second query result;

[0082] S330: Aggregate the first query result and the second query result to obtain a query result of the data query request.

[0083] In the data processing method for executing data query operations provided in an embodiment of the present application, for the first storage system and the second storage system, by configuring different computing task distribution ratios between the computing layer and the storage layer, the computing tasks can be concentrated on the computing layer in one storage system, and on the storage layer in another storage system. In this way, the first storage system and the second storage system with different system configurations can be used to form a complementary relationship, achieve balanced configuration of computing resources and storage resources, and improve resource utilization of the database storage system.

[0084] The following is a detailed description of the implementation scheme of each step of the data processing method in the embodiment of the present application in combination with a specific application scenario.

[0085] Figure 4 FIG. 1 is a flowchart showing a splitting process for a data query request in one embodiment of the present application. Figure 4 As shown, based on the above embodiment, the data query request is split into a first sub-request corresponding to the first storage system and a second sub-request corresponding to the second storage system in step S310, which may further include the following steps S311 to S313.

[0086] S311: Parse the data query request to obtain the data time series range carried in the data query request. The data time series range is used to represent the distribution range of the data to be queried in the time series.

[0087] When the embodiment of the present application is executed by a server, the data query request may be, for example, an HTTP request sent by a client through a network. The data query request includes parameters required for the query, such as an API endpoint, query parameters, and the like.

[0088] By parsing the data query request, key parameters can be extracted from it, which define the specific requirements of the query. For example, the query may require specific fields, filtering conditions, sorting methods, etc.

[0089] According to the extracted key parameters, the data time series range is identified. The data time series range defines the time window of the query, including the start time and the end time. The start point and the end point of the data time series range can be a date and time string in ISO8601 format.

[0090] In one embodiment of the present application, the validity of the identified data time series range can be verified to ensure that the data time series range has a reasonable time point and complies with the query restrictions of the system, such as the maximum time span restriction.

[0091] S312: Split the data time series range into a first time interval corresponding to the first storage system and a second time interval corresponding to the second storage system according to the time series capacity of the first storage system, where the time series capacity is used to represent the maximum time range of the time series data stored on the first storage system.

[0092] The entire data time series range can be divided into two time periods according to the time series capacity of the first storage system, wherein the time series range within the time series capacity of the first storage system is divided into a first time interval, and the time series range beyond the time series capacity of the first storage system is divided into a second time interval.

[0093] For example, in an IoT system of a smart city, it is necessary to store and query time series data from various sensors such as traffic, environment, and public facilities. These data are characterized by huge amounts of data and continuous growth.

[0094] Based on the solution provided by the embodiment of the present application, the first storage system can be used to store recent data and provide a fast access function; the second storage system can be used to store historical data with a lower access frequency. On this basis, the time series capacity of the first storage system can be configured to 1 day, that is, the first storage system only saves the time series data written in the last 1 day.

[0095] When the system receives a data query request, it is learned through analysis that the intention of the data query request is to query the traffic flow data in the past 7 days, that is, the data time series range carried in the data query request is a time window with 7 days ago as the start time and the current time as the end time.

[0096] Therefore, after splitting the data time series range according to the time series capacity of the first storage system, the first time interval is the most recent day, and the corresponding second time interval is the previous 6 days.

[0097] S313: Split the data query request into a first sub-request with the first time interval as the data time series range and a second sub-request with the second time interval as the data time series range.

[0098] The first sub-request uses the first time interval as the data time series range, and the first sub-request will use the high throughput and fast access capabilities of the first storage system to achieve an immediate query response to the most recent data. The second sub-request uses the second time interval as the data time series range, and this interval covers time series data that exceeds the capacity of the first storage system. The second sub-request is directed to the second storage system, so that historical data can be queried and processed in the second storage system configured to optimize storage costs and long-term data retention.

[0099] Continuing with the above application scenario as an example, the purpose of the first sub-request is to query the time series data written in the last day on the first storage system, while the purpose of the second sub-request is to query the time series data written in the previous 6 days on the second storage system.

[0100] In the embodiment of the present application, by adopting the splitting strategy for data query requests, not only can the efficiency of data query be ensured, but also different storage system resources can be reasonably allocated according to the timeliness and access mode of data to achieve cost-effectiveness and performance optimization. At the same time, this strategy also supports high scalability and flexibility, allowing the system to dynamically adjust the generation and processing logic of sub-requests according to changes in data growth and query requirements.

[0101] Figure 5 A flowchart of executing a first sub-request on a first storage system in one embodiment of the present application is shown.

[0102] In an embodiment of the present application, in the system configuration of the first storage system, the distribution ratio of computing tasks on the computing layer is smaller than that on the storage layer. For example, the first storage system distributes the main computing logics such as conditional filtering operations, database projections (such as operations to select specific columns), data table connection operations, and data aggregation operations (such as aggregate functions such as sum and count) to the storage layer for execution, while the computing layer only performs the final data aggregation and data return operations based on the data query results of the storage layer. By transferring part of the data analysis or processing tasks from the computing layer to the storage layer for execution, the data processing efficiency can be improved and the load of network transmission can be reduced.

[0103] like Figure 5 As shown, based on the above embodiment, executing the first sub-request on the first storage system to obtain the first query result in step S320 may further include the following steps S321 to S323.

[0104] S321: In the computing layer of the first storage system, split the first sub-request into multiple computing tasks.

[0105] In one embodiment of the present application, the storage layer of the first storage system may include multiple logical shards for storing different data. Based on this, the first sub-request may be split into multiple computing tasks corresponding to the multiple logical shards in the computing layer of the first storage system, and each logical shard is used to execute one computing task.

[0106] Logical Sharding is a database partitioning strategy that divides a large database into smaller, more manageable parts that are logically independent but may still be physically located on the same database server or cluster. Each logical shard can be maintained, backed up, and restored independently, thereby improving the performance, scalability, and availability of the database.

[0107] S322: In the storage layer of the first storage system, multiple storage nodes respectively query the original data corresponding to each computing task, and perform computational processing on the original data according to the computing task to obtain query result shards.

[0108] In one embodiment of the present application, the storage layer of the first storage system includes multiple logical slices for storing different data, the logical slices include multiple storage nodes, and the multiple storage nodes in the same logical slice are used to store the same data. On this basis, in the storage layer of the first storage system, multiple storage nodes respectively query the original data corresponding to each computing task, which can further include: in the storage layer of the first storage system, multiple computing tasks are respectively sent to multiple logical slices; in the logical slices, a target storage node is selected according to the operating status of each storage node, and the original data corresponding to the computing task is queried on the target storage node.

[0109] Logical sharding distributes data to different database instances or tables based on specific rules or keys (such as user ID, date range, etc.). The computing layer can route computing tasks to the correct logical shard based on the data's sharding key. By distributing computing tasks to multiple logical shards, it helps achieve load balancing and avoid overloading a single database instance.

[0110] S323: In the computing layer of the first storage system, aggregating the multiple query result shards to obtain a first query result.

[0111] The embodiment of the present application can achieve efficient data management and optimized resource utilization by implementing a logical sharding strategy in the storage layer of the first storage system and intelligently allocating multiple computing tasks to different logical shards. In each logical shard, the system dynamically selects the most suitable target storage node based on the operating status and performance indicators of the current storage node to ensure that data queries and computing tasks can be executed on the node with the best performance, thereby improving the response speed, concurrent processing capabilities and stability of the overall system. This method not only enhances the scalability and flexibility of the system, but also reduces the risk of single point failures through intelligent load balancing, and improves the overall efficiency and reliability of data processing.

[0112] Figure 6 A flowchart of executing a second sub-request on a second storage system in one embodiment of the present application is shown.

[0113] In the embodiment of the present application, in the system configuration of the second storage system, the distribution ratio of computing tasks on the computing layer is greater than that on the storage layer. For example, the second storage system retains the main computing logics such as conditional filtering operations, database projection (such as operations to select specific columns), data table connection operations, data aggregation operations (such as aggregate functions such as sum and count), and final data aggregation and data return operations in the computing layer, while the storage layer only performs operations to query the original data according to the request of the computing layer.

[0114] like Figure 6 As shown, based on the above embodiment, executing the second sub-request on the second storage system to obtain the second query result in step S320 may further include the following steps S324 to S325.

[0115] S324: Querying the original data corresponding to the second sub-request in the storage layer of the second storage system;

[0116] S325: In the computing layer of the second storage system, computing and processing are performed on the original data according to the second sub-request to obtain a second query result.

[0117] The embodiment of the present application optimizes the configuration of the second storage system and sets the proportion of computing tasks in the computing layer to be greater than that in the storage layer, thereby effectively improving the efficiency and flexibility of data processing. Specifically, the second storage system concentrates the main computing logics such as conditional filtering, database projection, data table connection, and data aggregation on the computing layer, while the storage layer focuses on responding to computing layer requests and efficiently querying and providing raw data. This design not only reduces the burden on the storage layer and ensures a fast response to data queries, but also enables the computing layer to flexibly handle complex data operations, thereby improving the performance and data processing capabilities of the overall system. At the same time, this separate architecture is also easy to maintain and expand, providing a solid foundation for the long-term development of the system and adapting to changing business needs.

[0118] In one embodiment of the present application, the computing layer of the second storage system includes multiple query pools corresponding to different query types, and the query pool includes multiple computing nodes. The multiple computing nodes in the same query pool are used to process data query requests with the same query type.

[0119] For example, an embodiment of the present application can divide multiple query types according to parameters such as the subject type of the business entity that initiates the data query request and the data query volume. Each query type can have a different query priority. Accordingly, an independent query pool can be configured for each query type, and each query pool is isolated from each other.

[0120] On this basis, the original data is calculated and processed according to the second sub-request to obtain the second query result, which can further include: in the computing layer of the second storage system, the original data is sent to the target query pool according to the query type of the second sub-request; in the target query pool, the target computing node is selected according to the operating status of each computing node, and the original data is calculated and processed on the target computing node to obtain the second query result.

[0121] Take an e-commerce platform application scenario as an example. The platform needs to process a large number of data query requests from different business entities, including but not limited to customer order query, inventory status query, financial report generation, etc. By parsing the data query request, the type of business entity initiating the query request can be identified, such as retailers, wholesalers, or internal employees.

[0122] Depending on the business entity type and query characteristics, multiple query types can be defined. For example, retailers may initiate large-scale order status queries, wholesalers may need to perform batch queries on inventory levels, and internal employees may perform detailed analytical queries on financial data. Each query type is assigned a different query priority based on business importance and urgency. For example, financial report generation may have the highest priority because it is critical to decision making.

[0123] An independent query pool is configured for each query type. Each query pool contains multiple computing nodes that are dedicated to processing query requests of that type. Query pools are isolated from each other to ensure resource specificity and processing efficiency.

[0124] When a data query request arrives, the system distributes it to the corresponding query pool based on the requested business subject type and query characteristics. In the target query pool, the most suitable computing node is selected to process the query request based on the current operating status and load of the computing node. The selected computing node performs the required computing processing on the original data, such as condition filtering, data aggregation, etc., to generate the query results.

[0125] The embodiment of the present application achieves efficient processing and optimized resource allocation of data query requests by setting multiple query pools for different query types in the computing layer of the second storage system, each query pool containing multiple computing nodes. This design allows the system to intelligently direct the original data to the corresponding query pool according to the query type of the second sub-request, and then select the most suitable computing node in the pool to perform data calculation processing according to the real-time operating status of the computing node. This method not only improves the parallelism and throughput of data processing, but also ensures the rapid response and high reliability of query tasks through load balancing and node status monitoring, thereby obtaining accurate and timely second query results. In addition, this architecture also facilitates the scalability and maintainability of the system, allowing the system to dynamically adjust the configuration of query pools and computing nodes according to changes in query load.

[0126] In one embodiment of the present application, in order to maintain data consistency between the first storage system and the second storage system, data synchronization may be performed between the first storage system and the second storage system.

[0127] Figure 7 A flow chart of a data processing method for performing data synchronization of a storage system in an embodiment of the present application is shown. The data processing method can be performed by a terminal device or a server, or can be performed by a terminal device and a server together. The present application embodiment takes the data processing method performed by a server as an example for explanation.

[0128] In an embodiment of the present application, the storage layer of the first storage system includes multiple logical slices for storing different data, the logical slices include multiple storage nodes, and the multiple storage nodes located in the same logical slice are used to store the same data.

[0129] like Figure 7 As shown, the data processing method for performing storage system data synchronization may include the following steps S710 to S740.

[0130] S710: In the computing layer of the first storage system, a management process corresponding to each logical slice is created, where the management process is used to synchronize data between the first storage system and the second storage system.

[0131] S720: Obtain a write-ahead log corresponding to the storage node from the logical shard according to the management process, where the write-ahead log is used to store operation records for the storage node.

[0132] S730: Generate a data update file according to the write-ahead log, where the data update file includes data update content on the storage node.

[0133] S740: Transmit the data update file to the second storage system, and update the data object stored in the second storage system according to the data update file.

[0134] The embodiment of the present application achieves efficient data synchronization and consistency maintenance by creating a management process corresponding to each logical shard in the computing layer of the first storage system. The management process is responsible for monitoring and collecting the pre-write logs in the logical shards, which record in detail all operations on the storage nodes. Subsequently, the system generates data update files based on these pre-write logs, which contain the data content that needs to be updated on the storage nodes. By transferring the data update files to the second storage system and applying these updates, the data objects in the second storage system are updated synchronously, ensuring the data consistency and integrity between the two storage systems. This method not only improves the efficiency of data management, but also enhances the reliability and fault tolerance of the system, allowing the system to recover quickly in the face of local failures while maintaining the real-time and accuracy of the data.

[0135] Figure 8 A schematic diagram of a first distribution method of a management process on a first storage system in an application scenario according to an embodiment of the present application is shown.

[0136] like Figure 8 As shown, the storage layer of the first storage system is composed of a number of logical shards, each of which includes a number of storage nodes (including master nodes and slave nodes) composed of a consistency protocol, and each storage node generates a corresponding write-ahead log after performing a data processing operation.

[0137] The minimum operation granularity of the management process in the embodiment of the present application is a logical slice. As shown in the figure, it is assumed that there are two access nodes in the computing layer of the first storage system, and two management processes are created on each access node.

[0138] The access node has a status pull interface, which is used to display the running status of all current management processes. By regularly collecting the running status of the management processes on the access node, the scheduling and allocation of system resources such as memory, CPU, and distribution dimensions can be performed.

[0139] In one embodiment of the present application, creating a management process corresponding to each logical slice may further include: creating a management process corresponding to each logical slice in an access node; obtaining the resource occupancy of all management processes in the access node, the resource occupancy being used to indicate the proportion of computing resources and storage resources used by the management process for the first storage system; when the resource occupancy is greater than a preset proportion threshold, configuring some management processes in the access node to be in a waiting state.

[0140] For example, in order to save costs, the management process runs together with the normal read and write process, and flow control rules are added to the management process. When the processor or memory usage exceeds the 20% ratio threshold, the flow limit is triggered to avoid affecting the normal read and write operations due to the execution of the management process.

[0141] In another embodiment of the present application, creating a management process corresponding to each logical shard may further include: obtaining the node type of each access node, the node type includes a read-write node or a management node; the read-write node is used to assign read-write tasks for performing data reading operations or data writing operations to the logical shards, and the management node is used to assign management tasks for performing data synchronization operations to the logical shards; selecting a management node from multiple access nodes according to the node type, and creating a management process corresponding to each logical shard in the management node.

[0142] Fig. 9 A schematic diagram of a second distribution method of management processes on a first storage system in an application scenario according to an embodiment of the present application is shown.

[0143] like Fig. 9 As shown, by configuring two types of access nodes, read-write nodes and management nodes, physical isolation of read-write nodes and management nodes can be achieved, that is, the management process can be run on a separate management node and separated from the read-write processes in the read-write nodes.

[0144] Fig.10 FIG. 1 is a schematic diagram showing the principle of executing data synchronization in an application scenario according to an embodiment of the present application. Fig.10 As shown, in the computing layer of the first storage system, the process of performing data synchronization with the second storage system through the management process includes the following steps S1001 to S1005.

[0145] S1001: The management process pulls the write-ahead log from the storage node of the storage layer.

[0146] The management process can specifically pull the write-ahead log from any master node or slave node in each logical shard.

[0147] S1002: Generate a data update file according to the write-ahead log.

[0148] In one embodiment of the present application, the write-ahead log is decoded to obtain an operation type of each log entry in the write-ahead log, where the operation type includes a write operation or a non-write operation; and a data update file is generated for the log entry of the write operation according to the operation type in the write-ahead log.

[0149] For example, after the management process performs pre-write log decoding, it can discard non-write log entries, such as operation records such as selecting a master node, metadata synchronization, and reading pre-write logs. Then, the log entries corresponding to the data write operations are parsed to generate a data update file. The data update file can be a data file in a storage format such as Parquet or Tsfile, and the data is still stored in memory.

[0150] Parquet is a columnar file format designed to efficiently store and process data from the Hadoop ecosystem. This format supports nested data structures, allowing the storage of complex nested patterns, similar to JSON or Protocol Buffers. Parquet files contain multiple column blocks, each of which stores a column of data from a table. This columnar storage method can significantly improve query performance, especially when only a few columns in the table need to be accessed. Parquet is supported by many big data processing frameworks, including Apache Spark, Apache Hive, Apache Impala, etc.

[0151] Tsfile is a columnar storage format designed for time series data, introduced by the Apache IoTDB (formerly LinkisIoTDB) project. It is particularly suitable for Internet of Things (IoT) scenarios, where data is often indexed by timestamps and data points accumulate over time. Tsfile supports high compression and high throughput writing, while providing efficient data query capabilities. It allows users to quickly retrieve data points based on timestamps. Tsfile also supports tags and attributes, which enables it to store metadata, providing richer data description capabilities.

[0152] S1003: Upload the data update file to the second storage system for storage.

[0153] Each data update file consists of data in an LSN interval. After being generated, it is uploaded to the secondary storage system to form persistent data. LSN is an identification number used to represent all log entries corresponding to the data update file. Specifically, it is a monotonically increasing sequence of numbers used to uniquely identify records in the log file. Each log record is assigned an LSN so that the database can track and restore data through the LSN.

[0154] S1004: Update the database directory of the second storage system according to the data update file.

[0155] In one embodiment of the present application, an identification interval of a data update file is obtained, and the identification interval is used to represent the identification serial numbers of all log entries corresponding to the data update file; a database directory of the second storage system is updated according to the identification interval, and the database directory is used to store metadata of data objects in the second storage system.

[0156] The database catalog is a collection of system tables that store database metadata. Metadata is information that describes the database structure and data, which defines how the data in the database is organized. The data in the database catalog can generally include table definitions (column names, data types, keys, etc.), index information, views, stored procedures, triggers, user information, permissions, and roles. Users can obtain database structural information by querying the database catalog. In some database systems, the concept of database catalog is closely related to or identical to the information model, which is a set of standard system tables used to store database metadata and follows the SQL standard. The data in the database catalog is usually persistent, stored on the database server, and is still available after the database is restarted. The database catalog can contain access control information that defines the access rights of users and roles to database objects. In some database systems, the database catalog can also contain database version information to help track the change history of the database structure.

[0157] S1005: Update the write-ahead log according to the storage result of the data update file.

[0158] In one embodiment of the present application, when the data objects and the database directory in the second storage system are updated successfully, the identification interval is marked as a committed state; and the log entries for which data updates have been completed in the write-ahead log are removed according to the identification interval in the committed state.

[0159] For example, a data update file shown in the figure corresponds to an identification interval of 100-200. After the identification interval is marked as a committed state, the log entry corresponding to the identification interval can be removed from the write-ahead log.

[0160] In one embodiment of the present application, after marking the identification interval as a committed state, fault error recovery can be performed based on the committed state of the identification interval. The specific method includes: identifying the running state of the management process in each access node; when the running state of the management process is a run failure, transferring the management process from the access node where it is located to another access node in the computing layer; extracting the log entries of the unfinished data update from the pre-write log according to the identification interval in the committed state; and updating the data objects and database directory in the second storage system according to the log entries of the unfinished data update.

[0161] For example, before pulling the write-ahead log to generate a data update file, the data is saved in the memory, and there may be a risk of file loss after the access node crashes; to address this problem, the embodiment of the present application adopts a mechanism for adding a commit status to the write-ahead log.

[0162] When an access node crashes, it can identify the failure of the management process and reallocate the management process to the access node for normal operation. It can consume and regenerate data update files from the identification point of the last submission status, and re-update the database directory. At the same time, it can clean up duplicate files in the database directory and object storage.

[0163] When an object storage upload fails or a database directory update fails, the operation to identify the submission status is not performed accordingly. The corresponding log entry is submitted and deleted only after the access node successfully retries the request to ensure data integrity.

[0164] In addition, the embodiments of the present application can use the database directory to perform data integrity verification. Specifically, the database directory will record the LSN interval corresponding to each file, and the object storage file will also contain the corresponding LSN interval; by regularly scanning the database directory and the corresponding object storage files, verify whether the LSN intervals are consistent. When inconsistencies occur, mark this segment of data as dirty data, and trigger a data supplement task to export the full amount of data for the corresponding time interval from the time series database of the corresponding time interval, generate a new object storage file, and replace the corresponding file in the object storage.

[0165] In one embodiment of the present application, after the data objects stored in the second storage system are updated according to the data update files, a data aggregation operation may be further performed. Specifically, the operation may include: obtaining metadata of the data objects from the database directory, and identifying the identification intervals corresponding to each data update file according to the metadata; marking multiple data update files with consecutive identification numbers as files to be aggregated according to the identification intervals; downloading the files to be aggregated from the storage layer of the second storage system according to a preset merging period, and aggregating the files to be aggregated to obtain aggregated files; uploading the aggregated files to the storage layer of the second storage system, and deleting the files to be aggregated from the storage layer of the second storage system after updating the database directory.

[0166] The online hot backup can only generate smaller files each time, which will generate a large number of small files, most of which can be aggregated into separate files. On the one hand, it can reduce the storage space, and on the other hand, it can speed up the pulling speed.

[0167] For example, the embodiment of the present application can periodically pull metadata from the database directory to determine the files that need to be aggregated; then, download these files from the object storage and perform aggregation calculations locally to generate new files. After uploading the new files to the object storage, change the database directory. Deleting files in the object storage is not allowed before the database directory is changed.

[0168] For example, in an application scenario, six data update files are stored in the second storage system, and the corresponding identification intervals are 100-199, 200-299 ... 600-699. After aggregation processing, a new data update file corresponding to the identification interval 100-699 can be generated.

[0169] In addition, because the time series database is configured with a data validity period TTL, the data needs to be deleted after it expires. The embodiment of the present application can inspect the expiration time of the files in the database directory and perform the deletion operation when it expires. During the deletion process, the metadata in the database directory is first deleted so that the query link is invisible to this file, and then the file is physically deleted in the object storage.

[0170] The embodiment of the present application achieves efficient management of data update files by obtaining metadata of data objects from the database directory and identifying identification intervals corresponding to data update files. By marking multiple files with consecutive identification numbers as files to be aggregated, the system can download these files from the storage layer of the second storage system according to a preset merge cycle, and perform aggregation processing to generate aggregated files. This process not only optimizes the use of storage resources and reduces the occupancy of storage space, but also improves data accessibility and query efficiency through aggregation operations. Finally, the aggregated files are uploaded back to the second storage system, and the original files to be aggregated are deleted after the database directory is updated, further ensuring data consistency and system cleanliness. This method improves the level of automation in data management, reduces manual intervention, and also provides a more accurate and comprehensive data view for data analysis and report generation.

[0171] Fig.11 The following is a block diagram showing the architecture of a storage system in an application scenario according to an embodiment of the present application. Fig.11 As shown, the local cloud time series database is used as the first storage system, and the external object storage is used as the second storage system. Based on this storage system, the embodiment of the present application can achieve effects such as storage and computing separation, read-write separation, and read-read separation.

[0172] In terms of storage and computing separation, the access node pool gw pool and the time series database storage node cluster are responsible for normal write links, generation of near-real-time hot backups, and real-time queries with high freshness. This can be dynamically specified internally, and the empirical value is generally set to 1 day. Queries within 1 day generally involve a small part of the resources of the underlying storage nodes, and queries involving a large range of time intervals are more resource-intensive. Queries within 1 day account for the vast majority of queries. These queries will be split and query tasks will be separated to multiple physical nodes for execution. Because the storage medium is a local solid-state drive SSD, the speed is much higher than that of an external storage system, which can bring extremely high performance.

[0173] In terms of read-write separation, queries beyond one day will no longer be split, but will be executed by the query node pool querypool. The query pool consists of a group of independent computing nodes. Its function is to pull the original data from the object storage and perform full computing operations. This process does not consume the resources of the gw pool and the time series database storage node, and effectively isolates the impact of previous large queries on the storage nodes.

[0174] Regarding read-read separation, businesses often require that different query types be executed without affecting each other. The embodiment of the present application allows the creation of multiple query pools, and user queries can specify different query pools, so that query executions of different priorities can be physically isolated.

[0175] Continue to refer Fig.11 As shown, in addition to the object storage and time series database, in order to prevent the same query from being executed multiple times and consuming computing resources, the embodiment of the present application can also introduce KV storage (key-value database) as a cache in the system. When the query is executed, the KV storage will be actively updated. In this process, the gw pool and query pool behave the same. When the same query statement sql is executed for the second time, it will first query whether there is a cache in the KV storage; if it exists, it only needs to actually execute the query statement sql outside the time range in the new cache. In this way, most queries can be converged to the time series database, so as to achieve efficient query.

[0176] For example, the first query statement executed is select count() from m where time>now()-1h; one hour later, a new query statement select count() from m where time>now()-2h is executed. On this basis, the data of the previous hour can be directly obtained from the KV storage without having to be recalculated in the time series database, which can save a lot of repeated calculation problems caused by repeated queries and save the cost of repeated calculations.

[0177] Based on the introduction of the above application scenarios, it can be seen that the embodiment of the present application innovatively introduces hot backup capabilities, using three-level storage of KV engine / physical machine SSD / object storage to provide storage and computing separation, high performance, and reliable time series database services.

[0178] Fully utilize the query characteristics of time series data query, which are large number of queries with high freshness, small query scale, high real-time requirements, and small number of historical queries, large query scale, and low real-time requirements. Near real-time queries are executed with high performance in the local data database, and historical queries are executed on independent computing nodes based on object storage, so as to achieve read-read separation and read-write separation; and object storage and computing nodes can be expanded independently; in addition, historical queries are not strongly dependent on object storage, and when object storage fails, queries are transferred to time series database queries without affecting user availability.

[0179] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0180] The following introduces an apparatus embodiment of the present application, which can be used to execute the data processing method in the above-mentioned embodiment of the present application. Fig.12 The structure block diagram of the data processing device provided in the embodiment of the present application is schematically shown. Fig.12 As shown, the data processing device 1200 includes:

[0181] The splitting module 1210 is configured to split the data query request into a first sub-request corresponding to a first storage system and a second sub-request corresponding to a second storage system; the first storage system and the second storage system have different system configurations, and the system configuration includes a distribution ratio of computing tasks between a computing layer and a storage layer;

[0182] An execution module 1220 is configured to execute the first sub-request on the first storage system to obtain a first query result, and execute the second sub-request on the second storage system to obtain a second query result;

[0183] The aggregation module 1230 is configured to aggregate the first query result and the second query result to obtain a query result of the data query request.

[0184] In some embodiments of the present application, based on the above technical solution, the splitting module 1210 may further include:

[0185] A request parsing module is configured to parse the data query request to obtain a data time series range carried in the data query request, where the data time series range is used to represent a distribution range of the data to be queried in a time series;

[0186] a time series splitting module, configured to split the data time series range into a first time interval corresponding to the first storage system and a second time interval corresponding to the second storage system according to the time series capacity of the first storage system, wherein the time series capacity is used to represent a maximum time range of the time series data stored on the first storage system;

[0187] The request splitting module is configured to split the data query request into a first sub-request with the first time interval as the data timing range and a second sub-request with the second time interval as the data timing range.

[0188] In some embodiments of the present application, based on the above technical solution, in the system configuration of the first storage system, the distribution proportion of computing tasks in the computing layer is smaller than that in the storage layer; the execution 1220 module can be further configured as: in the computing layer of the first storage system, the first sub-request is split into multiple computing tasks; in the storage layer of the first storage system, multiple storage nodes respectively query the original data corresponding to each of the computing tasks, and perform computational processing on the original data according to the computing tasks to obtain query result shards; in the computing layer of the first storage system, multiple query result shards are aggregated to obtain a first query result.

[0189] In some embodiments of the present application, based on the above technical solution, the storage layer of the first storage system includes multiple logical shards for storing different data, the logical shards include multiple storage nodes, and the multiple storage nodes located in the same logical shard are used to store the same data; the execution module 1220 can be further configured as: in the storage layer of the first storage system, sending the multiple computing tasks to the multiple logical shards respectively; within the logical shards, selecting a target storage node according to the operating status of each of the storage nodes, and querying the original data corresponding to the computing task on the target storage node.

[0190] In some embodiments of the present application, based on the above technical solution, in the system configuration of the second storage system, the distribution ratio of computing tasks on the computing layer is greater than that on the storage layer; the execution module 1220 can be further configured as: in the storage layer of the second storage system, query the original data corresponding to the second sub-request; in the computing layer of the second storage system, calculate and process the original data according to the second sub-request to obtain a second query result.

[0191] In some embodiments of the present application, based on the above technical solution, the computing layer of the second storage system includes multiple query pools corresponding to different query types, the query pool includes multiple computing nodes, and the multiple computing nodes located in the same query pool are used to process data query requests with the same query type; the execution module 1220 can be further configured as: in the computing layer of the second storage system, the original data is sent to the target query pool according to the query type of the second sub-request; in the target query pool, the target computing node is selected according to the operating status of each of the computing nodes, and the original data is calculated and processed on the target computing node to obtain the second query result.

[0192] In some embodiments of the present application, based on the above technical solution, the storage layer of the first storage system includes multiple logical slices for storing different data, the logical slices include multiple storage nodes, and the multiple storage nodes in the same logical slice are used to store the same data; the data processing device 1200 also includes:

[0193] A creation module, configured to create, in the computing layer of the first storage system, a management process corresponding to each of the logical slices, the management process being used to synchronize data between the first storage system and the second storage system;

[0194] an acquisition module, configured to acquire a write-ahead log corresponding to the storage node from the logical shard according to the management process, wherein the write-ahead log is used to store an operation record for the storage node;

[0195] A generating module, configured to generate a data update file according to the write-ahead log, wherein the data update file includes data update content on the storage node;

[0196] The saving module is configured to transmit the data update file to the second storage system, and update the data object saved in the second storage system according to the data update file.

[0197] In some embodiments of the present application, based on the above technical solution, the computing layer of the first storage system includes multiple access nodes for assigning tasks to the logical shards; the creation module is further configured to: obtain the node type of each of the access nodes, the node type including a read-write node or a management node; the read-write node is used to assign read-write tasks for performing data reading operations or data writing operations to the logical shards, and the management node is used to assign management tasks for performing data synchronization operations to the logical shards; select a management node from the multiple access nodes according to the node type, and create a management process corresponding to each of the logical shards in the management node.

[0198] In some embodiments of the present application, based on the above technical solution, the computing layer of the first storage system includes multiple access nodes for assigning tasks to the logical slices; the creation module is further configured to: create management processes corresponding to each of the logical slices in the access nodes; obtain the resource occupancy of all management processes in the access nodes, and the resource occupancy is used to indicate the usage ratio of the computing resources and storage resources of the first storage system by the management process; when the resource occupancy is greater than a preset ratio threshold, configure some management processes in the access nodes to a waiting state.

[0199] In some embodiments of the present application, based on the above technical solution, the generation module is further configured to: decode the write-ahead log to obtain the operation type of each log entry in the write-ahead log, where the operation type includes a write operation or a non-write operation; and generate a data update file for the log entry of the write operation according to the operation type in the write-ahead log.

[0200] In some embodiments of the present application, based on the above technical solution, the data processing device 1200 also includes an update module, which is configured to: obtain an identification interval of the data update file, the identification interval is used to represent the identification numbers of all log entries corresponding to the data update file; update the database directory of the second storage system according to the identification interval, the database directory is used to store metadata of data objects in the second storage system; when the data objects in the second storage system and the database directory are updated successfully, mark the identification interval as a committed state; remove the log entries in the write-ahead log that have completed data updates according to the identification interval in the committed state.

[0201] In some embodiments of the present application, based on the above technical solution, the computing layer of the first storage system includes multiple access nodes for running the management process; the update module is also configured to: identify the running status of the management process in each of the access nodes; when the running status of the management process is a failure, transfer the management process from the access node where it is located to another access node in the computing layer; extract log entries of incomplete data updates from the write-ahead log according to the identification interval in the committed state; and update the data objects and the database directory in the second storage system according to the log entries of the incomplete data updates.

[0202] In some embodiments of the present application, based on the above technical solution, the data processing device 1200 also includes an aggregation module, which is configured to: obtain metadata of the data object from the database directory, and identify the identification interval corresponding to each of the data update files according to the metadata; mark multiple data update files with consecutive identification numbers as files to be aggregated according to the identification interval; download the files to be aggregated from the storage layer of the second storage system according to a preset merging period, and aggregate the files to be aggregated to obtain aggregated files; upload the aggregated files to the storage layer of the second storage system, and delete the files to be aggregated from the storage layer of the second storage system after updating the database directory.

[0203] The specific details of the data processing device provided in each embodiment of the present application have been described in detail in the corresponding method embodiments and will not be repeated here.

[0204] Fig.13 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0205] It should be noted that Fig.13 The computer system 1300 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0206] like Fig.13 As shown, the computer system 1300 includes a central processing unit 1301 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1302 (ROM) or the program loaded from the storage part 1308 to the random access memory 1303 (RAM). Various programs and data required for system operation are also stored in the random access memory 1303. The central processing unit 1301, the read-only memory 1302 and the random access memory 1303 are connected to each other through a bus 1304. The input / output interface 1305 (Input / Output interface, i.e., I / O interface) is also connected to the bus 1304.

[0207] The following components are connected to the input / output interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the input / output interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed so that a computer program read therefrom is installed into the storage section 1308 as needed.

[0208] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1309, and / or installed from the removable medium 1311. When the computer program is executed by the central processor 1301, various functions defined in the system of the present application are executed.

[0209] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0210] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0211] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0212] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0213] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0214] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: include: Obtaining metadata of the data object from the database directory, and identifying identification intervals corresponding to each data update file according to the metadata; The database directory is used to store metadata of data objects in the second storage system, the data update file includes data update content of the first storage system, and the identification interval is used to represent identification serial numbers of all log entries corresponding to the data update file; Marking a plurality of data update files having consecutive identification numbers as files to be aggregated according to the identification interval; Downloading the to-be-aggregated files from the storage layer of the second storage system according to a preset merging period, and performing aggregation processing on the to-be-aggregated files to obtain aggregated files; uploading the aggregated file to a storage layer of the second storage system; Splitting the data query request into a first sub-request corresponding to the first storage system and a second sub-request corresponding to the second storage system; The first storage system and the second storage system have different system configurations, wherein the system configurations include a distribution ratio of computing tasks between a computing layer and a storage layer; Executing the first sub-request on the first storage system to obtain a first query result, and executing the second sub-request on the second storage system to obtain a second query result; The first query result and the second query result are aggregated to obtain a query result of the data query request.

2. The data processing method according to claim 1, characterized in that: Splitting the data query request into a first sub-request corresponding to the first storage system and a second sub-request corresponding to the second storage system includes: Parsing the data query request to obtain a data time series range carried in the data query request, where the data time series range is used to represent a distribution range of the data to be queried in a time series; Splitting the data time series range into a first time interval corresponding to the first storage system and a second time interval corresponding to the second storage system according to the time series capacity of the first storage system, wherein the time series capacity is used to represent a maximum time range of the time series data stored on the first storage system; The data query request is split into a first sub-request with the first time interval as the data timing range and a second sub-request with the second time interval as the data timing range.

3. The data processing method according to claim 1, characterized in that: In the system configuration of the first storage system, the distribution ratio of computing tasks on the computing layer is smaller than that on the storage layer; Executing the first sub-request on the first storage system to obtain a first query result includes: In a computing layer of the first storage system, splitting the first sub-request into a plurality of computing tasks; In the storage layer of the first storage system, multiple storage nodes respectively query the original data corresponding to each of the computing tasks, and perform computing processing on the original data according to the computing tasks to obtain query result shards; In the computing layer of the first storage system, a plurality of the query result shards are aggregated to obtain a first query result.

4. The data processing method according to claim 3, characterized in that: The storage layer of the first storage system includes a plurality of logical slices for storing different data, the logical slices include a plurality of storage nodes, and the plurality of storage nodes in the same logical slice are used to store the same data; In the storage layer of the first storage system, multiple storage nodes respectively query the original data corresponding to each of the computing tasks, including: In the storage layer of the first storage system, the plurality of computing tasks are sent to the plurality of logical shards respectively; In the logical slice, a target storage node is selected according to the operation status of each storage node, and original data corresponding to the computing task is queried on the target storage node.

5. The data processing method according to claim 1, characterized in that: In the system configuration of the second storage system, the computing tasks are distributed more heavily on the computing layer than on the storage layer; Executing the second sub-request on the second storage system to obtain a second query result includes: In a storage layer of the second storage system, query original data corresponding to the second sub-request; In the computing layer of the second storage system, the original data is computed and processed according to the second sub-request to obtain a second query result.

6. The data processing method according to claim 5, characterized in that: The computing layer of the second storage system includes a plurality of query pools corresponding to different query types, the query pool includes a plurality of computing nodes, and the plurality of computing nodes in the same query pool are used to process data query requests with the same query type; The second query result is obtained by calculating and processing the original data according to the second sub-request, including: In the computing layer of the second storage system, the original data is sent to a target query pool according to the query type of the second sub-request; In the target query pool, a target computing node is selected according to the running status of each computing node, and the original data is calculated and processed on the target computing node to obtain a second query result.

7. The data processing method according to any one of claims 1 to 6, characterized in that: The storage layer of the first storage system includes a plurality of logical slices for storing different data, the logical slices include a plurality of storage nodes, and the plurality of storage nodes in the same logical slice are used to store the same data; Before splitting the data query request into a first sub-request corresponding to the first storage system and a second sub-request corresponding to the second storage system, the method further includes: In the computing layer of the first storage system, a management process corresponding to each of the logical slices is created, wherein the management process is used to synchronize data between the first storage system and the second storage system; Acquire a write-ahead log corresponding to the storage node from the logical slice according to the management process, wherein the write-ahead log is used to store an operation record for the storage node; Generate a data update file according to the write-ahead log, wherein the data update file includes data update content on the storage node; The data update file is transmitted to the second storage system, and the data object stored in the second storage system is updated according to the data update file.

8. The data processing method according to claim 7, characterized in that: The computing layer of the first storage system includes a plurality of access nodes for allocating tasks to the logical slices; Creating a management process corresponding to each of the logical slices, including: Acquire the node type of each of the access nodes, where the node type includes a read-write node or a management node; the read-write node is used to assign a read-write task for performing a data read operation or a data write operation to the logical slice, and the management node is used to assign a management task for performing a data synchronization operation to the logical slice; A management node is selected from the plurality of access nodes according to the node type, and a management process corresponding to each of the logical slices is created in the management node.

9. The data processing method according to claim 7, characterized in that: The computing layer of the first storage system includes a plurality of access nodes for allocating tasks to the logical slices; Creating a management process corresponding to each of the logical slices, including: Creating a management process corresponding to each of the logical slices in the access node; Acquire resource usage of all management processes in the access node, where the resource usage is used to indicate a usage ratio of computing resources and storage resources of the first storage system by the management process; When the resource occupancy is greater than a preset ratio threshold, part of the management processes in the access node are configured to be in a waiting state.

10. The data processing method according to claim 7, characterized in that: Generating a data update file according to the write-ahead log includes: Decoding the write-ahead log to obtain an operation type of each log entry in the write-ahead log, where the operation type includes a write operation or a non-write operation; A data update file is generated for a log entry of a write operation according to the operation type in the write-ahead log.

11. The data processing method according to claim 7, characterized in that: After updating the data object stored in the second storage system according to the data update file, the method further includes: Obtaining an identification interval of the data update file; updating a database directory of the second storage system according to the identification interval; When the data object in the second storage system and the database directory are both updated successfully, marking the identification interval as a committed state; The log entries for which data update has been completed in the write-ahead log are removed according to the submitted status identification interval.

12. The data processing method according to claim 11, characterized in that: The computing layer of the first storage system includes a plurality of access nodes for running the management process; After marking the identification interval as a submitted state, the method further includes: identifying the running status of the management process in each of the access nodes; When the running state of the management process is running failure, transferring the management process from the access node where the management process is located to another access node in the computing layer; Extracting log entries of unfinished data updates from the write-ahead log according to the identification interval in the committed state; The data objects in the second storage system and the database directory are updated according to the log entries of the unfinished data update.

13. The data processing method according to claim 1, characterized in that: After uploading the aggregated file to the storage layer of the second storage system, the method further includes: After the database directory is updated, the to-be-aggregated file is deleted from the storage layer of the second storage system.

14. A data processing device, characterized in that: include: an aggregation module configured to obtain metadata of data objects from a database directory and identify identification intervals corresponding to respective data update files according to the metadata; The database directory is used to store metadata of data objects in the second storage system, the data update file includes data update content of the first storage system, and the identification interval is used to represent identification numbers of all log entries corresponding to the data update file; and multiple data update files with consecutive identification numbers are marked as files to be aggregated according to the identification interval; Downloading the to-be-aggregated files from the storage layer of the second storage system according to a preset merging period, and performing aggregation processing on the to-be-aggregated files to obtain aggregated files; uploading the aggregated file to a storage layer of the second storage system; a splitting module configured to split a data query request into a first sub-request corresponding to a first storage system and a second sub-request corresponding to a second storage system; the first storage system and the second storage system have different system configurations, and the system configurations include a distribution ratio of computing tasks between a computing layer and a storage layer; an execution module, configured to execute the first sub-request on the first storage system to obtain a first query result, and execute the second sub-request on the second storage system to obtain a second query result; The aggregation module is configured to aggregate the first query result and the second query result to obtain a query result of the data query request.

15. A computer readable medium, characterized in that The computer-readable medium stores a computer program, and when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 13 is implemented.

16. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the executable instructions to implement the data processing method as described in any one of claims 1 to 13.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 13 is implemented.

Images