Data processing method and device, and electronic device

Abstract

The application discloses a data processing method and device and electronic equipment, the method is applied to the technical field of big data, and the method comprises the steps of determining data to be written; writing the data to be written into a target shard of a data warehouse according to a shard data table, obtaining an updated data warehouse; in the case that a data query request is received, responding to the data query request, querying target data in the updated data warehouse according to the shard data table; and calculating and processing the target data according to a data analysis tool to obtain a calculation result of the target data. Through the application, the problem that, in the related art, when data in a relational database is subjected to online analytical processing, the relational database has poor real-time processing efficiency on massive data, and cannot meet the high-time-efficiency analysis requirement of online analytical processing is solved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a data processing method, apparatus, and electronic device. Background Technology

[0002] Currently, with the enhancement of computing power and the rapid development of the internet, the speed of foreign exchange information transmission in the financial market is also increasing rapidly. To grasp the trends of the financial market and react quickly to foreign exchange information, financial institutions need to obtain foreign exchange information in a timely manner, and then synthesize and analyze this information to adjust the product prices or business strategies of their financial products. In existing technologies, when obtaining foreign exchange information from the financial market, financial institutions can use information collection tools to store the collected transaction information in a relational database, query the data from the relational database, and then analyze and calculate the retrieved data to obtain the transaction information of the financial market. Although relational databases are convenient to use, their query efficiency is low and cannot meet the needs of high-timeliness, real-time analysis of massive amounts of foreign exchange information in the financial market.

[0003] There is currently no effective solution to the problem that relational databases are inefficient at processing massive amounts of data in real time when performing online analytical processing (OLAP) on data in relational databases. Summary of the Invention

[0004] The main objective of this application is to provide a data processing method, apparatus, and electronic device to solve the problem in the related art that when performing online analytical processing on data in relational databases, the relational databases are inefficient at real-time processing of massive amounts of data, thus failing to meet the high-timeliness analysis requirements of online analytical processing.

[0005] To achieve the above objectives, according to one aspect of this application, a data processing method is provided, comprising: determining data to be written; writing the data to be written into a target shard of a data warehouse according to a sharding data table to obtain an updated data warehouse, wherein the sharding data table stores sharding information corresponding to the data in the data warehouse; upon receiving a data query request, responding to the data query request and querying target data in the updated data warehouse according to the sharding data table; and performing calculations and processing on the target data using a data analysis tool to obtain a calculation result of the target data.

[0006] Furthermore, before writing the data to be written into the target shard of the data warehouse according to the shard data table, the method further includes: configuring a target container and creating a target process in the target container, wherein the target container is a different container from the container where the data warehouse is located; creating a write node using the target process and starting the write node.

[0007] Further, writing the data to be written into the target shard of the data warehouse according to the sharding data table includes: creating a cache queue and writing the data to be written into the cache queue one by one; determining the data stored in the cache queue after a first preset time interval to obtain batch data; determining the target shard in the data warehouse corresponding to each piece of data in the batch data according to the sharding data table; and using the writing node to write each piece of data in the batch data into the target shard corresponding to that piece of data.

[0008] Furthermore, before querying the target data in the updated data warehouse based on the sharded data table, the method further includes: loading the sharded data table into a Redis cache; querying the target data in the updated data warehouse based on the sharded data table includes: obtaining the sharded data table from the Redis cache through a query node, and obtaining the target data from the shards of the data warehouse based on the sharded data table, wherein the query node is used to query data in the data warehouse.

[0009] Furthermore, before loading the sharded data table into the Redis cache, the method further includes: when a target object initiates a modification instruction on the sharded data table, updating the sharded data table according to the modification instruction to obtain an updated sharded data table; and updating the sharded data table stored in the Redis cache according to the updated sharded data table to obtain an updated sharded data table.

[0010] Further, the process of using the write node to write each piece of data in the batch data into the target shard corresponding to that piece of data includes: if it is detected that the amount of data processed by the write node within a second preset time period is greater than or equal to a first preset data amount, then a first node is created, and the first node and the write node are used to write each piece of data in the batch data into the target shard corresponding to that piece of data; if it is detected that the amount of data processed by the write node within the second preset time period is less than a second preset data amount, then the first node is deleted, and the write node is used to write each piece of data in the batch data into the target shard corresponding to that piece of data.

[0011] Furthermore, the step of using the write node to write each piece of data in the batch data into the target shard corresponding to that piece of data includes: if it is detected that the amount of data written to the data warehouse within a third preset time period is greater than or equal to a third preset data amount, then the data exceeding the third preset data amount is stored in the message engine system; if it is detected that the amount of data written to the data warehouse within the third preset time period is less than a fourth preset data amount, then the data stored in the message engine system is used as the data to be written, and the step of writing the data to be written into the target shard of the data warehouse according to the shard data table is executed.

[0012] Furthermore, upon receiving a data query request, responding to the data query request and querying the target data in the updated data warehouse based on the sharded data table includes: if there is a query statement in the query node with a query duration greater than a fourth preset duration, then the query statement is recorded and optimized.

[0013] To achieve the above objectives, according to another aspect of this application, a data processing apparatus is provided, comprising: a determining unit for determining data to be written; a writing unit for writing the data to be written into a target shard of a data warehouse according to a sharding data table to obtain an updated data warehouse, wherein the sharding data table stores sharding information corresponding to the data in the data warehouse; a querying unit for responding to a data query request upon receiving the data query request and querying the target data in the updated data warehouse according to the sharding data table; and a calculation unit for performing calculations and processing on the target data using a data analysis tool to obtain a calculation result for the target data.

[0014] Furthermore, the apparatus further includes: a configuration unit, configured to configure a target container and create a target process in the target container before writing the data to be written into the target shard of the data warehouse according to the shard data table, wherein the target container is a container different from the container in which the data warehouse is located; and a creation unit, configured to create a write node using the target process and start the write node.

[0015] Further, the writing unit includes: a creation subunit for creating a cache queue and writing the data to be written into the cache queue one by one; a first determination subunit for determining the data stored in the cache queue after a first preset time interval to obtain batch data; a second determination subunit for determining the target shard corresponding to each piece of data in the batch data in the data warehouse based on the shard data table; and a writing subunit for using the writing node to write each piece of data in the batch data into the target shard corresponding to that piece of data.

[0016] Furthermore, the apparatus further includes: a loading unit, configured to load the sharded data table into a Redis cache before querying target data in the updated data warehouse based on the sharded data table; the query unit includes: an acquisition subunit, configured to acquire the sharded data table in the Redis cache through a query node, and acquire the target data in the shards of the data warehouse based on the sharded data table, wherein the query node is used to query data in the data warehouse.

[0017] Furthermore, the apparatus further includes: a first update unit, configured to update the sharded data table according to the modification instruction issued by the target object before loading the sharded data table into the Redis cache, thereby obtaining an updated sharded data table; and a second update unit, configured to update the sharded data table stored in the Redis cache according to the updated sharded data table, thereby obtaining an updated sharded data table.

[0018] Further, the writing subunit includes: a first writing module, configured to create a first node and use the first node and the writing node to write each piece of data in the batch data into the target shard corresponding to that piece of data when the amount of data processed by the writing node within the second preset time period is greater than or equal to the first preset data amount; and a second writing module, configured to delete the first node and use the writing node to write each piece of data in the batch data into the target shard corresponding to that piece of data when the amount of data processed by the writing node within the second preset time period is less than the second preset data amount.

[0019] Further, the writing subunit includes: a storage module, configured to store data exceeding the third preset data amount in the message engine system when the amount of data written to the data warehouse within the third preset time period is greater than or equal to the third preset data amount; and an execution module, configured to use the data stored in the message engine system as the data to be written, and execute the step of writing the data to be written into the target shard of the data warehouse according to the sharding data table, when the amount of data written to the data warehouse within the third preset time period is less than the fourth preset data amount.

[0020] Furthermore, the query unit includes a processing subunit, used to record and optimize the query statement if there is a query statement in the query node with a query duration greater than a fourth preset duration.

[0021] To achieve the above objectives, according to one aspect of this application, an electronic device is provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data processing method described in any of the above.

[0022] This application employs the following steps: determining the data to be written; writing the data to be written into the target shard of the data warehouse according to the sharding data table to obtain an updated data warehouse, wherein the sharding data table stores the sharding information corresponding to the data in the data warehouse; upon receiving a data query request, responding to the data query request and querying the target data in the updated data warehouse according to the sharding data table; and performing calculations and processing on the target data using data analysis tools to obtain the calculation results of the target data. This solves the problem in related technologies where, when performing online analytical processing (OLAP) on data in relational databases, the efficiency of relational databases in real-time processing of massive amounts of data is poor, failing to meet the high-timeliness analysis requirements of OLAP. By setting up a sharding data table, massive amounts of data can be evenly stored in various shards of the data warehouse, which facilitates the rapid retrieval of the target data required for data query requests in the data warehouse, and thus allows for rapid analysis and processing of the target data to obtain real-time foreign exchange information in the foreign exchange market, thereby improving the efficiency of real-time statistical analysis of foreign exchange market information. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 This is a flowchart of a data processing method according to Embodiment 1 of this application;

[0025] Figure 2 This is a schematic diagram of an optional data processing method provided according to Embodiment 1 of this application;

[0026] Figure 3 This is a schematic diagram of a data processing apparatus according to Embodiment 2 of this application;

[0027] Figure 4 This is a schematic diagram of a data processing electronic device provided according to Embodiment 5 of this application. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, user input information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, written data, publicly disclosed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0033] A data warehouse (DWH) is a subject-oriented, integrated, relatively stable collection of data that reflects historical changes and is used to support management decisions. Nowadays, it generally refers to a large-scale data warehouse used for OLAP operations.

[0034] OLAP (Online Analytical Processing) focuses primarily on analysis, involving numerous queries and fewer CRUD operations. Data warehousing and OLAP are complementary; modern OLAP systems typically use a data warehouse as a foundation, extracting a subset of detailed data from the data warehouse, aggregating it, and storing it in an OLAP storage device for front-end analytical tools to access.

[0035] PaaS (Platform as a Service) is a cloud computing service that provides a computing platform and solutions. Cloud computing is a type of distributed computing that refers to breaking down massive data processing programs into countless smaller programs via a network "cloud." These smaller programs are then processed and analyzed by a system composed of multiple servers to obtain the results and return them to the user.

[0036] Example 1

[0037] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of the data processing method provided in Embodiment 1 of this application, as follows: Figure 1 As shown, the method includes the following steps:

[0038] Step S101: Determine the data to be written.

[0039] In this first embodiment, the data to be written refers to the data in the data warehouse to be written, such as currency exchange rate information and bond information that change over time in the financial foreign exchange market. A data warehouse (DWH) is used to store large amounts of data and assists enterprises in making decisions based on data mining and data analysis.

[0040] Step S102: Write the data to be written into the target shard of the data warehouse according to the sharding data table to obtain the updated data warehouse. The sharding data table stores the sharding information corresponding to the data in the data warehouse.

[0041] In this first embodiment, to flexibly manage data in the data warehouse, sharded data tables can be added, and sharding strategies for these tables can be customized. Specifically, due to significant market volatility and active foreign exchange trading during specific time periods of the day (e.g., the time when the London Stock Exchange and the New York Stock Exchange are open), foreign exchange information for these specific periods can be stored in shard A, while foreign exchange information for other periods within the same date can be stored in shard B. Furthermore, the sharding strategy for the sharded data tables can be adjusted in real time based on actual conditions.

[0042] Step S103: Upon receiving a data query request, respond to the data query request and query the target data in the updated data warehouse based on the sharded data table.

[0043] In this first embodiment, in order to respond to data query requests in a timely manner, it is necessary to retrieve the data (i.e., the target data) requested in the data query request from the data warehouse based on the sharded data table. Since both the sharded data table and the data warehouse are constantly updated, when querying data, it is necessary to perform a query in the data warehouse based on the current sharded data table.

[0044] Step S104: Calculate and process the target data using data analysis tools to obtain the calculation results of the target data.

[0045] In this first embodiment, existing interactive analysis tools (such as Grafana, Kibana, and BI tools) can be used to calculate and process the foreign exchange information in the target data to obtain the calculation results. Specifically, the financial institution's integrated quotation platform obtains real-time foreign exchange market data through API requests and stores the foreign exchange market data in a data warehouse. Then, the data analysis tool Grafana is used to calculate the foreign exchange market data and draw a trend chart of the foreign exchange market data. Furthermore, the financial institution's quotation staff can monitor the market situation in real time based on the trend chart of the foreign exchange market data and react promptly when the prices of financial products fluctuate significantly.

[0046] In summary, the data processing method provided in Embodiment 1 of this application determines the data to be written; writes the data to be written into the target shard of the data warehouse according to the sharding data table to obtain an updated data warehouse, wherein the sharding data table stores the sharding information corresponding to the data in the data warehouse; upon receiving a data query request, responds to the data query request and queries the target data in the updated data warehouse according to the sharding data table; and calculates and processes the target data using data analysis tools to obtain the calculation result of the target data. This solves the problem in related technologies where, when performing online analytical processing (OLAP) on data in relational databases, the efficiency of relational databases in real-time processing of massive amounts of data is poor, failing to meet the high-timeliness analysis requirements of OLAP. By setting up a sharding data table, massive amounts of data can be evenly stored in various shards of the data warehouse, which is beneficial for quickly obtaining the target data required for data query requests in the data warehouse, and then quickly analyzing and processing the target data to obtain real-time foreign exchange information in the foreign exchange market, thereby improving the efficiency of real-time statistical analysis of foreign exchange market information.

[0047] Optionally, in the data processing method provided in Embodiment 1 of this application, before writing the data to be written into the target shard of the data warehouse according to the sharded data table, the above method further includes: configuring a target container and creating a target process in the target container, wherein the target container is a container different from the container where the data warehouse is located; using the target process to create a write node and starting the write node.

[0048] In this first embodiment, in order to save on the resource overhead of the data warehouse, a target container independent of the container where the data warehouse is located can be created. Write nodes are created and run in the target container through the target process, so that the data to be written can be written into the data warehouse through the write nodes.

[0049] By deploying write nodes in target containers independent of the data warehouse, the data warehouse and write nodes are decoupled, saving the resource overhead of the data warehouse and increasing the resources available to the write nodes, thereby improving the working efficiency of both the write nodes and the data warehouse.

[0050] Optionally, in the data processing method provided in Embodiment 1 of this application, writing the data to be written into the target shard of the data warehouse according to the shard data table includes: creating a cache queue and writing the data to be written into the cache queue one by one; determining the data stored in the cache queue after a first preset time interval to obtain batch data; determining the target shard in the data warehouse corresponding to each piece of data in the batch data according to the shard data table; and using a write node to write each piece of data in the batch data into the target shard corresponding to that piece of data.

[0051] In existing technologies, when writing massive amounts of data into a data warehouse in the form of files, there are often many small files in the massive files, and each file needs to undergo a data merging operation (Merge operation, which inserts the data to be written into the target table of the data warehouse according to specified conditions, so as to perform data analysis, report generation, and other operations) when writing data into the data warehouse. This results in a slow write speed for each file.

[0052] In this first embodiment, to improve the efficiency of the write node, a cache queue can be used to write the data to be written in batches to the data warehouse. Specifically, when writing the data to be written to the data warehouse, a temporary cache queue can be added, and each piece of data to be written can be written to the cache queue. When the cache queue is full or reaches a first preset time (e.g., 10 seconds, 60 seconds, etc.), the data in the cache queue is written to the data warehouse in batches according to the sharded data table.

[0053] It is important to note that after writing the data to be written to the data warehouse shards, the data in the shards can be backed up to avoid the loss of data stored in a shard due to the failure of a single shard.

[0054] By using a cache queue to write data to the data warehouse in batches, the number of data merging operations required when writing data to the data warehouse is significantly reduced, thereby improving the working efficiency of the write nodes.

[0055] Optionally, in the data processing method provided in Embodiment 1 of this application, before querying the target data in the updated data warehouse based on the sharded data table, the method further includes: loading the sharded data table into the Redis cache; querying the target data in the updated data warehouse based on the sharded data table includes: obtaining the sharded data table from the Redis cache through a query node, and obtaining the target data from the shards of the data warehouse based on the sharded data table, wherein the query node is used to query data in the data warehouse.

[0056] In this first embodiment, to improve query efficiency, sharded data tables can be loaded into the Redis cache. This allows query nodes in the data warehouse to share the sharded data tables in the Redis cache when processing data query requests, and then perform queries in the data warehouse based on the sharded data tables to obtain the target data. Multiple query nodes can be configured.

[0057] Specifically, before the data warehouse receives a data query request, a service cluster for proxy queries can be deployed within the data warehouse. At least one query node is set up within this service cluster, and after the service cluster starts, the sharded data tables are loaded into a Redis cache, which is shared by all query nodes. When the data warehouse receives a data query request, it first passes through Nginx to load the request into each query node. Each query node retrieves the sharded data table from the Redis cache and then accesses the shards in the data warehouse based on the sharded data table to obtain the target data.

[0058] By loading sharded data tables into the Redis cache, at least one query node in the data warehouse can query data through the shared Redis cache, making full use of cluster resources, improving the query performance of the data warehouse, and alleviating the query pressure on the query nodes of the data warehouse.

[0059] Optionally, in the data processing method provided in Embodiment 1 of this application, before loading the sharded data table into the Redis cache, the method further includes: when a target object initiates a modification instruction on the sharded data table, updating the sharded data table according to the modification instruction to obtain an updated sharded data table; and updating the sharded data table stored in the Redis cache according to the updated sharded data table to obtain an updated sharded data table.

[0060] In this first embodiment, to ensure balanced writes across the data warehouse shards, the shard data tables can be updated and adjusted in real time to maintain a balance in the amount of data stored in each shard. After adjusting the shard data tables, the updated tables need to be immediately synchronized to the Redis cache to ensure normal data retrieval.

[0061] Specifically, if the sharded data table is configured to write data acquired during time period A1 (e.g., 4 AM to 5 AM daily, or the first half hour of each hour) to shard B1, and data acquired during time period A2 to shard B2, and after a period of time, shard B1 contains 100 data entries and shard B2 contains 20 data entries, then the sharding strategy of the sharded data table can be adjusted. Time period A1 can be split into time period A3 and time period A4. Data from time period A3 can still be stored in shard B1, but a new shard B3 can be created, and data acquired during time period A4 from shard B1 can be moved to shard B3. After adjusting the sharded data table, the amount of data stored in shard B1, shard B2, and shard B3 will be approximately the same.

[0062] By flexibly adjusting the sharding strategy of the sharded data tables according to actual production conditions, the data distribution of each shard in the data warehouse is relatively balanced, avoiding the problem of slow query speed caused by querying data in shards with large amounts of data, thus improving the efficiency of data query. Furthermore, by updating the sharded data tables to the Redis cache in real time, the consistency of data written to and queried in the data warehouse is guaranteed.

[0063] Optionally, in the data processing method provided in Embodiment 1 of this application, writing each piece of data in the batch data to the target shard corresponding to that piece of data using a write node includes: if the amount of data processed by the write node within a second preset time period is greater than or equal to a first preset data amount, then a first node is created, and the first node and the write node are used to write each piece of data in the batch data to the target shard corresponding to that piece of data; if the amount of data processed by the write node within a second preset time period is less than a second preset data amount, then the first node is deleted, and the write node is used to write each piece of data in the batch data to the target shard corresponding to that piece of data.

[0064] In this first embodiment, to prevent a single write node from crashing due to excessively high write traffic (i.e., the amount of data processed by the write node within a second preset time period), the write traffic of write nodes can be monitored. When a write node with high write traffic (i.e., the amount of data processed by the write node within the second preset time period is greater than or equal to a first preset data amount) is detected, a new write node (i.e., the aforementioned first node) can be created, and the first node can share the data processing load of the write node with high write traffic. That is, the first node and the write node are used to write each piece of data in the batch data to the target shard corresponding to that piece of data. When the write traffic of the write node is low (i.e., the amount of data processed by the write node within the second preset time period is less than the second preset data amount), the first node can be deleted, and the write node can be used to process the data to be written. That is, the write node is used to write each piece of data in the batch data to the target shard corresponding to that piece of data.

[0065] By creating a new first node and using it to share the data processed by the write nodes when the write traffic is high, the pressure on the write nodes with high write traffic is distributed. At the same time, when the write traffic of the write nodes is low, the number of write nodes can be adjusted according to the actual situation, saving cluster resources and improving the stability of the write nodes.

[0066] Optionally, in the data processing method provided in Embodiment 1 of this application, the method of writing each piece of data in the batch data to the target shard corresponding to that piece of data by the write node includes: if it is detected that the amount of data written to the data warehouse within a third preset time period is greater than or equal to a third preset data amount, then the data exceeding the third preset data amount is stored in the message engine system; if it is detected that the amount of data written to the data warehouse within a third preset time period is less than a fourth preset data amount, then the data stored in the message engine system is used as the data to be written, and the step of writing the data to be written to the target shard of the data warehouse according to the shard data table is executed.

[0067] In this first embodiment, to avoid excessive write traffic to the data warehouse and its potential collapse, the write traffic can be monitored. When the write traffic reaches a bottleneck (i.e., the amount of data written to the data warehouse within a third preset time period is greater than or equal to a third preset data amount), the data exceeding the third preset data amount in the data warehouse is temporarily stored in the message engine system. This continues until the write traffic to the data warehouse is low (i.e., the amount of data written to the data warehouse within a third preset time period is less than a fourth preset data amount), at which point the data in the message engine system is written to the data warehouse in batches through write nodes and cache queues.

[0068] By monitoring the write traffic of the data warehouse, optimization can be performed when the write traffic reaches its peak, thus preventing the data warehouse from crashing due to excessive write traffic and improving its stability.

[0069] Optionally, in the data processing method provided in Embodiment 1 of this application, upon receiving a data query request, responding to the data query request and querying the target data in the updated data warehouse based on the sharded data table includes: if there is a query statement in the query node with a query duration greater than a fourth preset duration, then the query statement is recorded and optimized.

[0070] In this first embodiment, to prevent a single query statement from affecting the query efficiency of a query node, queries that consume system resources for an extended period in the data warehouse can be recorded and optimized. Specifically, if an SQL statement with an execution time exceeding 10 seconds (i.e., a query duration longer than the fourth preset duration) is detected, the SQL statement can be recorded, its execution stopped, and other SQL statements can continue to be executed to ensure the normal operation of the query node. The reasons for the long execution time of the SQL statement can then be analyzed to optimize the data warehouse and improve its overall availability and stability. For example, it can be analyzed whether the long execution time of the SQL statement is due to a large amount of data in a certain table or a problem with the data structure of that table, allowing for further optimization and adjustments to that table to avoid longer SQL statement execution times.

[0071] By optimizing and processing query statements that consume system resources for extended periods in the data warehouse, the normal operation of query nodes in the data warehouse can be ensured, thereby improving the overall availability and stability of the data warehouse.

[0072] Optionally, in this first embodiment, the process of writing the data to be written to the data warehouse and analyzing the data in the data warehouse can be as follows: Figure 2 As shown. First, data to be written can be obtained from foreign exchange trading centers and other data sources. Then, the data to be written is integrated into batch data through a cache queue, and the write nodes write the batch data to the various shards of the data warehouse according to the shard data table. After the data is written to the various shards of the data warehouse, the data in each shard can also be backed up (e.g., ...). Figure 2 (Middle shard 1 backup and shard 2 backup). Secondly, in data analysis tools (such as... Figure 2 When Grafana, Kibana, and BI in a data warehouse initiate a data query command, the query nodes in the data warehouse (such as Grafana, Kibana, and BI) Figure 2Query nodes 1 and 2 in the Redis cache retrieve the target data by querying the sharded data tables. Finally, after obtaining the target data, the analysis tool performs calculations on it to obtain the results.

[0073] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0074] Example 2

[0075] This application also provides a data processing apparatus in Embodiment 2. It should be noted that the data processing apparatus in Embodiment 2 can be used to execute the data processing method provided in Embodiment 1. The data processing apparatus provided in Embodiment 2 is described below.

[0076] Figure 3 This is a schematic diagram of a data processing apparatus according to Embodiment 2 of this application. Figure 3 As shown, the device includes: a determination unit 301, a writing unit 302, a query unit 303, and a calculation unit 304.

[0077] Specifically, the determining unit 301 is used to determine the data to be written.

[0078] The writing unit 302 is used to write the data to be written into the target shard of the data warehouse according to the sharding data table, so as to obtain the updated data warehouse. The sharding data table stores the sharding information corresponding to the data in the data warehouse.

[0079] The query unit 303 is used to respond to a data query request upon receiving it, and to query the target data in the updated data warehouse based on the sharded data table.

[0080] The calculation unit 304 is used to perform calculations and processing on the target data based on data analysis tools to obtain the calculation results of the target data.

[0081] The data processing apparatus provided in Embodiment 2 of this application determines the data to be written by a determining unit 301; a writing unit 302 writes the data to be written into the target shard of the data warehouse according to the sharding data table, resulting in an updated data warehouse. The sharding data table stores the sharding information corresponding to the data in the data warehouse; a query unit 303, upon receiving a data query request, responds to the request and queries the target data in the updated data warehouse according to the sharding data table; and a calculation unit 304 calculates and processes the target data using data analysis tools to obtain the calculation results. This solves the problem in related technologies where, when performing online analytical processing (OLAP) on data in relational databases, the low efficiency of relational databases in real-time processing of massive amounts of data fails to meet the high-timeliness analysis requirements of OLAP. By setting up a sharding data table, massive amounts of data can be evenly stored in various shards of the data warehouse, facilitating the rapid retrieval of the target data required for data query requests within the data warehouse. This allows for rapid analysis and processing of the target data to obtain real-time foreign exchange information from the foreign exchange market, thereby improving the efficiency of real-time statistical analysis of foreign exchange market information.

[0082] Optionally, in the data processing apparatus provided in Embodiment 2 of this application, the apparatus further includes: a configuration unit, configured to configure a target container and create a target process in the target container before writing the data to be written into the target shard of the data warehouse according to the sharded data table, wherein the target container is a container different from the container where the data warehouse is located; and a creation unit, configured to create a write node using the target process and start the write node.

[0083] Optionally, in the data processing apparatus provided in Embodiment 2 of this application, the writing unit 302 includes: a creation subunit for creating a cache queue and writing the data to be written into the cache queue one by one; a first determination subunit for determining the data stored in the cache queue after a first preset time interval to obtain batch data; a second determination subunit for determining the target shard corresponding to each piece of data in the batch data in the data warehouse based on the shard data table; and a writing subunit for using a writing node to write each piece of data in the batch data into the target shard corresponding to that piece of data.

[0084] Optionally, in the data processing apparatus provided in Embodiment 2 of this application, the apparatus further includes: a loading unit, used to load the sharded data table into the Redis cache before querying the target data in the updated data warehouse based on the sharded data table; the query unit 303 includes: an acquisition subunit, used to acquire the sharded data table in the Redis cache through a query node, and acquire the target data in the shards of the data warehouse based on the sharded data table, wherein the query node is used to query data in the data warehouse.

[0085] Optionally, in the data processing apparatus provided in Embodiment 2 of this application, the apparatus further includes: a first update unit, configured to update the sharded data table according to the modification instruction when a target object initiates a modification instruction on the sharded data table before loading the sharded data table into the Redis cache, thereby obtaining an updated sharded data table; and a second update unit, configured to update the sharded data table stored in the Redis cache according to the updated sharded data table, thereby obtaining an updated sharded data table.

[0086] Optionally, in the data processing apparatus provided in Embodiment 2 of this application, the aforementioned writing subunit includes: a first writing module, configured to create a first node and use the first node and the writing node to write each piece of data in the batch data into the target fragment corresponding to that piece of data when the amount of data processed by the writing node within a second preset time period is greater than or equal to a first preset data amount; and a second writing module, configured to delete the first node and use the writing node to write each piece of data in the batch data into the target fragment corresponding to that piece of data when the amount of data processed by the writing node within a second preset time period is less than a second preset data amount.

[0087] Optionally, in the data processing apparatus provided in Embodiment 2 of this application, the aforementioned writing subunit includes: a storage module, configured to store data exceeding the third preset data amount in the message engine system when the amount of data written to the data warehouse within the third preset time period is detected to be greater than or equal to the third preset data amount; and an execution module, configured to use the data stored in the message engine system as the data to be written, and execute the step of writing the data to be written into the target shard of the data warehouse according to the sharding data table, when the amount of data written to the data warehouse within the third preset time period is detected to be less than the fourth preset data amount.

[0088] Optionally, in the data processing apparatus provided in Embodiment 2 of this application, the query unit 303 includes a processing subunit, which is used to record and optimize the query statement if there is a query statement in the query node with a query duration greater than a fourth preset duration.

[0089] The data processing device includes a processor and a memory. The aforementioned determination unit 301, writing unit 302, query unit 303, and calculation unit 304 are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0090] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the efficiency of data retrieval can be improved by adjusting kernel parameters.

[0091] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0092] Embodiment 3 of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a data processing method.

[0093] Embodiment 4 of the present invention provides a processor for running a program, wherein the program executes a data processing method during runtime.

[0094] like Figure 4 As shown, Embodiment 5 of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: determining the data to be written; writing the data to be written into the target shard of the data warehouse according to the sharding data table to obtain an updated data warehouse, wherein the sharding data table stores the sharding information corresponding to the data in the data warehouse; responding to the data query request upon receiving it, querying the target data in the updated data warehouse according to the sharding data table; and performing calculations and processing on the target data using data analysis tools to obtain the calculation result of the target data.

[0095] When the processor executes the program, it also performs the following steps: Before writing the data to be written to the target shard of the data warehouse according to the shard data table, the above method also includes: configuring the target container and creating a target process in the target container, wherein the target container is a different container from the container where the data warehouse is located; using the target process to create a write node and start the write node.

[0096] When the processor executes the program, it also performs the following steps: writing the data to be written to the target shard of the data warehouse according to the shard data table includes: creating a cache queue and writing the data to be written to the cache queue one by one; determining the data stored in the cache queue after a first preset time interval to obtain batch data; determining the target shard in the data warehouse corresponding to each piece of data in the batch data according to the shard data table; and using a write node to write each piece of data in the batch data to the target shard corresponding to that piece of data.

[0097] When the processor executes the program, it also performs the following steps: Before querying the target data in the updated data warehouse based on the sharded data table, the above method further includes: loading the sharded data table into the Redis cache; querying the target data in the updated data warehouse based on the sharded data table includes: obtaining the sharded data table from the Redis cache through the query node, and obtaining the target data from the shards in the data warehouse based on the sharded data table, wherein the query node is used to query data in the data warehouse.

[0098] When the processor executes the program, it also performs the following steps: Before loading the sharded data table into the Redis cache, the above method further includes: if a modification instruction is initiated by the target object on the sharded data table, updating the sharded data table according to the modification instruction to obtain the updated sharded data table; updating the sharded data table stored in the Redis cache according to the updated sharded data table to obtain the updated sharded data table.

[0099] When the processor executes the program, it also performs the following steps: writing each piece of data in the batch data to the target shard corresponding to that piece of data using a write node includes: if the amount of data processed by the write node within a second preset time period is greater than or equal to a first preset data amount, then a first node is created, and the first node and the write node are used to write each piece of data in the batch data to the target shard corresponding to that piece of data; if the amount of data processed by the write node within a second preset time period is less than a second preset data amount, then the first node is deleted, and the write node is used to write each piece of data in the batch data to the target shard corresponding to that piece of data.

[0100] When the processor executes the program, it also performs the following steps: using write nodes to write each piece of data in the batch data to the target shard corresponding to that piece of data, including: if it is detected that the amount of data written to the data warehouse within a third preset time period is greater than or equal to a third preset data amount, then the data exceeding the third preset data amount is stored in the message engine system; if it is detected that the amount of data written to the data warehouse within a third preset time period is less than a fourth preset data amount, then the data stored in the message engine system is used as the data to be written, and the step of writing the data to be written to the target shard of the data warehouse according to the shard data table is executed.

[0101] When the processor executes the program, it also performs the following steps: upon receiving a data query request, it responds to the data query request and queries the target data in the updated data warehouse based on the sharded data table, including: if there is a query statement in the query node with a query duration greater than the fourth preset duration, the query statement is recorded and optimized.

[0102] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0103] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: determining the data to be written; writing the data to be written into the target shard of the data warehouse according to the sharding data table to obtain an updated data warehouse, wherein the sharding data table stores the sharding information corresponding to the data in the data warehouse; responding to a data query request upon receiving the data query request and querying the target data in the updated data warehouse according to the sharding data table; and performing calculations and processing on the target data using data analysis tools to obtain the calculation results of the target data.

[0104] When executed on a data processing device, it is also suitable to execute an initialization procedure with the following steps: before writing the data to be written to the target shard of the data warehouse according to the sharded data table, the above method further includes: configuring the target container and creating a target process in the target container, wherein the target container is a container different from the container where the data warehouse is located; using the target process to create a write node and start the write node.

[0105] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: writing the data to be written to the target shard of the data warehouse according to the shard data table, including: creating a cache queue and writing the data to be written to the cache queue one by one; determining the data stored in the cache queue after a first preset time interval to obtain batch data; determining the target shard in the data warehouse corresponding to each piece of data in the batch data according to the shard data table; and using a write node to write each piece of data in the batch data to the target shard corresponding to that piece of data.

[0106] When executed on a data processing device, it is also suitable to execute an initialization procedure with the following steps: before querying target data in the updated data warehouse based on the sharded data table, the above method further includes: loading the sharded data table into the Redis cache; querying target data in the updated data warehouse based on the sharded data table includes: obtaining the sharded data table from the Redis cache through a query node, and obtaining target data from the shards of the data warehouse based on the sharded data table, wherein the query node is used to query data in the data warehouse.

[0107] When executed on a data processing device, it is also suitable to execute an initialization procedure with the following steps: before loading the sharded data table into the Redis cache, the above method further includes: if a modification instruction is detected from the target object to the sharded data table, updating the sharded data table according to the modification instruction to obtain the updated sharded data table; updating the sharded data table stored in the Redis cache according to the updated sharded data table to obtain the updated sharded data table.

[0108] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: using a write node to write each piece of data in the batch data to the target shard corresponding to that piece of data, including: if it is detected that the amount of data processed by the write node within a second preset time period is greater than or equal to a first preset data amount, then a first node is created, and the first node and the write node are used to write each piece of data in the batch data to the target shard corresponding to that piece of data; if it is detected that the amount of data processed by the write node within a second preset time period is less than a second preset data amount, then the first node is deleted, and the write node is used to write each piece of data in the batch data to the target shard corresponding to that piece of data.

[0109] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: using a write node to write each piece of data in the batch data to the target shard corresponding to that piece of data, including: if it is detected that the amount of data written to the data warehouse within a third preset time period is greater than or equal to a third preset data amount, then the data exceeding the third preset data amount is stored in the message engine system; if it is detected that the amount of data written to the data warehouse within a third preset time period is less than a fourth preset data amount, then the data stored in the message engine system is used as the data to be written, and the step of writing the data to be written to the target shard of the data warehouse according to the shard data table is executed.

[0110] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: upon receiving a data query request, responding to the data query request and querying the target data in the updated data warehouse based on the sharded data table, including: if there is a query statement in the query node with a query duration greater than the fourth preset duration, then the query statement is recorded and optimized.

[0111] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0116] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0117] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0118] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0119] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data processing method, characterized in that, include: Determine the data to be written; The data to be written is written to the target shard of the data warehouse according to the shard data table to obtain the updated data warehouse. The shard data table stores the shard information corresponding to the data in the data warehouse. Before writing the data to be written into the target shard of the data warehouse according to the sharded data table, the method further includes: Configure a target container and create a target process within the target container, wherein the target container is a different container from the container where the data warehouse is located; The target process is used to create a write node and start the write node to write the data to be written into the data warehouse. Upon receiving a data query request, respond to the data query request and query the target data in the updated data warehouse based on the sharded data table; The target data is calculated and processed using data analysis tools to obtain the calculation results for the target data.

2. The method according to claim 1, characterized in that, Writing the data to be written into the target shard of the data warehouse according to the shard data table includes: Create a cache queue, and write the data to be written to the cache queue one by one; After a first preset time interval, the data stored in the cache queue is determined to obtain batch data; Based on the sharded data table, determine the target shard corresponding to each piece of data in the batch data in the data warehouse; The write node is used to write each piece of data in the batch data into the target shard corresponding to that piece of data.

3. The method according to claim 1, Its characteristics are: Before querying the target data in the updated data warehouse based on the sharded data table, the method further includes: loading the sharded data table into the Redis cache; Querying target data in the updated data warehouse based on the sharded data table includes: obtaining the sharded data table from the Redis cache through a query node, and obtaining the target data from the shards of the data warehouse based on the sharded data table, wherein the query node is used to query data in the data warehouse.

4. The method according to claim 3, characterized in that, Before loading the sharded data table into the Redis cache, the method further includes: If a target object initiates a modification instruction on the sharded data table, the sharded data table is updated according to the modification instruction to obtain the updated sharded data table. The sharding data table stored in the Redis cache is updated based on the updated sharding data table to obtain the updated sharding data table.

5. The method according to claim 2, characterized in that, Writing each piece of data in the batch data to the target shard corresponding to that piece of data using the write node includes: If the amount of data processed by the write node within the second preset time period is greater than or equal to the first preset data amount, then a first node is created, and the first node and the write node are used to write each piece of data in the batch data into the target fragment corresponding to that piece of data. If the amount of data processed by the write node within the second preset time period is less than the second preset data amount, then the first node is deleted, and the write node is used to write each piece of data in the batch data into the target shard corresponding to that piece of data.

6. The method according to claim 2, characterized in that, Writing each piece of data in the batch data to the target shard corresponding to that piece of data using the write node includes: If the amount of data written to the data warehouse within a third preset time period is greater than or equal to a third preset data amount, then the data exceeding the third preset data amount will be stored in the message engine system. If the amount of data written to the data warehouse within the third preset time period is less than the fourth preset data amount, then the data stored in the message engine system is used as the data to be written, and the step of writing the data to be written to the target shard of the data warehouse according to the shard data table is executed.

7. The method according to claim 3, characterized in that, Upon receiving a data query request, responding to the data query request and retrieving the target data from the updated data warehouse based on the sharded data table includes: If a query statement with a query duration greater than the fourth preset duration exists in the query node, then the query statement is recorded and optimized.

8. A data processing apparatus, characterized in that, include: The determination unit is used to determine the data to be written. The writing unit is used to write the data to be written into the target shard of the data warehouse according to the shard data table to obtain the updated data warehouse, wherein the shard data table stores the shard information corresponding to the data in the data warehouse; The query unit is used to respond to a data query request upon receiving the data query request and query the target data in the updated data warehouse based on the sharded data table. The calculation unit is used to calculate and process the target data according to the data analysis tools to obtain the calculation result of the target data; The apparatus further includes: a configuration unit, configured to configure a target container and create a target process in the target container before writing the data to be written to the target shard of the data warehouse according to the shard data table, wherein the target container is a container different from the container in which the data warehouse is located; and a creation unit, configured to create a write node using the target process and start the write node to write the data to be written to the data warehouse through the write node.

9. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data processing method according to any one of claims 1 to 7.

Images