Data processing method and system, electronic device and readable storage medium
By mapping the virtual lookup table in the data processing system to the offline real-time data subsystem and splitting the query instructions at preset times, the problem of needing two sets of code to maintain batch and stream processing in the existing technology is solved, and efficient integrated batch and stream data processing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NETEASE (HANGZHOU) NETWORK CO LTD
- Filing Date
- 2022-08-24
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, batch processing and stream processing require maintaining two separate sets of code, resulting in low efficiency in developing data requirements. Furthermore, existing solutions cannot guarantee the performance of both batch and stream processing when maintaining them with a single set of code.
A data processing system is adopted, and the backend service subsystem is configured with a virtual query table that has a mapping relationship with the offline and real-time data subsystems. Data query commands are split at preset times to read data from the offline and real-time data subsystems respectively, and the results are integrated.
It enables users to maintain batch and stream processing with only one set of code, improving development efficiency and ensuring better processing performance and throughput, achieving seamless data processing of both batch and stream processing.
Smart Images
Figure CN115510113B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to a data processing method, system, electronic device, and computer-readable storage medium. Background Technology
[0002] With the rapid development of computer technology, the massive amount of data has brought enormous challenges to data analysis and processing. Currently, there are two main processing requirements for large amounts of data: batch processing (also known as batch computing or offline computing) and stream processing (also known as stream computing or real-time computing). In related technologies, the architecture of the data processing chain mostly adopts a pseudo-batch-stream integrated architecture. In this architecture, batch computing tasks and stream computing tasks need to be processed separately in two separate chains. As a result, developers need to maintain two sets of code for batch processing and stream processing when developing data-related requirements, thus reducing the development efficiency of data requirements. Summary of the Invention
[0003] This application provides a data processing method, system, electronic device, and computer-readable storage medium that enables batch processing and stream processing to be maintained with only one set of code when developing data-related requirements, thereby improving the efficiency of data requirement development. The specific method is as follows.
[0004] In a first aspect, this application provides a data processing method applied to a backend service subsystem of a data processing system. The system further includes an offline data subsystem and a real-time data subsystem. The backend service subsystem, the offline data subsystem, and the real-time data subsystem are interconnected. The backend service subsystem is configured with at least one virtual lookup table, and the at least one virtual lookup table has a mapping relationship with data stored in the offline data subsystem and the real-time data subsystem. The method includes:
[0005] Receive a data query instruction for the virtual query table, wherein the data query instruction carries at least a field value filtering range for a specified time field of the data to be queried;
[0006] In response to the data query instruction, the field value filtering interval is divided into a first time sub-interval before a preset time and a second time sub-interval after the preset time.
[0007] Read first data from the offline data subsystem whose field value of the specified time field satisfies the first time sub-interval, and read second data from the real-time data subsystem whose field value of the specified time field satisfies the second time sub-interval;
[0008] Based on the first data and the second data, output the data query results.
[0009] Secondly, this application also provides a data processing system, which includes a backend service subsystem, an offline data subsystem, and a real-time data subsystem that are interconnected in pairs. The backend service subsystem is configured with at least one virtual query table, and the at least one virtual query table has a mapping relationship with the data stored in the offline data subsystem and the real-time data subsystem.
[0010] The offline data subsystem is used to store data whose field values for a specified time field are prior to a preset time.
[0011] The real-time data subsystem is used to store data whose field values for the specified time field are after the preset time.
[0012] The backend service subsystem is configured to receive a data query instruction for the virtual query table, wherein the data query instruction carries at least a field value filtering range for a specified time field of the data to be queried; in response to the data query instruction, the field value filtering range is divided into a first time sub-range before a preset time and a second time sub-range after the preset time; first data from the offline data subsystem whose field value of the specified time field satisfies the first time sub-range, and second data from the real-time data subsystem whose field value of the specified time field satisfies the second time sub-range; and output the data query result based on the first data and the second data.
[0013] Thirdly, embodiments of this application also provide an electronic device, including:
[0014] Processor; and
[0015] A memory for storing a data processing program, which, when the electronic device is powered on and runs by the processor, performs the method as described in any of the first aspects.
[0016] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a data processing program that is executed by a processor to perform the method described in any of the first aspects.
[0017] Compared with the prior art, this application has the following advantages:
[0018] In the data processing method provided in this application embodiment, after the backend service subsystem receives the user's data query instruction for the virtual query table, it partitions the time filtering conditions carried in the data query instruction by using a preset time as the dividing node, thereby splitting it into a first time sub-interval before the preset time and a second time sub-interval after the preset time. Then, it reads offline data whose field value of the specified time field satisfies the first time sub-interval from the offline data subsystem, and reads real-time data whose field value of the specified time field satisfies the second time sub-interval from the real-time data subsystem, and outputs the query result based on the read data.
[0019] Because the virtual query table configured in the backend service subsystem has a mapping relationship with the data stored in the offline data subsystem and the real-time data subsystem, at the user level, users can perform data query operations on the virtual query table to form query instructions. This allows the data processing system to transform user requirements into dual data source operations for both offline and real-time parts at the underlying implementation level by splitting the data filtering conditions over time. Therefore, in this embodiment, when developing data tasks, users only need to develop and maintain a set of code based on the virtual query table, which improves the development efficiency of data requirements and achieves seamless stream and batch processing without user awareness of separate computation, while ensuring good processing performance and throughput. Attached Figure Description
[0020] Figure 1 This is an architectural block diagram of a data processing system provided in an embodiment of this application;
[0021] Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application;
[0022] Figure 3 This is an architectural block diagram of another data processing system provided in the embodiments of this application;
[0023] Figure 4 This is a flowchart of another data processing method provided in an embodiment of this application;
[0024] Figure 5 This is a flowchart of the third data processing method provided in the embodiments of this application;
[0025] Figure 6 This is a flowchart of the fourth data processing method provided in the embodiments of this application;
[0026] Figure 7 This is a schematic diagram of the logical structure of an electronic device for implementing data processing provided in an embodiment of this application. Detailed Implementation
[0027] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.
[0028] Before detailing the implementation methods of this application, the relevant technologies will be further introduced first.
[0029] With the growing popularity of the integrated stream and batch processing concept, in addition to the pseudo-integrated stream and batch architecture mentioned in the background technology, some other solutions have emerged that aim to balance batch processing and stream processing, as follows:
[0030] Option A: Kappa Architecture. In the Kappa architecture, the integration of stream and batch processing is mainly achieved through stream processing and batch processing, both of which employ stream processing computation methods.
[0031] Option B: Data Lake Storage Technology. In data lake platforms such as Hudi, Iceberg, and Deltaake, data can be stored using a specific data storage format. These formats provide timeline functionality, supporting sequential backtracking or streaming.
[0032] However, although the above solution can be maintained with a single codebase, it still has shortcomings.
[0033] For Option A, the storage design of the Kappa architecture is mainly designed for stream processing, and the storage method has performance bottlenecks for batch processing, random queries, etc.
[0034] For Option B, although data lake storage technology can provide timeline functionality, it has limitations in performance such as real-time read granularity and throughput.
[0035] While solutions A and B can be maintained using a single codebase, they sacrifice batch or stream processing performance to some extent. Therefore, neither the pseudo-batch / stream integrated solution in the background technology, nor solutions A and B, can maintain the data with a single codebase without sacrificing batch / stream processing performance.
[0036] To address the aforementioned problems, this application provides a data processing method applicable to, for example, Figure 1 The data processing system 1000 shown includes a backend service subsystem 1001. The system 1000 also includes an offline data subsystem 1002 and a real-time data subsystem 1003. The backend service subsystem 1001, the offline data subsystem 1002, and the real-time data subsystem 1003 are interconnected.
[0037] In this application, the backend service subsystem 1001 can be an electronic device primarily serving backend developers and other relevant technical personnel, used for developing and maintaining backend data requirements. It has an interactive platform for interacting with the served users, such as... Figure 1 As shown, technicians can use this interactive platform to develop and maintain data requirements. In one specific implementation, the interactive platform can be an SQL (Structured Query Language) interactive platform, which can perform related operations based on SQL statements entered by the user, thus achieving human-computer interaction.
[0038] It should be noted beforehand that in this application, the term "user" will be used to refer to the service targets of the backend service subsystem 1001, including backend developers, testers and other related technical personnel. This application is not intended to limit them.
[0039] In this application, reference is made to Figure 1 The control core of the data processing system 1000 can be an OLAP (Online Analytical Processing) engine, and correspondingly, the back-end service subsystem 1001 can be the engine server corresponding to the OLAP engine.
[0040] The offline data subsystem 1002 can be used to store offline data, which can serve as source data for batch processing. The real-time data subsystem 1003 can be used to store real-time data, which can serve as source data for stream processing. In practical applications, the stored data can be represented by at least one time field to indicate the production time of each piece of data, such as a timestamp field or a datestamp field. Therefore, in this application, the distinction between offline and real-time data can be based on a preset time. If the value of a specified time field (such as a datestamp) of a piece of data is before the preset time, then the data is offline data and needs to be stored in the offline data subsystem 1002; if the value of a specified time field (such as a datestamp) of a piece of data is after the preset time, then the data is real-time data and needs to be stored in the real-time data subsystem 1003.
[0041] It should be noted that in this application, the preset time is an adjustable value, or a value that can change according to set rules. In a specific example, the preset time can be 0:00 of the current day. That is, when the time crosses 0:00, the preset time becomes 0:00 of today, and the data of the entire previous day will be converted from real-time data to offline data. The storage location of the data of the entire previous day will also be migrated from the real-time data subsystem 1003 to the offline data subsystem 1002.
[0042] In one specific implementation, the offline data subsystem 1002 can be an offline data warehouse (data warehouse, or simply data warehouse), such as a Hive data warehouse. Offline data warehouses have advantages for batch processing requirements of offline data.
[0043] In one specific implementation, the real-time data subsystem 1003 can be a real-time data warehouse, such as a Kafka data warehouse. Real-time data warehouses have advantages for meeting the requirements of real-time data streaming.
[0044] In this application, the physical form of the backend service subsystem 1001, the offline data subsystem 1002, and the real-time data subsystem 1003 can be electronic devices, such as single electronic devices or electronic device clusters, and this application does not impose any specific limitations on them.
[0045] It should also be noted that, in order to avoid misunderstanding, in this application, "before XX time" and "after XX time" uniformly refer to excluding XX time itself, and "before XX time" and "after XX time" uniformly refer to including XX time itself. Similar expressions have the same meaning as above.
[0046] Based on the aforementioned data processing system, this application provides a data processing method applied to a backend service subsystem within that system. The backend service subsystem is configured with at least one virtual query table, which is mapped to data stored in both the offline and real-time data subsystems. Thus, by querying the virtual query table, data stored in both the offline and real-time data subsystems can be retrieved.
[0047] It should be noted that the mapping relationship between the at least one virtual query table and the offline and real-time data can be a direct mapping relationship or an indirect mapping relationship.
[0048] like Figure 2 As shown, the method includes the following steps S110 to S140.
[0049] Step S110: Receive a data query instruction for a virtual query table, wherein the data query instruction carries at least a field value filtering range for a specified time field of the data to be queried.
[0050] In this embodiment, users can perform data query operations on any virtual query table through the interactive platform in the backend service subsystem, forming query commands, that is, querying any virtual query table. The interactive platform in the backend service subsystem can provide users with the queryable fields of each virtual query table. Users can input data filtering conditions based on the queryable fields of the virtual query table to obtain offline data and / or real-time data that meet the filtering conditions.
[0051] Since the production time of each data is different, when users input data filtering conditions, they need to provide filtering conditions that can distinguish different production times. In this embodiment of the application, different production times can be represented by different field values of a specified time field (such as a date stamp).
[0052] In practice, different virtual query tables can be distinguished by different table names. When a user performs a data query operation and forms a query command, in addition to the data filtering conditions, they can also enter the name of the virtual query table to be queried, so that the backend service subsystem can determine which virtual query table to query.
[0053] In this embodiment, the data mapped by different virtual lookup tables may not overlap, meaning that different data can be retrieved through different virtual lookup tables. Further optionally, each virtual lookup table can be mapped to a portion of offline data and a portion of real-time data, and the mapping of all virtual lookup tables can cover all offline and real-time data; that is, each piece of offline or real-time data can be retrieved by looking up a specific virtual lookup table.
[0054] The following example illustrates the mapping relationship described above. In one example, all offline data in the offline data subsystem can be divided into n non-overlapping offline datasets, namely offline datasets A1-A2. n All real-time data in the real-time data subsystem can also be divided into n non-overlapping real-time datasets, namely real-time datasets B1-B2. n Accordingly, the backend service subsystem can be configured with n virtual query tables, namely virtual query table T. V1 -T Vn Among them, the virtual lookup table T V1 There is a mapping relationship between the data in offline dataset A1 and real-time dataset B1. That is, any offline data in offline dataset A1 and any real-time data in real-time dataset B1 can be retrieved by querying the virtual lookup table T. V1 The data is read using a query command. Similarly, the virtual query table T is used. V2There is a mapping relationship between the data and the data in offline dataset A2 and real-time dataset B2, and so on, which will not be elaborated here.
[0055] In this way, all offline and real-time data can be managed through virtual lookup tables. At the same time, there is no need to maintain more than two virtual lookup tables for the same data, which reduces the maintenance cost of virtual lookup tables.
[0056] Of course, in practical applications, as data processing needs continue to change, the data mapped by different virtual query tables may overlap, thus satisfying more data processing needs.
[0057] When the backend service subsystem receives a data query instruction for the virtual query table, it executes the following step S120.
[0058] Step S120: In response to the data query command, the field value filtering range is split into a first time sub-range before the preset time and a second time sub-range after the preset time.
[0059] Since the virtual query table is mapped to both offline and real-time data, the backend service subsystem, in response to a data query command, needs to split the field value filtering range of the specified time field of the data to be queried based on the distinguishing node (i.e., the preset time) between offline and real-time data. In other words, it partitions the time filtering conditions entered by the user, thereby splitting the time filtering range for offline and real-time data into a time filtering range for offline data (i.e., the first time sub-range before the preset time) and a time filtering range for real-time data (i.e., the second time sub-range after the preset time).
[0060] Step S130: Read the first data from the offline data subsystem whose field value of the specified time field satisfies the first time sub-interval, and read the second data from the real-time data subsystem whose field value of the specified time field satisfies the second time sub-interval.
[0061] In this step, the backend service subsystem can read the first data from the offline data subsystem whose field value of a specified time field satisfies the first time sub-interval, thereby obtaining the offline data in the user's queried data. The backend service subsystem can also read the second data from the real-time data subsystem whose field value of a specified time field satisfies the second time sub-interval, thereby obtaining the real-time data in the user's queried data.
[0062] Thus, at the user level, users only need to perform a single lookup operation on the virtual lookup table based on data filtering conditions. This allows the data processing system to transform user needs into dual data source operations, including offline and real-time components, at the underlying implementation level through time-based splitting of data filtering conditions. Therefore, in this embodiment, when developing data tasks, users only need to develop and maintain a single set of code based on the virtual lookup table, thereby achieving seamless integration of batch and stream processing without the user's awareness of separate computation.
[0063] In practical applications, the steps of the backend service subsystem reading the first data from the offline data subsystem and the steps of the backend service subsystem reading the second data from the real-time data subsystem can be executed in parallel or in sequence. The choice can be made based on the specific circumstances of the application, and this application does not impose any specific limitations on this.
[0064] Step S140: Output the data query results based on the first data and the second data.
[0065] In this step, the backend service subsystem can integrate the first data and the second data. The integrated data is the data query result, which the backend service subsystem can then output for the user to view. In an optional implementation, the backend service subsystem can display the data query result to the user through a data query result interface.
[0066] In the data processing method provided in this application embodiment, after the backend service subsystem receives the user's data query instruction for the virtual query table, it partitions the time filtering conditions carried in the data query instruction by using a preset time as the dividing node, thereby splitting it into a first time sub-interval before the preset time and a second time sub-interval after the preset time. Then, it reads offline data whose field value of the specified time field satisfies the first time sub-interval from the offline data subsystem, and reads real-time data whose field value of the specified time field satisfies the second time sub-interval from the real-time data subsystem, and outputs the query result based on the read data.
[0067] Because the virtual query table configured in the backend service subsystem has a mapping relationship with the data stored in the offline data subsystem and the real-time data subsystem, at the user level, users can perform data query operations on the virtual query table to form query instructions. This allows the data processing system to transform user requirements into dual data source operations for both offline and real-time parts at the underlying implementation level by splitting the data filtering conditions over time. Therefore, in this embodiment, when developing data tasks, users only need to develop and maintain a set of code based on the virtual query table, which improves the development efficiency of data requirements and achieves seamless stream and batch processing without user awareness of separate computation, while ensuring good processing performance and throughput.
[0068] In one implementation, the data stored in the aforementioned offline data subsystem and real-time data subsystem may be user data of the game application.
[0069] Furthermore, in an optional implementation, the mapping relationship between the at least one virtual query table and the offline and real-time data can be an indirect mapping relationship, referring to... Figure 3 The details are as follows:
[0070] The offline data subsystem 1002 can be configured with at least one offline data query table T. B The at least one offline data query table T B There is a mapping relationship between the data and the data stored in the offline data subsystem 1002;
[0071] The real-time data subsystem 1003 can be configured with at least one real-time data query table T. S The at least one real-time data query table T S There is a mapping relationship between the data and the data stored in the real-time data subsystem 1003;
[0072] Accordingly, virtual lookup table T V Offline data query table T with the same name B and real-time data query table T S There is a mapping relationship. That is, a virtual lookup table is actually mapped to an offline data lookup table and a real-time data lookup table.
[0073] In this implementation, offline data query table T B There is a mapping relationship with offline data, and the virtual query table T V It also has the same name as the offline data query table T. B A mapping relationship exists; therefore, through the two direct mapping relationships mentioned above, a virtual lookup table T is achieved. V The indirect mapping relationship between offline and real-time data. Similarly, real-time data query table T. S There is a mapping relationship with real-time data, and the virtual query table T V It also has the same name as the real-time data query table T. S A mapping relationship exists; therefore, through the two direct mapping relationships mentioned above, a virtual lookup table T is achieved. V The indirect mapping relationship between the data and real-time data. Thus, it is possible to map a specific virtual query table T... V The data query command is converted into offline data query table T with the same table name. B The data query command, and the real-time data query table T with the same table name. S Data query commands.
[0074] In practical applications, both the offline data subsystem and the real-time data subsystem store a large amount of data. Therefore, it is necessary to establish tables for managing their own data to facilitate separate maintenance and data migration of the offline data subsystem and the real-time data subsystem.
[0075] Based on this, in this embodiment, the offline data stored in the offline data subsystem can be accessed through the offline data lookup table T. B For management purposes, the real-time data stored in the real-time data subsystem can be accessed through the real-time data query table T. S For management, the backend service subsystem then uses a virtual query table T. V Query table T for offline data B and real-time data query table T S Management is then implemented. Compared to solutions that directly manage offline and real-time data through virtual lookup tables, this implementation method can directly build a stream-batch integrated solution based on virtual lookup tables, building upon the existing data management methods of the offline and real-time data subsystems. This allows for direct utilization of existing offline and real-time data subsystems, reducing the underlying development of these subsystems, saving development costs for the entire data processing system, and lowering the maintenance difficulty of the data processing system.
[0076] In one optional example, the offline data subsystem can use a Hive data warehouse, and the real-time data subsystem can use a Kafka data warehouse. Correspondingly, an offline data query table can be a Hive table, and a real-time data query table can be a Kafka Topic. That is, a virtual query table can be mapped to a Kafka Topic and a Hive table. Optionally, in this example, the metadata of the virtual query table can be stored and managed in the offline data subsystem (such as a Hive metastore). The Hive metastore is primarily a centralized location for Hive metadata, but it can also be used to store the metadata of virtual query tables.
[0077] In the specific implementation process, in order to facilitate the development and integration with the backend service subsystem, the data format in Kafka Topic can be uniformly defined. For example, the data format in Kafka Topic can be serialized and stored using JSON.
[0078] Based on the above implementation method, the data query instruction further needs to carry the name of the virtual query table to be queried. Through this table name, it can be determined which specific virtual query table T the user wants to query. V Then we can further determine which offline data query table T needs to be queried.B And which real-time data query table T S .
[0079] Reference Figure 4 When the data query instruction also carries the table name of the virtual query table to be queried, the step "reading the field value of the specified time field from the offline data subsystem that satisfies the first data of the first time sub-interval" in step S130 can be implemented through the following steps S131 to S132.
[0080] Step S131: From the offline data query table with the same table name as the virtual query table, query the storage information of the first data whose field value of the specified time field satisfies the first time sub-interval.
[0081] Step S132: Read the first data from the offline data subsystem according to the storage information of the first data.
[0082] In a specific example, a data query instruction can carry a virtual lookup table T to be queried. V1 Given the table name mytable1, the backend service subsystem can query the storage information of the first data whose date stamp meets the first time sub-interval from the offline data query table, which also has the table name mytable1. Based on this storage information, the backend service subsystem can read the first data from the offline data subsystem, which is the offline data part of the data queried by the user.
[0083] In this way, by using a unified table name, the scope of data queries in the offline data subsystem can be narrowed down, and a specific offline data query table T can be quickly located by the table name. B This allows for data retrieval during the process, thereby improving data retrieval efficiency.
[0084] Reference Figure 4 When the data query instruction also carries the table name of the virtual query table to be queried, the step "reading the field value of the specified time field from the real-time data subsystem that satisfies the second data of the second time sub-interval" in step S130 can be implemented through the following steps S133 to S134.
[0085] Step S133: From the real-time data query table with the same table name as the virtual query table, query the storage information of the second data whose field value of the specified time field satisfies the second time sub-interval.
[0086] Step S134: Read the second data from the real-time data subsystem according to the storage information of the second data.
[0087] In a specific example, the data query instruction can carry the table name mytable1 of the virtual query table to be queried. Correspondingly, the backend service subsystem can query the storage information of the second data whose date stamp meets the second time sub-interval from the real-time data query table with the same table name mytable1. Based on this storage information, the second data, that is, the real-time data part of the data queried by the user, can be read from the real-time data subsystem.
[0088] In this way, by using standardized table names, the scope of data queries within the real-time data subsystem can be narrowed down, allowing for quick targeting of a specific real-time data query table T by table name. S This allows for data retrieval during the process, thereby improving data retrieval efficiency.
[0089] It should be noted that in practical applications, in addition to the time-based filtering conditions and virtual query table names mentioned in the above embodiments, data query commands can also carry other more specific filtering conditions, such as a specific application account, to filter out relevant data for that application account. This application does not specifically limit the other filtering conditions carried in the data query command; users can input them according to their actual needs.
[0090] In the above implementation, the reason why the three tables in the three subsystems can be associated through the same table name is that the creation, data writing, deletion and other processes of the three tables are all associated through the same table name. This enables the data query process to associate a virtual query table in the backend service subsystem with an offline data query table in the offline data subsystem and a real-time data query table in the real-time data subsystem through the table name of a virtual query table in the backend service subsystem.
[0091] The following details the creation, data writing, and deletion processes of virtual query tables, offline data query tables, and real-time data query tables to demonstrate the relationships between these three types of tables with the same name.
[0092] I. The table creation process S150 includes the following steps S151 to S152:
[0093] Step S151: Receive a virtual lookup table creation instruction, wherein the virtual lookup table creation instruction carries at least the table name of the virtual lookup table to be created.
[0094] The backend service subsystem can receive virtual query table creation instructions from users through the interactive platform. Users need to provide the name of the virtual query table to be created, for example, mytable1, which means that the user wants to create a virtual query table named mytable1.
[0095] Step S152: In response to the virtual query table creation instruction, create a virtual query table with the table name, create an offline data query table with the table name in the offline data subsystem, and create a real-time data query table with the table name in the real-time data subsystem.
[0096] In response to the virtual query table creation command, the backend service subsystem can create a virtual query table named `mytable1` at the user level. At the underlying implementation level, it can also create an offline data query table named `mytable1` in the offline data subsystem and a real-time data query table named `mytable1` in the real-time data subsystem. Thus, three tables with the same name exist in the three subsystems. The creation of the virtual query table named `mytable1` is mapped at the underlying implementation to the creation of both an offline data query table named `mytable1` and a real-time data query table named `mytable1`, while the user remains unaware of these two separate underlying creation operations.
[0097] In one specific embodiment, the offline data query table is a Hive table, and the real-time data query table is a KafkaTopic. When it is necessary to create a virtual query table named mytable1, the backend service subsystem will automatically create a Hive table named mytable1 in the Hive data warehouse and a Topic named mytable1 in the Kafka data warehouse.
[0098] II. The data writing process for the table, S160, includes the following steps S161 to S164:
[0099] Step S161: Receive a data write instruction for the virtual query table, wherein the data write instruction carries at least the table name of the virtual query table and the field value of the specified time field of the target data to be written.
[0100] The backend service subsystem can receive data write commands from users for a specific virtual query table through the interactive platform. Users need to provide the name of the virtual query table to be written, for example, `mytable1`, meaning the user wants to write data to the virtual query table named `mytable1`. Users also need to provide the target data to be written, especially the value of the specified time field. The backend service subsystem then needs to determine, based on this field value, whether to write the target data to the offline data query table named `mytable1` or the real-time data query table named `mytable1`.
[0101] Step S162: In response to the data write instruction, if the field value is before a preset time, store the target data in the offline data subsystem and write the storage information of the target data into the offline data query table with the table name.
[0102] The target data refers to any data that meets the write requirements.
[0103] In this step, the backend service subsystem parses the information carried by the data write command to obtain the name of the virtual query table for the data to be written, as well as the target data to be written and the field value of its specified time field. Then, the backend service subsystem can determine whether to write the target data to an offline data query table with the same name as the virtual query table or to a real-time data query table with the same name as the virtual query table based on the relationship between the field value and the current preset time.
[0104] If the field value is before the preset time, the backend service subsystem will store the target data in the offline data subsystem and write the storage information of the target data into the offline data query table with the same table name as the virtual query table.
[0105] In one specific embodiment, the specified time field is a timestamp field, and the current preset time is 0:00:00 on 2022-06-23. The data write instruction carries the table name mytable1 of the virtual query table and the timestamp field value of the target data to be written, 2022-06-21, 10:00:00. Since this timestamp field value is before the current preset time, the backend service subsystem will store the target data in the offline data subsystem as offline data before 0:00 on 2022-06-23, and write the storage information of the target data into the offline data query table named mytable1.
[0106] Step S163: In response to the data write instruction, if the field value is after a preset time, store the target data in the real-time data subsystem and write the storage information of the target data into the real-time data query table with the table name.
[0107] If the value of this field is after a preset time, the backend service subsystem will store the target data in the real-time data subsystem and write the storage information of the target data into the real-time data query table with the same table name as the virtual query table.
[0108] In one specific embodiment, the specified time field is a timestamp field, and the current preset time is 0:00:00 on 2022-06-23. The data write instruction carries the table name mytable1 of the virtual query table and the timestamp field value of the target data to be written, 2022-06-23,10:00:00. Since the timestamp field value is after the current preset time, the backend service subsystem will store the target data in the real-time data subsystem as real-time data after 0:00 on 2022-06-23, and write the storage information of the target data into the offline data query table named mytable1.
[0109] Step S164: Synchronize the storage information of the target data to the virtual query table with the table name to update the metadata of the virtual query table.
[0110] In this embodiment, the virtual lookup table is directly user-facing. Therefore, after writing the storage information of the target data into the offline or real-time data lookup table, it is also necessary to synchronize the storage information of the target data into the virtual lookup table with the same table name. This enables data writing to the virtual lookup table at the user level.
[0111] Similarly, when it is necessary to delete a certain data in a virtual query table, the data will be deleted simultaneously in the offline data query table and the real-time data query table with the same table name as the virtual query table. This will not be repeated in the embodiments of this application.
[0112] III. The table deletion process S170 includes the following steps S171 to S172:
[0113] Step S171: Receive a delete instruction for the target virtual lookup table, wherein the delete instruction carries at least the table name of the target virtual lookup table.
[0114] The target virtual query table refers to any virtual query table in the backend service subsystem.
[0115] Similar to the creation process of a virtual query table, the backend service subsystem can receive deletion instructions from users for a target virtual query table through an interactive platform. The user needs to provide the name of the virtual query table to be deleted, for example, mytable1, which means that the user wants to delete a virtual query table named mytable1.
[0116] Step S172: In response to the deletion command, delete the target virtual query table, delete the offline data query table with the same table name as the target virtual query table from the offline data subsystem, and delete the real-time data query table with the same table name as the target virtual query table from the real-time data subsystem.
[0117] The backend service subsystem responds to the delete command for the target virtual query table. At the user level, it can delete the virtual query table named mytable1. At the underlying implementation level, it can delete the offline data query table with the same name mytable1 in the offline data subsystem, and the real-time data query table with the same name mytable1 in the real-time data subsystem. In this way, three tables with the same name are deleted in the three subsystems respectively. The deletion of the virtual query table named mytable1 is mapped at the underlying implementation to the deletion of an offline data query table named mytable1 and a real-time data query table named mytable1, while the user is unaware of the two separate underlying deletion operations.
[0118] In one specific embodiment, the offline data query table is a Hive table, and the real-time data query table is a KafkaTopic. When it is necessary to delete the virtual query table named mytable1, the backend service subsystem will automatically delete the Hive table named mytable1 in the Hive data warehouse and the Topic named mytable1 in the Kafka data warehouse.
[0119] In practical applications, due to the existence of both batch processing and stream processing data processing requirements, data can be read in two ways: batch reading (or simply batch read) and stream reading (or simply stream read). Batch reading is suitable when a calculation result for a specific time period is needed. For example, calculating the total number of logins for a game player within the last 24 hours up to the current moment is a fixed value, making batch reading suitable. Stream reading is suitable when a continuous calculation result is needed. For example, continuously calculating the total revenue of an application game project, where the total revenue changes constantly, requires continuously calculating the changing result value, making stream reading suitable.
[0120] In this embodiment of the application, the user can select a suitable data reading method according to actual needs.
[0121] Therefore, in another optional implementation, the data processing system can provide users with the option to select between batch read mode and streaming read mode, allowing users to choose between batch read and streaming read modes according to their actual needs. For batch read mode, the backend service subsystem can also support the output of static query results, that is, output the batch calculation results once. For streaming read mode, the backend service subsystem can also support the output of dynamic query results, that is, continuously output the latest streaming calculation results.
[0122] In practical applications, users can optionally set either batch read mode or streaming read mode as the default data reading method.
[0123] In one specific embodiment, taking the backend service subsystem using an SQL platform as an example, users can use an SQL statement such as "SET RUNTIME-MODE='Streaming'" to switch to batch read mode, thereby switching the data reading method to batch read mode; users can also use an SQL statement such as "SET RUNTIME-MODE='Batch'" to switch to streaming read mode, thereby switching the data reading method to streaming read mode.
[0124] It should be noted that the above SQL statement is only an example and does not constitute a limitation on this application.
[0125] In one implementation, reference is made to Figure 5 When the data reading method is configured as batch reading and the data query instruction does not carry the end time node of the field value filtering range, step S133 may specifically include the following step S1331.
[0126] Step S1331: From the real-time data query table with the same table name as the virtual query table, query the second data stored in the real-time data subsystem before the time the data query instruction was received, and whose field value of the specified time field satisfies the second time sub-interval.
[0127] In this implementation, when the data reading method is configured as batch reading, when a user wants to query the batch calculation results from a certain time to the current time, the user only needs to enter the start time node of the field value filtering interval, without needing to enter the end time node of the field value filtering interval. Thus, the backend service subsystem can, based on the batch reading method, default the end time node of the field value filtering interval to the time of receiving the data query instruction, and then read the data from the start time node to the time of receiving the instruction.
[0128] Accordingly, such as Figure 5 As shown, step S140 can be achieved through the following steps S141 to S142.
[0129] Step S141: Integrate the first data and the second data to obtain the data query results.
[0130] In this step, the backend service subsystem can integrate the first data read from the offline data subsystem and the second data read from the real-time data subsystem through a concatenation operation (such as a JOIN operation in SQL) to obtain the integrated data, which is the data query result. Specifically, the concatenation can be performed sequentially according to the field values of a specified time field, such as concatenating according to the date stamp order.
[0131] Step S142: Display the data query results interface, which includes the data query results.
[0132] In this step, after the backend service subsystem integrates and obtains the data query results, it can display the data query results interface to show the data query results to the user.
[0133] In another implementation, refer to Figure 6 If the data reading method is configured as streaming reading and the data query instruction does not carry the end time node of the field value filtering range, the following steps S181 to S182 may be included after step S142.
[0134] Step S181: Monitor the latest data stored in the real-time data subsystem after the data query command is received.
[0135] Step S182: When the field value of the specified time field of the latest data meets the second time sub-interval, read the latest data from the real-time data subsystem.
[0136] In this implementation, when the data reading method is configured as streaming, when a user wants to query the streaming calculation results from a certain time up to the latest time, the user can also only enter the start time node of the field value filtering interval, without having to enter the end time node of the field value filtering interval. Thus, the backend service subsystem can, based on the streaming reading method, default the end time node of the field value filtering interval to the latest time, rather than the time when the data query command is received, and then continuously read the data from the start time node to the latest time.
[0137] If the streaming read mode is adopted, the data computing task will run in the form of a streaming task for a long time. Therefore, in addition to the data read in step S1331, once new data enters the real-time data subsystem after the user submits a query, the backend service subsystem will also read it out and process it immediately.
[0138] Accordingly, such as Figure 6 As shown, after step S182, the following steps S191 to S192 may also be included.
[0139] Step S191: Integrate the latest data with the data query results to obtain updated data query results.
[0140] In stream read mode, once new data is stored in the real-time data subsystem after a user submits a query, the backend service subsystem can integrate the latest data with the data query results obtained from the previous calculation in this stream task to obtain the latest data query results.
[0141] Step S192: Refresh the data query results interface based on the updated data query results.
[0142] In this step, the backend service subsystem can refresh the data query results interface based on the updated data query results, so that the refreshed data query results interface displays the latest data query results.
[0143] After step S142, the latest data query results can be continuously calculated and the data query result interface can be refreshed in a timely manner until the user actively stops the streaming task.
[0144] In the two implementation methods described above, batch reading mode supports the calculation and output of static query results, while streaming reading mode supports the dynamic calculation and output of dynamic query results. Compared to solutions A and B mentioned earlier, which sacrifice streaming or batch processing performance, the data processing method provided in this application embodiment can maintain the data with a single codebase without sacrificing streaming and batch processing performance. The streaming and batch processing performance and throughput of this data processing method are not significantly different from existing solutions that develop streaming and batch processing separately.
[0145] The following section provides a detailed introduction to the human-computer interaction methods between users and the backend service subsystem.
[0146] In one alternative implementation, step S110 can be specifically implemented through the following steps S111 to S112.
[0147] Step S111: Display the data query interface, which includes the queryable fields of each virtual query table.
[0148] In this step, the backend service subsystem can display a data query interface as an interactive interface for users to perform data queries. This interface displays the searchable fields of each virtual query table, allowing users to input filter criteria and other necessary elements for their queries, such as the table name.
[0149] Step S112: Receive the input data query statement in the data query interface. The data query statement includes the table name of the virtual query table to be queried and the field value filtering range of the specified time attribute of the data to be queried.
[0150] Users can organize the table name of the virtual query table to be queried, the field value filter range of the specified time attribute of the data to be queried, and other elements required for the query into a data query statement. This data query statement is a data query statement supported by the backend service subsystem.
[0151] In one specific implementation, the backend service subsystem can support SQL statements, allowing users to input specific SQL query statements in the data query interface to perform data queries.
[0152] In this implementation, the backend service subsystem can provide a visual data query interface to interact with users. Users can interact with the backend service subsystem by inputting data query statements. This hides the underlying interactions with the offline data subsystem and the real-time data subsystem at the user level, making user operations more convenient and improving the user experience.
[0153] Based on the above implementation method, step S120 can be further implemented by the following steps S121 to S122.
[0154] Step S121: In response to the input of the data query statement, parse the data query statement to obtain the table name and field value filtering range.
[0155] In this step, the backend service subsystem is configured with a data query statement parsing function, which can parse the data query statement entered by the user, thereby obtaining the table name of the virtual query table to be queried, and the field value filtering range of the specified time field of the queried data.
[0156] In one specific implementation, the backend service subsystem's interaction platform can support SQL statements. Users can input specific SQL query statements in the data query interface. The interaction platform can be configured with an SQL analyzer. The backend service subsystem can receive the SQL query statements input by users through the data query interface and analyze the SQL syntax through the SQL analyzer, thereby parsing out elements such as the virtual query table name and the field value filtering range of the specified time attribute of the data to be queried in the SQL query statement.
[0157] Step S122: Divide the field value filtering range into a first time sub-range before the preset time and a second time sub-range after the preset time.
[0158] In this step, the backend service subsystem splits the parsed field value filtering range into a first time sub-range before a preset time and a second time sub-range after a preset time. Then, it reads the first data from the offline data subsystem whose field value of the specified time field satisfies the first time sub-range, and the second data from the real-time data subsystem whose field value of the specified time field satisfies the second time sub-range.
[0159] For example, a user can enter a data query statement including the following: "SELECT * FROM mytable1 WHERE ds >= 2022-06-28" (assuming the query date is 2022-07-01), aiming to retrieve data from the virtual query table named mytable1, with ds (date stamp) from 2022-06-28 to the current time. At the preset time of 0:00 on 2022-07-01, the data from 2022-06-28, 2022-06-29, and 2022-06-30 (i.e., the first data) needs to be read from the offline data subsystem using the offline data query table named mytable1. The data from 0:00 on 2022-07-01 to the current time (i.e., the second data) needs to be read from the real-time data subsystem using the real-time data query table named mytable1.
[0160] In this implementation, the backend service subsystem can obtain the field value filtering range by parsing the data query statement entered by the user, and then partition the field value filtering range into time segments according to a preset time. This eliminates the need for the user to perform time partitioning and then separately query the data using two data query statements, making the user operation more convenient and improving the user experience.
[0161] Corresponding to the data processing method provided in the first embodiment of this application, the second embodiment of this application provides a data processing system. For example... Figure 1 and Figure 3 As shown, the system 1000 includes a backend service subsystem 1001, an offline data subsystem 1002, and a real-time data subsystem 1003 that are interconnected in pairs. The backend service subsystem 1001 is configured with at least one virtual query table, and the at least one virtual query table has a mapping relationship with the data stored in the offline data subsystem 1002 and the real-time data subsystem 1003.
[0162] The offline data subsystem 1002 is used to store data whose field values for a specified time field are before a preset time.
[0163] The real-time data subsystem 1003 is used to store data whose field values of the specified time field are after the preset time.
[0164] The backend service subsystem 1001 is configured to receive a data query instruction for the virtual query table, wherein the data query instruction carries at least a field value filtering range for a specified time field of the data to be queried; in response to the data query instruction, the field value filtering range is divided into a first time sub-range before a preset time and a second time sub-range after the preset time; first data from the offline data subsystem 1002 whose field value of the specified time field satisfies the first time sub-range, and second data from the real-time data subsystem 1003 whose field value of the specified time field satisfies the second time sub-range; and output the data query result based on the first data and the second data.
[0165] Corresponding to the data processing method provided in the first embodiment of this application, the third embodiment of this application also provides an electronic device for processing data. For example... Figure 7 As shown, the electronic device includes: a processor 701; and a memory 702 for storing a program for a data processing method. After the device is powered on and the program for the data processing method is run by the processor, the following steps are performed:
[0166] Receive a data query instruction for the virtual query table, wherein the data query instruction carries at least a field value filtering range for a specified time field of the data to be queried;
[0167] In response to the data query instruction, the field value filtering interval is divided into a first time sub-interval before a preset time and a second time sub-interval after the preset time.
[0168] Read first data from the offline data subsystem whose field value of the specified time field satisfies the first time sub-interval, and read second data from the real-time data subsystem whose field value of the specified time field satisfies the second time sub-interval;
[0169] Based on the first data and the second data, output the data query results.
[0170] Corresponding to the data processing method provided in the first embodiment of this application, the fourth embodiment of this application provides a computer-readable storage medium storing a program for a data processing method, which is executed by a processor to perform the following steps:
[0171] Receive a data query instruction for the virtual query table, wherein the data query instruction carries at least a field value filtering range for a specified time field of the data to be queried;
[0172] In response to the data query instruction, the field value filtering interval is divided into a first time sub-interval before a preset time and a second time sub-interval after the preset time.
[0173] Read first data from the offline data subsystem whose field value of the specified time field satisfies the first time sub-interval, and read second data from the real-time data subsystem whose field value of the specified time field satisfies the second time sub-interval;
[0174] Based on the first data and the second data, output the data query results.
[0175] It should be noted that for a detailed description of the data processing system, electronic device and computer-readable storage medium provided in the embodiments of this application, please refer to the relevant description of the data processing method in the first embodiment of this application, which will not be repeated here.
[0176] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.
[0177] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0178] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0179] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage media, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.
[0180] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0181] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.
Claims
1. A data processing method, characterized in that, A backend service subsystem is applied to a data processing system. The system further includes an offline data subsystem and a real-time data subsystem. The backend service subsystem, the offline data subsystem, and the real-time data subsystem are interconnected. Each backend service subsystem is configured with at least one virtual lookup table, and these virtual lookup tables are mapped to data stored in the offline data subsystem and the real-time data subsystem. The method includes: Receive a data query instruction for the virtual query table, wherein the data query instruction carries at least a field value filtering range for a specified time field of the data to be queried; In response to the data query instruction, the field value filtering interval is divided into a first time sub-interval before a preset time and a second time sub-interval after the preset time. Read first data from the offline data subsystem whose field value of the specified time field satisfies the first time sub-interval, and read second data from the real-time data subsystem whose field value of the specified time field satisfies the second time sub-interval; Based on the first data and the second data, output the data query results; The offline data subsystem is configured with at least one offline data query table, and the at least one offline data query table has a mapping relationship with the data stored in the offline data subsystem; The real-time data subsystem is configured with at least one real-time data query table, and the at least one real-time data query table has a mapping relationship with the data stored in the real-time data subsystem. The virtual query table is mapped to the offline data query table and the real-time data query table, which have the same table name.
2. The method according to claim 1, characterized in that, The data query instruction also carries the name of the virtual query table to be queried, and the step of reading the field value of the specified time field from the offline data subsystem that satisfies the first data of the first time sub-interval includes: From the offline data query table with the same table name as the virtual query table, query the storage information of the first data in the first time sub-interval whose field value of the specified time field satisfies the first time sub-interval; Based on the storage information of the first data, the first data is read from the offline data subsystem.
3. The method according to claim 1, characterized in that, The data query instruction also carries the name of the virtual query table to be queried, and the step of reading the second data from the real-time data subsystem whose field value of the specified time field satisfies the second time sub-interval includes: From the real-time data query table with the same table name as the virtual query table, query the storage information of the second data whose field value of the specified time field satisfies the second time sub-interval; The second data is read from the real-time data subsystem based on the storage information of the second data.
4. The method according to claim 1, characterized in that, Before receiving the data query instruction for the virtual query table, the method further includes: Receive a virtual query table creation instruction, wherein the virtual query table creation instruction carries at least the table name of the virtual query table to be created; In response to the virtual query table creation instruction, a virtual query table is created with the table name, an offline data query table is created with the table name in the offline data subsystem, and a real-time data query table is created with the table name in the real-time data subsystem.
5. The method according to claim 4, characterized in that, After responding to the virtual query table creation instruction, creating a virtual query table with the table name, creating an offline data query table with the table name in the offline data subsystem, and creating a real-time data query table with the table name in the real-time data subsystem, the method further includes: Receive a data write instruction for the virtual query table, wherein the data write instruction carries at least the table name of the virtual query table and the field value of the specified time field of the target data to be written; In response to the data write instruction, if the field value is before the preset time, the target data is stored in the offline data subsystem, and the storage information of the target data is written into the offline data query table with the table name; In response to the data write instruction, if the field value is after the preset time, the target data is stored in the real-time data subsystem, and the storage information of the target data is written into the real-time data query table with the table name. The storage information of the target data is synchronized to the virtual query table with the table name to update the metadata of the virtual query table.
6. The method according to claim 4, characterized in that, After responding to the virtual query table creation instruction, creating a virtual query table with the table name, creating an offline data query table with the table name in the offline data subsystem, and creating a real-time data query table with the table name in the real-time data subsystem, the method further includes: Receive a deletion instruction for a target virtual query table, wherein the deletion instruction carries at least the table name of the target virtual query table; In response to the deletion instruction, the target virtual query table is deleted, and an offline data query table with the same name as the target virtual query table is deleted from the offline data subsystem, and a real-time data query table with the same name as the target virtual query table is deleted from the real-time data subsystem.
7. The method according to claim 3, characterized in that, When the data reading method is configured for batch reading and the data query instruction does not carry the end time node of the field value filtering interval, the step of querying the storage information of the second data whose field value of the specified time field satisfies the second time sub-interval from the real-time data query table with the same table name as the virtual query table includes: From the real-time data query table with the same table name as the virtual query table, query the second data stored in the real-time data subsystem before the time the data query instruction was received, and whose field value of the specified time field satisfies the second time sub-interval.
8. The method according to claim 7, characterized in that, The step of outputting data query results based on the first data and the second data includes: The first data and the second data are integrated to obtain the data query results; The data query results interface displays the data query results.
9. The method according to claim 8, characterized in that, When the data reading method is configured as streaming and the data query instruction does not carry the end time node of the field value filtering range, after displaying the data query result interface including the data query result, the following is also included: Real-time monitoring of the latest data stored in the real-time data subsystem after the moment the data query command is received; When the field value of the specified time field of the latest data satisfies the second time sub-interval, the latest data is read from the real-time data subsystem.
10. The method according to claim 9, characterized in that, When the field value of the specified time field of the latest data satisfies the second time sub-interval, after reading the latest data from the real-time data subsystem, the method further includes: The latest data is integrated with the data query results to obtain updated data query results; Refresh the data query results interface based on the updated data query results.
11. The method according to claim 1, characterized in that, Receiving a data query instruction for the virtual query table includes: Display a data query interface, wherein the data query interface includes the queryable fields of each of the virtual query tables; The data query interface receives input data query statements, which include the name of the virtual query table to be queried and the field value filtering range of the specified time attribute of the data to be queried.
12. The method according to claim 11, characterized in that, The step of responding to the data query instruction by splitting the field value filtering interval into a first time sub-interval before a preset time and a second time sub-interval after the preset time includes: In response to the input of the data query statement, the data query statement is parsed to obtain the table name and the field value filtering range; The field value filtering range is divided into a first time sub-range before a preset time and a second time sub-range after the preset time.
13. A data processing system, characterized in that, The system includes a backend service subsystem, an offline data subsystem, and a real-time data subsystem that are interconnected in pairs. The backend service subsystem is configured with at least one virtual query table, and the at least one virtual query table has a mapping relationship with the data stored in the offline data subsystem and the real-time data subsystem. The offline data subsystem is used to store data whose field values for a specified time field are prior to a preset time. The real-time data subsystem is used to store data whose field values for the specified time field are after the preset time. The backend service subsystem is configured to receive a data query instruction for the virtual query table, wherein the data query instruction carries at least a field value filtering range for a specified time field of the data to be queried; in response to the data query instruction, the field value filtering range is divided into a first time sub-range before a preset time and a second time sub-range after the preset time; first data from the offline data subsystem whose field value of the specified time field satisfies the first time sub-range, and second data from the real-time data subsystem whose field value of the specified time field satisfies the second time sub-range; and output the data query result based on the first data and the second data. The offline data subsystem is configured with at least one offline data query table, and the at least one offline data query table has a mapping relationship with the data stored in the offline data subsystem; The real-time data subsystem is configured with at least one real-time data query table, and the at least one real-time data query table has a mapping relationship with the data stored in the real-time data subsystem. The virtual query table is mapped to the offline data query table and the real-time data query table, which have the same table name.
14. An electronic device, characterized in that, include: processor; as well as A memory for storing a data processing program, which, when the electronic device is powered on and runs through the processor, executes the method as described in any one of claims 1-12.
15. A computer-readable storage medium, characterized in that, The system contains a data processing program that is executed by a processor to perform the method as described in any one of claims 1-12.