Data query method and apparatus

Abstract

The application provides a data query method, comprising: generating target metadata in the process of data writing to a target partition; wherein the target metadata comprises partition attribute information, and the partition attribute information indicates that the target partition supports real-time query and a current writing state; receiving a query request and determining the type of the query request; and in the case that the query request is a real-time query request, obtaining a corresponding query result from the target partition. In the case that the target partition has not been completely written, if a real-time query request is received, a query operation is performed based on the data already existing in the target partition. When data is continuously written, the query can obtain the latest content. That is, the technical solution of the application can make real-time query based on the data already existing in the target partition, is suitable for a business that does not require complete data but has high real-time requirements, and has high timeliness.

Description

Technical Field

[0001] This application relates to the field of databases, and more particularly to a data query method, apparatus, computer device, and computer-readable storage medium. Background Technology

[0002] With the development of computer and big data technologies, data storage and utilization have become key technological focuses. In many application scenarios, the more real-time the data, the more valuable it is. For example, near real-time data is needed for data analysis in scenarios such as live streaming, recommendation systems, and content moderation. However, traditional data warehouses / data lakes have the following problems:

[0003] Traditional data warehouses are based on hourly / daily partitions, requiring complete data for querying. However, some users don't need complete data, only the most recent data for trend analysis. Traditional data warehouses / data lakes struggle to meet these time-sensitive data requirements. Summary of the Invention

[0004] The purpose of this application is to provide a data query method, apparatus, computer device, and computer-readable storage medium to solve the above-mentioned problems.

[0005] One aspect of this application provides a data query method, the method comprising:

[0006] During the process of writing data to the target partition, target metadata is generated; wherein, the target metadata includes partition attribute information, which indicates that the target partition supports real-time query and current write status;

[0007] Receive a query request and determine the type of the query request; and

[0008] In the case where the query request is a real-time query request, the corresponding query results are obtained from the target partition.

[0009] Optionally, the method further includes:

[0010] In the case where the query request is an offline query request, the current write status is determined based on the partition attribute information; wherein, the current write status includes whether the target partition has completed writing or has not completed writing;

[0011] If the current write status is that the write is not complete, then the offline query request will not be responded to.

[0012] Optionally, the method further includes:

[0013] If the current write status is "write completed", the corresponding query results are obtained from the target partition.

[0014] Optionally, determining the type of the query request includes:

[0015] If the query statement in the query request includes a target identifier, the query request is determined to be the real-time query request;

[0016] If the query statement in the query request does not include the target identifier, the query request is determined to be an offline query request.

[0017] Optionally, the process of generating target metadata during data writing to the target partition includes:

[0018] The current write status is determined based on the data processing progress of the target partition;

[0019] The target metadata is generated based on the current write status, the partition path of the target partition, and the partition name.

[0020] Another aspect of the embodiments of this application provides a data query apparatus, the apparatus comprising:

[0021] The generation module is used to generate target metadata during the process of writing data to the target partition; wherein, the target metadata includes partition attribute information, which indicates that the target partition supports real-time query and current write status;

[0022] A determination module is used to receive a query request and determine the type of the query request; and

[0023] The acquisition module is used to acquire the corresponding query results from the target partition when the query request is a real-time query request.

[0024] Another aspect of this application provides a data query method, the method comprising:

[0025] During the process of writing data into the HUDI table, a target watermark is provided, which represents the processing progress;

[0026] Provide Instant information, wherein the Instant information is used to record the operation information of the data during the writing process;

[0027] Based on the target watermark and the Instant information, generate target Instant information; and

[0028] The Instant information is submitted to the database and set as a node on a preset timeline for querying.

[0029] Optionally, providing a target watermark during the data writing process to the HUDI table includes:

[0030] The data is written into the HUDI table through multiple data writing components;

[0031] Multiple watermarks are provided through the plurality of streaming data writing components; wherein, a watermark is provided when the snapshot state of any one of the plurality of streaming data writing components is triggered; and

[0032] Determine the target watermark, wherein the target watermark is the minimum value of the plurality of watermarks.

[0033] Optionally, generating target Instant information based on the target watermark and the Instant information further includes:

[0034] The target Instant information is generated based on the target watermark, the Instant information, and the checkpoint information.

[0035] Optionally, the method further includes:

[0036] In response to a progress query request, retrieve the target Instant information from the time sample.

[0037] Extract the target watermark from the target Instant information and return response information based on the target watermark.

[0038] Another aspect of the embodiments of this application provides a data query apparatus, the apparatus comprising:

[0039] The first module provides a target watermark during the process of writing data into the HUDI table, whereby the target watermark represents the processing progress.

[0040] The second providing module provides Instant information, wherein the Instant information is used to record the operation information of the data during the writing process;

[0041] The generation module generates target Instant information based on the target watermark and the Instant information; and

[0042] The submission module submits the Instant information to the database and sets it as a node on a preset timeline for querying.

[0043] Another aspect of this application provides a computer device, the computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data query method described above.

[0044] Another aspect of this application provides a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the data query method described above.

[0045] The data query method, apparatus, computer equipment, and computer-readable storage medium provided in this application have the following technical advantages:

[0046] If a real-time query request is received before the target partition has been fully written, the query operation will be performed based on the existing data in the target partition. As data continues to be written, the query can retrieve the latest content. Therefore, the technical solution proposed in this application can perform real-time queries based on the existing data in the target partition, making it suitable for businesses that do not require complete data but have high real-time requirements (such as trend analysis), and offering high timeliness. Attached Figure Description

[0047] Figure 1 A schematic diagram illustrating an environment according to an embodiment of this application is shown.

[0048] Figure 2 A flowchart illustrating a data query method according to Embodiment 1 of this application is shown schematically;

[0049] Figure 3 Schematic illustration Figure 2 Sub-flowchart of step S200;

[0050] Figure 4 The illustration shows a new step in the data query method according to Embodiment 1 of this application;

[0051] Figure 5 A flowchart illustrating the data query method according to Embodiment 2 of this application is shown schematically;

[0052] Figure 6 The diagram illustrates the operation flowchart of the data query method according to Embodiment 2 of this application in an exemplary application.

[0053] Figure 7 A schematic diagram of the timeline is shown.

[0054] Figure 8 A block diagram of a data query device according to Embodiment 3 of this application is shown schematically;

[0055] Figure 9 A block diagram of a data query device according to Embodiment 4 of this application is schematically shown; and

[0056] Figure 10 The illustration shows a schematic diagram of the hardware architecture of a computer device suitable for implementing a data query method according to Embodiment 5 of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present application and are not intended to limit the present application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0058] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0059] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0060] The following are explanations of some terms used in this application:

[0061] Flink is a distributed real-time computing system used for stateful computation on unbounded and bounded data streams. Flink is designed to run in all common cluster environments, performing computations at memory speeds and any scale. Here, "state" generally refers to the state of a specific task / operator.

[0062] HDFS (Hadoop Distributed File System) is the Hadoop Distributed File System.

[0063] Stream processing is used to process (e.g., query) streaming data. Streaming data is typically online data and has no boundaries.

[0064] SQL (Structured Query Language) is a database language with multiple functions, including data manipulation and definition. This interactive language provides great convenience to users, and database management systems should fully utilize SQL to improve the quality and efficiency of computer application systems. SQL can not only be used independently in the terminal but also serve as a sub-language to effectively assist in other program design. In program applications, SQL can work with other programming languages ​​to optimize program functionality, thereby providing users with more comprehensive information.

[0065] HUDI (Hadoop Updates and Incrementals) uses and manages large analytical datasets stored in DFS (HDFS or cloud storage), supporting update operations on the current data table. Hudi organizes tables into a directory structure under a specified directory (basepath) on HDFS. The table is divided into multiple partitions, which exist in the form of directories. Each directory contains multiple files belonging to that partition, similar to Hive tables. Each Hudi table partition is uniquely identified by a partition path.

[0066] A data lake is a repository or system that stores data in its raw format.

[0067] A watermark is a mechanism proposed by Flink for processing EventTime window calculations; it is essentially a timestamp.

[0068] Commit writes the records atomically into the dataset.

[0069] A Topic categorizes messages, with each category called a Topic. Topics allow for the management of data of the same type. Using the same Topic for the same type of data makes data processing more efficient.

[0070] A partition is a physical grouping of a topic. A topic can typically be divided into multiple partitions, each of which is an ordered queue. Each message in a partition is assigned an ordered identifier. When a producer sends messages to the server cluster, it can send them to a specified partition, or it can use a load balancing strategy to send messages to different partitions, or it can send messages randomly to different partitions.

[0071] To facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below:

[0072] Traditional data warehouses are based on hourly / daily partitions, requiring complete data for querying. However, some users don't need complete data, only the most recent data for trend analysis. Traditional data warehouses / data lakes struggle to meet these time-sensitive data requirements.

[0073] Given the requirement for data timeliness, this application provides data query solutions. These solutions offer two levels of extension:

[0074] First: At the partition level, partition attribute information has been introduced into the metadata. In other words, the metadata has been extended to improve partition capabilities. This allows for querying even when a partition is not ready. Specifically:

[0075] Real-time query: All files within a partition can be queried using a hint. As data is continuously written, the query will retrieve the latest content.

[0076] Offline query: When commit=false, it means the partition is not ready, and an empty string is returned. When commit=true, it means the partition is ready and can be queried.

[0077] Second: For the Instant level, a watermark has been introduced in Instant.

[0078] Watermarks are key time points used to collect the overall processing progress of job data streams. On the Timeline, downstream query engines can extract the overall data progress of the current job through Instant information.

[0079] The following provides an exemplary application environment for this application. Figure 1 The illustration shows an environmental application diagram according to an embodiment of this application.

[0080] Service platform 2 can provide services such as data querying, and it can be a single server, a server cluster, or a cloud computing service center. Service platform 2 has a built-in data query system 10 for providing real-time or offline queries.

[0081] Terminal 4 can be configured to send query requests, etc. The terminal can be any type of computing device, such as a smartphone, tablet, laptop, smart TV, in-vehicle terminal, etc. The terminal and service platform 2 can be connected via a network. The network can include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, and / or proxy devices. The network can include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, and combinations thereof and / or the like. The network can include wireless links, such as cellular links, satellite links, Wi-Fi links, and / or the like.

[0082] It should be noted that, Figure 1 The number of terminals and servers shown is merely illustrative and is not intended to limit the scope of patent protection in this application.

[0083] The following will provide several embodiments using the aforementioned service platform 2 as the execution entity to illustrate the data query scheme.

[0084] Example 1

[0085] Figure 2 A flowchart illustrating a data query method according to Embodiment 1 of this application is shown.

[0086] like Figure 2 As shown, the data query method may include steps S200 to S204, wherein:

[0087] Step S200 During the process of writing data to the target partition, target metadata is generated; wherein, the target metadata includes partition attribute information, which indicates that the target partition supports real-time query and current write status.

[0088] The data can be real-time data generated by mobile terminals collected through tracking points, or other real-time streaming data; it can also be MySQL Binlog logs, or incremental data describing database data changes.

[0089] The target partition is created after the previous partition has been written to and new data is available to be written. It should be noted that when upstream nodes provide data, they can send it to a specified partition, or they can use a load balancing strategy to send the data to different partitions, or randomly send the data to different partitions.

[0090] In traditional data warehouses, metadata (partition indexes) are generated and stored in the database (e.g., Hive MetaStore) only after all data in each partition has been completely written, which fails to meet the timeliness requirements of downstream nodes. In contrast to traditional data warehouses, this embodiment generates target metadata during the data writing process. This target element data incorporates partition-level attribute information, which is determined based on the data writing status.

[0091] The process of generating the partition attribute information can be as follows: Determine whether the target partition has been completely written (whether the target partition is ready) based on the watermark in the data process. If the target partition has not been completely written (is still being written), then partition attribute information with "commit='false'" is generated. If the target partition has been completely written, then partition attribute information with "commit='true'" is generated.

[0092] In optional embodiments, such as Figure 3 As shown, step S200, "generating target metadata during the data writing process to the target partition," may include: step S300, determining the current write status based on the data processing progress of the target partition; and step S302, generating the target metadata based on the current write status, the partition path of the target partition, and the partition name. The current write status can be an identifier, such as "-20." The identifier "-20" informs downstream nodes that the target partition is currently being written, supporting real-time queries for the target partition, but not offline queries. Furthermore, the target partition can be effectively indexed based on the partition path and partition name.

[0093] Step S202 It receives query requests and determines the type of the query request.

[0094] After receiving a query request, the query request is parsed to obtain a query statement. The query statement can be an SQL statement.

[0095] The types can include real-time query requests and offline query requests.

[0096] Real-time query requests can be made even when the target partition is not yet ready.

[0097] Offline query requests can only be made if the target partition is already ready.

[0098] In an optional embodiment, step S202, "determining the type of the query request," may include: if the query statement in the query request includes a target identifier, determining that the query request is a real-time query request; if the query statement in the query request does not include a target identifier, determining that the query request is an offline query request. The type of the query request can be effectively determined through the target identifier.

[0099] The target identifier can be a hint, and the hint should be set in accordance with the relevant syntax.

[0100] For example:

[0101] SQL statement 1: select from table / *includeUnCommit=true* / where field! a andlog_date='20220302'.

[0102] In the SQL statement above, "includeUnCommit=true" is a hint indicating a real-time query.

[0103] SQL statement 2: select from table where field! a and log_date='20220302'.

[0104] The SQL statement above does not contain a hint indicating a request for a real-time query, suggesting an offline query.

[0105] Step S204 In the case where the query request is a real-time query request, the corresponding query results are obtained from the target partition.

[0106] If a real-time query request is received before the target partition has been fully written, the query operation will be performed based on the existing data in the target partition. As data continues to be written, the query can retrieve the latest content. Therefore, the technical solution of this embodiment can perform real-time queries based on the existing data in the target partition, which is suitable for businesses that do not require complete data but have high real-time requirements (such as trend analysis), and has high timeliness.

[0107] In an optional embodiment, the query request may also be an offline query request, that is, the query needs to be performed on the premise that the target partition already has complete data. Such queries usually have high requirements for the integrity of the partition data.

[0108] like Figure 4As shown, the method may further include: step S400, in the case that the query request is an offline query request, determining the current write status based on the partition attribute information; wherein, the current write status includes whether the target partition has completed writing or has not completed writing; step S402, in the case that the current write status is that writing has not been completed, rejecting the response to the offline query request; step S404, in the case that the current write status is that writing has been completed, obtaining the corresponding query results from the target partition. The partition attribute information is used to determine whether an offline query can be performed, ensuring query accuracy.

[0109] When the current write status is "not completed," it means the target partition is not ready, and an empty value is returned. When the current write status is "completed," it means the target partition is ready and can be queried.

[0110] Example 2

[0111] Figure 5 A flowchart illustrating a data query method according to Embodiment 2 of this application is shown.

[0112] like Figure 5 As shown, the data query method may include steps S500 to S506, wherein:

[0113] Step S500 During the process of writing data to the HUDI table, a target watermark is provided, which represents the processing progress.

[0114] A watermark is a key time point used to collect the overall processing progress of the job data stream.

[0115] The key time points can be determined by preset rules or by changes in a predefined write state.

[0116] For example, during the writing process of the HUDI table, key time nodes correspond to the triggering of snapshot status.

[0117] In an optional embodiment, step S500 can be implemented through the following steps: Step 1, writing the data into the HUDI table using multiple data writing components; Step 2, providing multiple watermarks using the multiple streaming data writing components; wherein, a watermark is provided when the snapshot state of any of the multiple streaming data writing components is triggered; Step 3, determining the target watermark, wherein the target watermark is the minimum value of the multiple watermarks. In the above optional embodiment, based on the watermarks at the time the snapshot state is triggered, the key time points of the data during the process of writing to the HUDI table can be effectively and accurately collected and determined.

[0118] Step S502 It provides Instant information, wherein the Instant information is used to record the operation information of the data during the writing process.

[0119] The Instant information may include the following:

[0120] (1) Instant action: The type of operation performed on the HUDI table. Instant actions include, but are not limited to: COMMIT, which atomically writes a set of records to the HUDI table; CLEAN, a background activity that deletes old file versions that are no longer needed in the HUDI table; DELTA_COMMIT, which atomically writes a batch of records to a MergeOnRead storage type dataset, some or all of which may be written only to the incremental log; COMPACTION, a background activity that coordinates the differential data structures in the HUDI table.

[0121] (2) Instant time: Usually a timestamp that monotonically increases in the order of operation start time.

[0122] (3) Instant state: The current state of an Instant. Each Instant has metadata information in Avro or JSON format, which describes in detail the state of the operation and the state of the Instant at this instant. Any given instant will be in one of the following states: REQUESTED, indicating that it has been scheduled but not yet initialized; INFLIGHT, indicating that the operation is currently being executed; COMPLETED, indicating that the operation has been completed on the timeline.

[0123] Step S504Based on the target watermark and the Instant information, target Instant information is generated.

[0124] The target Instant information includes the target watermark and the Instant information, thus enhancing the richness of the information.

[0125] In an optional embodiment, step S504 may further include the following step: generating the target Instant information based on the target watermark, the Instant information, and the checkpoint information. The checkpoint information may include a checkpoint identifier (checkpointId). Adding the checkpointId to the target Instant information allows the downstream query engine to extract the checkpoint of the current job using the target Instant information. It should be noted that other information, such as commit metadata, may also be added as needed.

[0126] Step S506 The Instant information is submitted to the database and set as a node on a preset timeline for querying.

[0127] HUDI provides a timeline. This timeline includes all instant operations performed on the HUDI table at different instant times, providing an immediate view of the HUDI table and supporting data retrieval in arrival order. Multiple instants exist on this timeline. HUDI guarantees the atomicity of operations performed on the timeline and the consistency of the timeline based on instant times. The timeline corresponds to a set of files in the .hoodie metadata folder under the table's base path. Specifically, new instants are saved as a single file, while older instants are archived in the timeline archive folder to limit the number of files listed by writers and queries.

[0128] The data query method provided in this application embodiment offers target instant information carrying target watermarks on a HUDI-based timeline. This target instant information not only helps provide a real-time view of the HUDI table, effectively supporting data retrieval in arrival order, but also provides data write progress information to downstream query engines.

[0129] Based on the data writing progress, determine various subsequent operations such as real-time query or offline query.

[0130] In an optional embodiment, the method further includes: in response to a progress query request, querying the target Instant information from the time sample; extracting the target watermark from the target Instant information; and returning response information based on the target watermark. In this optional embodiment, the data writing progress can be fed back in a timely manner based on Instant information (such as target Instant information), providing key time nodes of the data stream.

[0131] For ease of understanding, the following is combined with Figure 6 and Figure 7 An example application is provided.

[0132] S1: In HUDISink (data writer), data (such as Hudi records) is written to the Hudi table through multiple streaming data writing components (WriteFunction).

[0133] S2: WriteFunction provides a watermark based on the snapshot state (SnapshotState).

[0134] The triggering conditions for SnapshotState can be set in advance.

[0135] S3: Each WriteFunction submits watermarks, events, etc. to StreamWriteOperatorCoordinator.

[0136] S4: StreamWriteOperatorCoordinator generates a target Instant carrying minwatermark, Instant, and checkpoint. minwatermark is the minimum value of all watermarks provided by each WriteFunction.

[0137] S5: Store the target Instant as HUDIMeta (metadata) in the database for downstream nodes to view.

[0138] like Figure 7 As shown, each Instant (such as Instant1, 2, 3) is set on the timeline according to time for quick querying.

[0139] Example 3

[0140] Figure 8A block diagram of a data query device according to Embodiment 3 of this application is shown. This data query device can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of this application. The program module referred to in the embodiments of this application refers to a series of computer program instruction segments capable of performing a specific function. The following description will specifically introduce the function of each program module in this embodiment. For example... Figure 8 As shown, the data query device 800 may include a generation module 810, a determination module 820, and an acquisition module 830, wherein:

[0141] The generation module 810 is used to generate target metadata during the process of writing data to the target partition; wherein, the target metadata includes partition attribute information, which indicates that the target partition supports real-time query and current write status;

[0142] The determining module 820 is used to receive a query request and determine the type of the query request; and

[0143] The acquisition module 830 is used to acquire the corresponding query results from the target partition when the query request is a real-time query request.

[0144] In an optional embodiment, the device may further include a response module (unidentified) for:

[0145] In the case where the query request is an offline query request, the current write status is determined based on the partition attribute information; wherein, the current write status includes whether the target partition has completed writing or has not completed writing;

[0146] If the current write status is that the write is not complete, then the offline query request will not be responded to.

[0147] In an optional embodiment, the response module (unidentified) is used to:

[0148] If the current write status is "write completed", the corresponding query results are obtained from the target partition.

[0149] In an optional embodiment, the determining module 820 is configured to:

[0150] If the query statement in the query request includes a target identifier, the query request is determined to be the real-time query request;

[0151] If the query statement in the query request does not include the target identifier, the query request is determined to be an offline query request.

[0152] In an optional embodiment, the generation module 810 is configured to:

[0153] The current write status is determined based on the data processing progress of the target partition;

[0154] The target metadata is generated based on the current write status, the partition path of the target partition, and the partition name.

[0155] Example 4

[0156] Figure 9 A block diagram of a data query device according to Embodiment 4 of this application is shown. This data query device can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of this application. The program module referred to in the embodiments of this application refers to a series of computer program instruction segments capable of performing a specific function. The following description will specifically introduce the function of each program module in this embodiment. For example... Figure 9 As shown, the data query device 900 may include a first providing module 910, a second providing module 920, a generating module 930, and a submitting module 940, wherein:

[0157] The first providing module 910 provides a target watermark during the process of writing data into the HUDI table, the target watermark representing the processing progress;

[0158] The second providing module 920 provides Instant information, wherein the Instant information is used to record the operation information of the data during the writing process;

[0159] Generation module 930 generates target Instant information based on the target watermark and the Instant information; and

[0160] The submission module 940 submits the Instant information to the database and sets it as a node on a preset timeline for querying.

[0161] In an optional embodiment, the first providing module 910 is further configured to:

[0162] The data is written into the HUDI table through multiple data writing components;

[0163] Multiple watermarks are provided through the plurality of streaming data writing components; wherein, a watermark is provided when the snapshot state of any one of the plurality of streaming data writing components is triggered; and

[0164] Determine the target watermark, wherein the target watermark is the minimum value of the plurality of watermarks.

[0165] In an optional embodiment, the generation module 930 is further configured to:

[0166] The target Instant information is generated based on the target watermark, the Instant information, and the checkpoint information.

[0167] In an optional embodiment, the device further includes a return module (unidentified) for:

[0168] In response to a progress query request, retrieve the target Instant information from the time sample.

[0169] Extract the target watermark from the target Instant information and return response information based on the target watermark.

[0170] Example 5

[0171] Figure 10 This diagram schematically illustrates the hardware architecture of a computer device suitable for implementing a data query method according to Embodiment 5 of this application. The computer device 10000 can be a service platform 2 or a part thereof. The computer device 10000 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc. Figure 10 As shown, the computer device 10000 includes, but is not limited to, at least the following: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked to each other via a system bus. Wherein:

[0172] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for data query methods. Furthermore, the memory 10010 can also be used to temporarily store various types of data that have already been output or will be output.

[0173] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0174] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication connection between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0175] It should be pointed out that, Figure 10 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0176] In this embodiment, the data query method stored in memory 10010 can be further divided into one or more program modules and executed by one or more processors (processor 10020 in this embodiment) to complete this application.

[0177] Example 6

[0178] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data query method in this embodiment.

[0179] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium may include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the data query method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0180] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0181] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A data query method, characterized in that, The method includes: During the process of writing data to the target partition, the data processing progress information is determined based on the watermark of the target partition; Based on the data processing progress information, target metadata is generated; wherein, the target metadata includes partition attribute information, which indicates that the target partition supports real-time query and current write status; Receive a query request and determine the type of the query request; and In the case where the query request is a real-time query request, the corresponding query results are obtained from the target partition; The method further includes: In the case where the query request is an offline query request, the current write status is determined based on the partition attribute information; wherein, the current write status includes whether the target partition has completed writing or has not completed writing; If the current write status is that the write is not complete, then the offline query request will not be responded to. If the current write status is "write completed", the corresponding query results are obtained from the target partition.

2. The method according to claim 1, characterized in that, Determining the type of the query request includes: If the query statement in the query request includes a target identifier, the query request is determined to be the real-time query request; If the query statement in the query request does not include the target identifier, the query request is determined to be an offline query request.

3. The method according to any one of claims 1 to 2, characterized in that, During the process of writing data to the target partition, the generation of target metadata includes: The current write status is determined based on the data processing progress of the target partition; The target metadata is generated based on the current write status, the partition path of the target partition, and the partition name.

4. A data query device, characterized in that, The device includes: The determination module is used to determine the data processing progress information based on the watermark of the target partition during the process of writing data to the target partition; The generation module is used to generate target metadata based on the data processing progress information; wherein, the target metadata includes partition attribute information, which indicates that the target partition supports real-time query and current write status; A determination module is used to receive a query request and determine the type of the query request; and The acquisition module is used to acquire the corresponding query results from the target partition when the query request is a real-time query request; The device further includes a response module for: In the case where the query request is an offline query request, the current write status is determined based on the partition attribute information; wherein, the current write status includes whether the target partition has completed writing or has not completed writing; If the current write status is that the write is not complete, then the offline query request will not be responded to. If the current write status is "write completed", the corresponding query results are obtained from the target partition.

5. A data query method, characterized in that, The method includes: During the process of writing data into the HUDI table, a target watermark is provided, which represents the data processing progress of the corresponding partition in the HUDI table. Provide Instant information, wherein the Instant information is used to record the operation information of the data during the writing process; Based on the target watermark and the Instant information, target Instant information is generated, wherein the target Instant information includes the target watermark and the Instant information; and The target Instant information is submitted to the database and set as a node on a preset timeline for querying.

6. The method according to claim 5, characterized in that, The process of writing data to the HUDI table includes providing a target watermark, including: The data is written into the HUDI table using multiple streaming data writing components; Multiple watermarks are provided through the plurality of streaming data writing components; wherein, a watermark is provided when the snapshot state of any one of the plurality of streaming data writing components is triggered; and Determine the target watermark, wherein the target watermark is the minimum value of the plurality of watermarks.

7. The method according to claim 6, characterized in that, The step of generating target Instant information based on the target watermark and the Instant information further includes: The target Instant information is generated based on the target watermark, the Instant information, and the checkpoint information.

8. The method according to any one of claims 5 to 7, characterized in that, Also includes: In response to a progress query request, retrieve the target Instant information from the time sample. Extract the target watermark from the target Instant information and return response information based on the target watermark.

9. A data query device, characterized in that, The device includes: The first module provides a target watermark during the process of writing data into the HUDI table. The target watermark represents the data processing progress of the corresponding partition in the HUDI table. The second providing module provides Instant information, wherein the Instant information is used to record the operation information of the data during the writing process; The generation module generates target instant information based on the target watermark and the instant information, wherein the target instant information includes the target watermark and the instant information; and The submission module submits the target Instant information to the database and sets it as a node on a preset timeline for querying.

10. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data query method according to any one of claims 1 to 3 or 5 to 8.

11. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program may be executed by at least one processor to cause the at least one processor to perform the steps of the data query method according to any one of claims 1 to 3 or 5 to 8.

Images