Data query method, device, system, apparatus, and storage medium

Abstract

The embodiment of the present specification provides a data query method, device, system, equipment and storage medium. For the scene that the data to be queried by the user from the OSS is multi-segment data with discontinuous storage addresses in the to-be-queried file, the data pushing function can be used to change the multiple query requests of the client into "one query request + multiple pushing requests". When sending the query request, the client can carry the identification information of the multi-segment data at the same time, one segment of data in the multi-segment data is taken as the to-be-requested data of the current query request, and the remaining segments of data are taken as the pushing data. After receiving the query request, the object storage server can obtain the to-be-requested data as the response result of the query request and return the to-be-requested data to the client, and then obtain the remaining segments of data in sequence as the pushing data and push the pushing data to the client. Thus, all to-be-queried data can be obtained through one query request, the QPS is reduced, and the access efficiency is improved.

Description

Technical Field

[0001] This specification relates to the field of data storage technology, and in particular to a data query method, apparatus, system, device, and storage medium. Background Technology

[0002] Files are typically stored in a specific format. When querying files on OSS (Object Storage Service), there are scenarios where the data matching the user's query criteria consists of multiple segments with non-contiguous storage addresses. In such scenarios, these multiple data segments need to be read from the OSS device (object storage device) multiple times, resulting in low efficiency in accessing OSS files. For example, taking Parquet format files as an example, since consecutive columns in such files often have consecutive storage addresses, and non-consecutive columns also have non-consecutive storage addresses, the general process for accessing such files on OSS is as follows: the client determines which columns to query based on the user's query criteria, then forms a query request for consecutive columns and sends it to the object storage service server so that the object storage service server can retrieve the data to be queried. If the data to be queried consists of multiple non-contiguous columns, multiple file accesses are required. For example, if a file has 100 columns (1-100), and the user needs to query odd-numbered columns such as 1, 3, 5, etc., 50 query requests need to be sent, requiring 50 file accesses.

[0003] It is evident that accessing files from OSS is inefficient in scenarios where the data to be queried consists of multiple segments of data with non-contiguous storage addresses. Therefore, it is necessary to provide a solution to improve access efficiency. Summary of the Invention

[0004] To overcome the problems existing in related technologies, embodiments of this specification provide a data query method, apparatus, system, device, and storage medium.

[0005] According to a first aspect of the embodiments of this specification, a data query method is provided, applicable to a client, wherein the client and an object storage service client can communicate based on a communication protocol with data push functionality, the method comprising:

[0006] Based on the query conditions, the data to be queried is determined from the file to be queried, and the data to be queried is divided into data segments, wherein the data in the same data segment is stored at consecutive addresses.

[0007] A query request is generated, wherein, when the data segments obtained are at least two data segments, the query request carries the identification information of the at least two data segments and the object identifier of the object to which the file to be queried belongs, wherein the at least two data segments include the data segment to be requested;

[0008] The query request is sent to the object storage service client, so that the object storage service client obtains the data in the at least two data segments based on the identification information and the object identifier, and returns the data of the requested data segment as the response result of the query request to the client; the data in each of the other data segments in the at least two data segments, excluding the requested data segment, is pushed to the client as push data in sequence.

[0009] In some embodiments, the communication protocol is the HTTP / 2.0 protocol.

[0010] In some embodiments, the requested data segment is determined based on the identification information of the at least two data segments in the order of the query request; or

[0011] The query request also carries indication information for indicating the requested data segment, which is determined based on the indication information.

[0012] In some embodiments, the file to be queried is a file stored in a columnar format.

[0013] In some embodiments, the file to be queried is a Parquet format file, and each of the at least two data segments includes one or more column blocks from the file to be queried.

[0014] According to a second aspect of the embodiments of this specification, a data query method is provided, applicable to an object storage service client, wherein the object storage service client and the client can communicate based on a communication protocol with data push functionality, the method comprising:

[0015] The system receives a query request sent by a client. The query request carries identification information of at least two data segments and an object identifier of the object to which the file to be queried belongs. The at least two data segments are obtained by dividing the data to be queried in the file to be queried. The data in the same data segment are stored in consecutive addresses. The at least two data segments include the data segment to be requested.

[0016] For the requested data segment, the data in the requested data segment is obtained based on the identification information and the object identifier, and returned to the client as the response result of the query request;

[0017] For the data segments other than the requested data segment among the at least two data segments, the data of each other data segment is obtained sequentially based on the identification information and pushed to the client as push data.

[0018] In some embodiments, the communication protocol is the HTTP / 2.0 protocol.

[0019] In some embodiments, the requested data segment is determined based on the identification information of the at least two data segments in the order of the query request; or

[0020] The query request also carries indication information for indicating the requested data segment, which is determined based on the indication information.

[0021] In some embodiments, obtaining data from the requested data segment based on the identification information and the object identifier includes:

[0022] Based on the object identifier, obtain the metadata of the object to which the file to be queried belongs;

[0023] Based on the metadata and the identification information, the storage location of the data in the requested data segment is determined, and the data is obtained based on the determined storage location.

[0024] In some embodiments, after obtaining the data in the requested data segment based on the identification information and the object identifier, the method further includes:

[0025] The metadata is cached locally.

[0026] In some embodiments, for data segments other than the requested data segment among the at least two data segments, data of each other data segment is sequentially obtained based on the identification information, including:

[0027] For each other data segment, the storage location of the data in each other data segment is determined based on the cached metadata and the identification information, and the data is obtained based on the determined storage location.

[0028] In some embodiments, the file to be queried is a file stored in a columnar format.

[0029] In some embodiments, the file to be queried is a Parquet format file, and each of the at least two data segments includes one or more column blocks from the file to be queried.

[0030] According to a fourth aspect of the embodiments of this specification, a data query device is provided, suitable for a client, wherein the client and an object storage service client can communicate based on a communication protocol with data push functionality, and the device includes:

[0031] The partitioning module is used to determine the data to be queried from the file to be queried based on the query conditions, and to divide the data to be queried into data segments, wherein the data in the same data segment are stored at consecutive addresses.

[0032] A request generation module is used to generate a query request, wherein, when the data segments obtained are at least two data segments, the query request carries the identification information of the at least two data segments and the object identifier of the object to which the file to be queried belongs, wherein the at least two data segments include the data segment to be requested;

[0033] The sending module is configured to send the query request to the object storage service client, so that the object storage service client obtains the data in the at least two data segments based on the identification information and the object identifier, returns the data of the requested data segment as the response result of the query request to the client, and pushes the data in each of the at least two data segments other than the requested data segment to the client as push data in sequence.

[0034] According to a fifth aspect of the embodiments of this specification, a data query apparatus is provided, suitable for an object storage service client, wherein the object storage service client and the client can communicate based on a communication protocol with data push functionality, the apparatus comprising:

[0035] The receiving module is used to receive a query request sent by the client. The query request carries identification information of at least two data segments and an object identifier of the object to which the file to be queried belongs. The at least two data segments are obtained by dividing the data to be queried in the file to be queried. The storage addresses of the data in the same data segment are continuous, while the storage addresses of the data in different data segments are not continuous. The at least two data segments include the data segment to be requested.

[0036] The processing module is configured to, for the requested data segment, obtain the data in the requested data segment based on the identification information and the object identifier, and return it to the client as the response result of the query request; for the other data segments among the at least two data segments excluding the requested data segment, sequentially obtain the data of each other data segment based on the identification information, and push it to the client as push data.

[0037] According to a sixth aspect of the embodiments of this specification, a data query system is provided, including a client and an object storage service terminal, wherein the client and the object storage service terminal can communicate based on a communication protocol with data push function;

[0038] The client is used to implement the methods mentioned in the embodiments of the first aspect above;

[0039] The object storage service client is used to implement the methods mentioned in the embodiments of the second aspect above.

[0040] According to a seventh aspect of the embodiments of this specification, an apparatus is provided, the apparatus including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program, when executed, implements the methods mentioned in the first aspect and / or the second aspect above.

[0041] According to an eighth aspect of the embodiments of this specification, a computer storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the methods mentioned in the first aspect and / or the second aspect above.

[0042] The beneficial effects of the embodiments in this specification are as follows: Considering that some communication protocols between the client and the server have data push functionality, for scenarios where the data a user wants to query from OSS consists of multiple segments of data with non-contiguous storage addresses in the file to be queried, multiple query requests from the client can be changed to "one query request + multiple push requests". When the client sends a query request, it can simultaneously carry the identification information of these multiple data segments. One of these data segments serves as the data to be requested in the current query request, and the remaining segments serve as push data. After receiving the query request, the object storage server can obtain the data to be requested and return it to the client as the response result of the query request. Then, it can sequentially obtain the remaining data segments and push them to the client as push data. Thus, all the data to be queried can be obtained with a single query request, reducing QPS and improving access efficiency.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the embodiments of this specification. Attached Figure Description

[0044] The accompanying drawings, which are incorporated herein by reference and form part of the embodiments thereof, illustrate embodiments consistent with those described herein and, together with the description, serve to explain the principles of those embodiments.

[0045] Figure 1 A schematic diagram illustrating a Parquet format file as an exemplary embodiment of this specification;

[0046] Figure 2 This is a schematic diagram illustrating a query for data from OSS, as shown in an exemplary embodiment of this specification.

[0047] Figure 3 This is a schematic diagram illustrating an application scenario as an exemplary embodiment of this specification;

[0048] Figure 4 A timing diagram illustrating a data query method as an exemplary embodiment of this specification;

[0049] Figure 5 This is a schematic diagram illustrating a data query method as an exemplary embodiment of this specification;

[0050] Figure 6 A logic block diagram of a data query apparatus shown in an exemplary embodiment of this specification;

[0051] Figure 7 A logic block diagram of a data query apparatus shown in an exemplary embodiment of this specification;

[0052] Figure 8 This is a logic block diagram of an electronic device illustrated in an exemplary embodiment of this specification. Detailed Implementation

[0053] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those described in this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the embodiments described in this specification as detailed in the appended claims.

[0054] The terminology used in the embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of this specification. The singular forms “a,” “described,” and “the” as used in the embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0055] It should be understood that although the terms first, second, third, etc., may be used to describe various information in the embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0056] In big data storage technologies, files stored in storage typically have a specific format. For example, Delta, Hudi, and Iceberg, three common data lake storage technologies, all store files in Parquet (a columnar storage format), while Hive, a data warehouse tool based on Hadoop, stores files in ORC (another columnar storage format). For Parquet and ORC files, the file is usually divided into multiple data blocks, and these blocks are stored according to a specific format. For instance, with tabular data, one or more columns (or rows) can be considered as a single data block and stored in a specific format. When accessing data, an index can be used to determine which storage area to access. Because files are stored in a specific format, when performing a Range query on a file, the data being retrieved may consist of multiple segments of data with non-contiguous storage addresses.

[0057] For example, taking a Parquet file as an example, such as Figure 1 The diagram illustrates a Parquet file format. Typically, a Parquet file groups multiple rows of tabular data into a rowgroup, and each rowgroup contains multiple column blocks, each containing data from the same column. When storing a Parquet file, these column blocks are usually stored sequentially, so data from the same column is often stored at contiguous memory addresses. However, when a user queries the file, the data matching the query criteria may be from non-contiguous columns; that is, the data to be queried consists of multiple segments of data with non-contiguous memory addresses.

[0058] In OSS, data is stored as objects in OSS buckets. An object is the basic unit of data storage in OSS, also known as an OSS file. Unlike traditional file systems, objects do not have a hierarchical file directory structure. An object consists of metadata, user data, and an object key; each object is identified by a unique key within the bucket. Object metadata is a set of key-value pairs representing some of the object's attributes, such as its location within OSS. Figure 2As shown, metadata and user data are typically stored separately. Metadata is stored on a metadata server, while user data is stored on OSS devices or device clusters. When a user wants to access data on OSS, they can send a query request to the object storage service provider through a client, carrying the key of the file (i.e., the object) to be queried. The object storage service provider can first retrieve the metadata of the file to be queried from the metadata server based on the key, determine the storage location of the file to be queried on the OSS device or device cluster based on the metadata, then retrieve the file to be queried from the corresponding storage location and return it to the client.

[0059] Because file formats such as Parquet and ORC rely on the POSIX / HDFS file interface for reading and writing, they are not well-suited for object storage protocols. For example, when a user wants to access data that consists of multiple segments with non-contiguous storage addresses, the access efficiency is often low. Taking Parquet files as an example, the general process is as follows: the client determines the columns to be queried based on the user's input query conditions, then forms a query request for consecutive columns and sends it to the object storage service to retrieve the data. If the columns to be queried are non-contiguous, multiple file accesses are required. For instance, if a file has 100 columns (1-100), and the user only queries the odd-numbered columns (1, 3, 5, etc.), 50 query requests and 50 file accesses are needed. Therefore, for scenarios where the data to be queried consists of multiple segments with non-contiguous storage addresses, accessing files from OSS is inefficient.

[0060] To improve access efficiency in the above scenarios, two solutions exist. One is to use one query request for each data segment, shortening latency by having the client send multiple concurrent query requests. For example, taking a Parquet file, assuming the user-input query conditions determine the data to be queried to be columns 1, 2, 4, and 6, based on the continuity of storage addresses, these four columns can be divided into three segments: segment 1 (columns 1-2), segment 2 (column 4), and segment 3 (column 6). Therefore, three query requests can be generated and sent concurrently to the object storage service. Since different query requests are typically sent to different object storage services, multiple object storage service providers can process these three query requests in parallel. This approach can improve access efficiency to some extent, but it generates extremely high Qps (Query Per Second), resulting in significant resource overhead for both the object storage service and the client. Furthermore, since different query requests are often sent to different object storage service providers, each object storage service provider has to obtain the metadata of the file to be queried from the metadata server, which will increase the number of times the metadata is accessed and cause excessive processing pressure on the metadata server.

[0061] Another approach is to use an aggregated interface. Referring to the example above, suppose the client needs to send three query requests. The client can encapsulate these three requests in a single XML file and submit it to the object storage service provider with a single request. The object storage service provider needs to retrieve all the data from the three queries, encapsulate it in the body, and return it to the client. The client needs to receive all the data and then parse each piece of data individually. Since data parsing takes time, this will increase latency to some extent. Furthermore, the client can only retrieve all the data at a time; individual data segments cannot be retrieved separately.

[0062] Based on this, the embodiments of this specification provide a data query method. Considering that some communication protocols between the client and the server have data push functionality, for example, when the client sends a query request to the server, the server can return the data currently requested in the query request, and at the same time, can push other data that the client may request to the client. Therefore, for scenarios where the data that the user wants to query from OSS consists of multiple segments of data with non-contiguous storage addresses in the file to be queried, the client's multiple query requests can be changed to "one query request + multiple push requests". When the client sends a query request, it can simultaneously carry the identification information of these multiple data segments. One of these data segments is used as the data to be requested in the current query request, and the remaining data segments are used as push data. After receiving the query request, the object storage server can obtain the data to be requested and return it to the client as the response result of the query request. Then, it can sequentially obtain the remaining data segments and push them to the client as push data. Thus, all the data to be queried can be obtained through a single query request, reducing QPS and improving access efficiency.

[0063] like Figure 3 The diagram illustrates an application scenario of the data query method provided in this specification. This data query method can be collaboratively completed by a client and an object storage service in a data query system. The data query system may include one or more clients and one or more object storage service clients. Each client can be deployed on an electronic device such as a mobile phone, tablet, or laptop, and can be a browser or an app. Each object storage service client can be deployed on different servers, where the server and the OSS device used to store the data can be the same device or different devices. When a user wants to query data in file A stored on OSS, they can send a query request to the object storage service client through the client. The object storage service client can determine the metadata corresponding to file A from the metadata server, determine the storage location of file A on the OSS device based on the metadata, and then retrieve the data to be queried based on the storage location.

[0064] For scenarios where the data accessed from OSS consists of multiple segments with non-contiguous storage addresses, this specification's embodiments deploy a communication protocol with data push functionality on both the client and the object storage service provider, enabling them to communicate via this protocol. The data push function means that when a client sends a query request to the object storage service provider, the provider not only returns the currently requested data as a response, but also continues to push other data that the client needs to retrieve.

[0065] In some embodiments, the communication protocol may be the HTTP 2.0 protocol, or other communication protocols with similar data push functions. This specification does not limit the embodiments.

[0066] like Figure 4 The diagram shown illustrates the sequence of interactions between the client and the object storage service during a data query process, which may include the following steps:

[0067] S401. The client determines the data to be queried from the file to be queried based on the query conditions, and divides the data to be queried into data segments, wherein the storage addresses of the data in the same data segment are consecutive.

[0068] In step S401, when a user wants to query data that meets certain conditions from a file, they can input the query conditions through the client. The query conditions can be the conditions that the data to be queried must meet, such as "names of students whose Chinese language scores are below 60". Of course, in some scenarios, the client can also receive query conditions from other devices. The client can parse the query conditions to determine which data meets the criteria. Since files are usually stored in a certain format, for example, in row-based storage, data is stored row by row, and the storage addresses of adjacent rows are usually continuous. Similarly, in column-based storage, data is stored column by column, and the storage addresses of adjacent columns are usually continuous. Therefore, the client can determine which rows or columns of data meet the query conditions based on the file's storage format, then determine whether the storage addresses of these rows or columns are continuous, and then divide the data into one or more data segments based on the continuity of the data storage addresses. Data within the same data segment has continuous storage addresses, while data in different data segments has non-contiguous storage addresses.

[0069] Of course, in some scenarios, files are first divided into data blocks and then stored according to the data blocks. When determining the data to be queried that meets the conditions, it can be done in units of data blocks. For example, it can be determined that there are multiple data blocks that meet the query conditions, and then it can be determined whether the storage addresses of the data blocks are continuous. Based on the continuity of the storage addresses of the data blocks, these multiple data blocks are divided into multiple data segments, and each data segment includes one or more data blocks.

[0070] For example, suppose the file to be queried is a data table stored in a columnar format. Assume the client determines the data to be queried based on the user's input query conditions as columns 1, 2, 4, 5, and 6. The data matching the query conditions can be the entire column of data in columns 1, 2, 4, 5, and 6, or it can be a portion of the data in these columns. Considering that consecutive columns are usually stored in consecutive memory locations, the above data can be divided into three segments: columns 1-2 as one segment, columns 4-5 as another segment, and column 6 as a third segment.

[0071] S402. The client generates a query request, wherein, when the divided data segment is at least two data segments, the query request carries the identification information of the at least two data segments and the object identifier of the object to which the file to be queried belongs, wherein the at least two data segments include the data segment to be requested;

[0072] In step S402, if the divided data segments are at least two, that is, after the client determines at least two data segments to query from OSS, it can generate a query request based on these at least two data segments. Data in OSS is stored in objects as the basic unit, and each object has a unique object identifier (key) in OSS. Therefore, the query request can include the object identifier of the object to which the file to be queried belongs, so that the object storage service client can find the object to which the file to be queried belongs from OSS. The file to be queried can be an object on OSS or a part of an object; this specification and embodiment do not impose limitations.

[0073] In addition, to locate the data to be queried from the file to be queried, the query request may also carry the identification information of at least two data segments. The identification information of the data segments can be used to determine the storage location of the data in each data segment on the OSS storage device. The identification information of the data segments can be represented by various forms of data, as long as it can locate the data in the data segment. For example, taking a file in columnar storage format as an example, the data segment identifier can be the column numbers included in the data segment (e.g., columns 1, 2, and 3), or it can be the number of the first or last column plus the column number (e.g., column 1 + column number 3, or column 3 + column number 3). Of course, other information can also be used as the identification information, and this specification does not limit the examples.

[0074] Since the storage addresses of these at least two data segments are not contiguous, reading data from these at least two data segments from the OSS storage device requires multiple reads, with only one segment being read at a time. Therefore, when sending a query request, one of these at least two data segments can be designated as the requested data segment, so that the object storage service can return it to the client as a response to the query request. The remaining data segments can be designated as push data segments, so that the object storage service can push them to the client as push data.

[0075] The client can randomly select one of at least two data segments as the data segment to be requested, and the rest as the data segment to be pushed. Alternatively, the user can customize the return order of these data segments, and select the data segment that needs to be returned first as the data segment to be requested.

[0076] S403. The client sends the query request to the object storage service client.

[0077] In step S403, after generating the query request, the client can send the query request to the object storage service provider. For example, assuming that the client and the object storage service provider communicate via the HTTP protocol, the client can send an HTTP request to the object storage service provider.

[0078] Given the large number of query requests sent by clients, there are typically multiple object storage service providers deployed on different servers. Therefore, client query requests can be randomly routed to one of the object storage service providers, or a load balancing strategy can be used to distribute query requests to object storage service providers with relatively idle processing resources.

[0079] S404. The object storage service client receives query requests sent by the client.

[0080] In step S404, the object storage service client can receive query requests sent by the client.

[0081] S405. The object storage service client, for the requested data segment among the at least two data segments, obtains the data in the requested data segment based on the identification information and the object identifier, and returns it to the client as the response result of the query request;

[0082] In step S405, for the requested data segment among the at least two data segments, the object storage service client can first determine the storage location of the object to which the queried file belongs in the OSS device based on the object identifier carried in the query request. For example, the OSS devices used to store data in the OSS service are usually a cluster composed of multiple devices. Therefore, the metadata of the queried file can be determined based on the object identifier, and based on the metadata, it can be determined which OSS device in the cluster the queried file is located in, as well as its specific storage location in that OSS device. Then, the data in the requested data segment can be retrieved from that OSS device based on the identifier information of the requested data segment, and returned to the client as the response result of the query request.

[0083] S406. The object storage service client sequentially obtains the data of each of the other data segments (excluding the data segment to be requested) based on the identification information, and pushes it to the client as push data.

[0084] In step S406, after returning the data in the requested data segment to the client, for the other data segments in the at least two data segments, the object storage service client can sequentially obtain the data of the other data segments based on the identification information of each other data segment, and push them to the client as push data in sequence.

[0085] For example, suppose the client determines three data segments based on the user's query conditions: data segment A, data segment B, and data segment C. Data segment A is the data segment to be requested. After receiving the query request, the object storage service client can first retrieve the data in data segment A from the OSS device based on the object identifier of the file to be queried and the identifier information of data segment A, and then return it to the client as the response result of the query request. Next, the object storage service client can retrieve the data in data segment B from the OSS device based on the identifier information of data segment B, and push it to the client as push data. Finally, the object storage service client can retrieve the data in data segment C from the OSS device based on the identifier information of data segment C, and push it to the client as push data.

[0086] In some embodiments, to facilitate the object storage service provider's identification of which of the at least two data segments is the requested data segment, the client and the object storage service provider can pre-agree on a method for determining the requested data segment. For example, the requested data segment can be determined based on the identification information of the at least two data segments in the order of the query request. For instance, when the client includes the identification information of the at least two data segments in the query request, it can place the identification information of the requested data segment at the beginning or end. After receiving the query request, the object storage service provider can determine the identification information of the requested data segment based on the order of the identification information of the data segments in the query request.

[0087] In some embodiments, the query request may also include additional indication information indicating the data segment to be requested, so that the object storage service client can determine the identification information of the data segment to be requested from the identification information of the at least two data segments based on the indication information.

[0088] In some embodiments, when retrieving data from a requested data segment, the object storage service client can first obtain the metadata of the object to which the queried file belongs based on the object identifier; then, based on the metadata and the identifier information of the requested data segment, it can determine the storage location of the data in the requested data segment within the queried file, and retrieve the data from the requested data segment based on the determined storage location. The metadata on OSS can be used to describe information such as the storage location of objects stored on OSS. Therefore, based on the metadata, the storage location of the queried file on OSS can be determined, such as which OSS device it is on and its specific storage location on that OSS device. Furthermore, for queried files in a certain format, the file itself also contains index information to determine the storage location of each data block within the file. Taking a Parquet format file as an example, each rowgroup in the file also includes corresponding index information to determine the storage location of each column block within the rowgroup. Therefore, by combining the metadata and the identifier information of the data segment, the storage location of each piece of data within the data segment can be located.

[0089] In some embodiments, considering that when retrieving other data segments besides the requested data segment, it is also necessary to utilize the metadata of the object to which the file to be queried belongs, in order to improve the retrieval speed of other data segments and reduce the number of accesses to the metadata server, the object storage service client can cache the metadata to the local machine after obtaining the metadata of the object to which the file to be queried belongs, so that the cached metadata can be reused when retrieving data in other data segments later.

[0090] In some embodiments, after returning the requested data segment to the client, for the other data segments among the at least two data segments excluding the requested data segment, the object storage service client can sequentially determine the storage location of the data in each other data segment on the OSS device based on the identification information of each data segment and the locally cached metadata, obtain the data in each other data segment based on the determined storage location, and then sequentially return the data of each other data segment.

[0091] Considering that for files stored in a columnar format, users typically query certain columns when searching for data, it's common to encounter scenarios where the queried data consists of multiple segments of data with non-contiguous storage addresses when querying files stored in this format on OSS. Therefore, in some embodiments, the file to be queried can be a file stored in a columnar format, such as a Parquet file, an ORC file, or a file in another columnar storage format.

[0092] In some embodiments, if the file to be queried is a Parquet format file, each of the at least two data segments can be one or more column blocks in the file to be queried. For example, the user can determine which column blocks contain data that meets the query criteria, or in which column blocks the data meets the query criteria, and then determine whether the storage addresses of these column blocks are contiguous, dividing the column blocks with contiguous storage addresses into a data segment.

[0093] For example, such as Figure 5 As shown, assuming the client and object storage service can communicate via HTTP / 2.0, and the file to be queried is in Parquet format, the client determines the data to be queried based on the user-input query conditions to be columns 1, 2, 4, 5, and 6. This data can be divided into three segments: segment A (columns 1-2), segment B (columns 4-5), and segment C (column 6), where segment A is the requested data segment. The client can send an HTTP query request to the object storage service. Upon receiving the query request, the object storage service first obtains the metadata of the file to be queried based on its object identifier. Then, based on the metadata and the identifier information of segment A, it retrieves the data in segment A from the OSS device (where, if the data matching the query conditions is an entire column, all columns in segment A can be retrieved; if the data matching the query conditions is a partial column, only the partial data matching the query conditions in each column of segment A can be retrieved). This data is then returned to the client as the response to the query request. After obtaining the metadata, the object storage service can cache it locally. Then, the object storage service client can retrieve the data in data segment B from the OSS device based on the cached metadata and the identification information of data segment B, and push it to the client as push data. Next, the object storage service client can retrieve the data in data segment C from the OSS device based on the cached metadata and the identification information of data segment C, and push it to the client as push data.

[0094] For scenarios where a client queries multiple segments of data with non-contiguous storage addresses from an object storage service, this specification proposes utilizing the data push function of certain communication protocols. This eliminates the need for the client to send multiple query requests to the object storage service to retrieve these data segments separately. Instead, the client only needs to send a single query request containing identifiers for the multiple data segments. The object storage service can then return one segment of the data as the response to the query request to the client, while simultaneously pushing the remaining segments sequentially to the client. This allows for retrieving multiple data segments with a single query request, reducing QPS and improving access efficiency.

[0095] Meanwhile, since only one query request needs to be sent, the task of retrieving these multiple data segments will only be executed by an object storage service client. After the object storage service client obtains the metadata, it can cache it locally. Thus, when retrieving these multiple data segments, the cached metadata can be reused. This improves the data query efficiency to a certain extent. At the same time, it does not need to retrieve metadata from the metadata server multiple times, reducing the processing pressure on the metadata server.

[0096] Furthermore, compared to existing aggregation interfaces, where the object storage service requires retrieving multiple data segments before returning all data to the client, the method provided in this specification allows for the independent and sequential return of each data segment, and the client can customize the return order of these segments, offering greater flexibility.

[0097] In addition, the embodiments of this specification also include a data query system, such as Figure 3 As shown, the data query system includes a client and an object storage service, wherein the client and the object storage service can communicate based on a communication protocol with data push functionality;

[0098] The client is used to determine the data to be queried from the file to be queried based on the query conditions input by the user, and divide the data to be queried into at least two data segments, wherein the storage addresses of the data in the same data segment are continuous, and the storage addresses of the data in different data segments are not continuous; and send the query request to the object storage service client, wherein the query request carries the identification information of the at least two data segments, as well as the object identifier of the object to which the file to be queried belongs;

[0099] The object storage service client is used to obtain data from the requested data segment based on the identification information and the object identifier for the requested data segment among the at least two data segments, and return it to the client as the response result of the query request; for other data segments among the at least two data segments excluding the requested data segment, it sequentially obtains data from each other data segment based on the identification information, and pushes it to the client as push data.

[0100] The specific process of executing the data query method on the client and object storage service side can be referred to the description in the above method embodiments, and will not be repeated here.

[0101] Corresponding to the data query method embodiments provided in this specification, this specification also provides a data query device for a client. The client and the object storage service client can communicate based on a communication protocol with data push functionality, such as... Figure 6 As shown, the device 60 includes:

[0102] The partitioning module 62 is used to determine the data to be queried from the file to be queried based on the query conditions, and to divide the data to be queried into data segments, wherein the storage addresses of the data in the same data segment are consecutive.

[0103] The request generation module 64 is used to generate a query request, wherein, when the data segments obtained are at least two data segments, the query request carries the identification information of the at least two data segments and the object identifier of the object to which the file to be queried belongs, wherein the at least two data segments include the data segment to be requested;

[0104] The sending module 66 is used to send the query request to the object storage service terminal, so that the object storage service terminal obtains the data in the at least two data segments based on the identification information and the object identification, returns the data of the requested data segment as the response result of the query request to the client, and pushes the data in each of the at least two data segments other than the requested data segment as push data to the client in sequence.

[0105] The specific implementation process of the functions and roles of each unit in the above-mentioned device can be found in the implementation process of the corresponding steps in the above-mentioned memory allocation method, and will not be repeated here.

[0106] Corresponding to the data query method embodiments provided in this specification, this specification also provides a data query device. This data query device is suitable for object storage service clients, whereby the object storage service client and client can communicate based on a communication protocol with data push functionality, such as... Figure 7 As shown, the device 70 includes:

[0107] The receiving module 72 is used to receive a query request sent by the client. The query request carries identification information of at least two data segments and an object identifier of the object to which the file to be queried belongs. The at least two data segments are obtained by dividing the data to be queried in the file to be queried. The storage addresses of the data in the same data segment are continuous, while the storage addresses of the data in different data segments are not continuous. The at least two data segments include the data segment to be requested.

[0108] The processing module 74 is configured to, for the requested data segment, obtain the data in the requested data segment based on the identification information and the object identifier, and return it to the client as the response result of the query request; for the other data segments among the at least two data segments excluding the requested data segment, sequentially obtain the data of each other data segment based on the identification information, and push it to the client as push data.

[0109] The specific implementation process of the functions and roles of each unit in the above-mentioned device can be found in the implementation process of the corresponding steps in the above-mentioned memory allocation method, and will not be repeated here.

[0110] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiments in this specification, depending on actual needs. Those skilled in the art can understand and implement this without creative effort.

[0111] From a hardware perspective, such as Figure 8 The diagram shown is a hardware structure diagram of the device containing the compatibility testing apparatus according to an embodiment of this specification. Except for... Figure 8 In addition to the processor 82 and memory 84 shown, the device may also include other hardware, such as a forwarding chip responsible for processing messages; from a hardware structure perspective, the device may also be a distributed device, possibly including multiple interface cards to extend message processing at the hardware level. The memory 84 stores computer instructions, and when the processor 82 executes the computer instructions, it implements the compatibility testing method mentioned in any of the above embodiments.

[0112] Since the parts of the embodiments in this specification that contribute to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, the computer software product is stored in a storage medium and includes several instructions to cause a terminal device to execute all or part of the steps of the methods in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0114] The above description is merely a preferred embodiment of the embodiments of this specification and is not intended to limit the embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this specification should be included within the scope of protection of the embodiments of this specification.

Claims

1. A data query method, applicable to a client, wherein the client and an object storage service communicate based on a communication protocol with data push functionality, the method being used to query data from an object storage device, the method comprising: Based on the query conditions, the data to be queried is determined from the file to be queried, and the data to be queried is divided into data segments, wherein the data in the same data segment is stored at consecutive addresses. A query request is generated, wherein, when the data segments obtained are at least two data segments, the query request carries the identification information of the at least two data segments and the object identifier of the object to which the file to be queried belongs, wherein one of the at least two data segments is used as the data segment to be requested in the query request, and the other data segments in the at least two data segments other than the data segment to be requested are used as data segments to be pushed. The query request is sent to the object storage service client, so that the object storage service client obtains data from the at least two data segments from the object storage device based on the identification information and the object identifier, and returns the data of the requested data segment as the response result of the query request to the client; and pushes the data of each data segment to be pushed to the client as push data in sequence.

2. The method according to claim 1, wherein the communication protocol is the HTTP 2.0 protocol.

3. The method according to claim 1, wherein the requested data segment is determined based on the identification information of the at least two data segments in the order of the query request; or The query request also carries indication information for indicating the requested data segment, which is determined based on the indication information.

4. The method according to claim 1, wherein the file to be queried is a file stored in a columnar format.

5. The method according to claim 4, wherein the file to be queried is a Parquet format file, and each of the at least two data segments includes one or more column blocks in the file to be queried.

6. A data query method, applicable to an object storage service client, wherein the object storage service client and the client communicate based on a communication protocol with data push functionality, the method being used to query data from an object storage device, the method comprising: The system receives a query request sent by a client. The query request carries identification information for at least two data segments and an object identifier of the object to which the file to be queried belongs. The at least two data segments are obtained by dividing the data to be queried in the file to be queried, and the data in the same data segment is stored in a contiguous location. One of the at least two data segments is used as the data segment to be requested in the query request, and the other data segments in the at least two data segments are used as data segments to be pushed. For the requested data segment, data in the requested data segment is retrieved from the object storage device based on the identification information and the object identifier, and returned to the client as the response result of the query request; For each data segment to be pushed, the data of each data segment to be pushed is obtained sequentially based on the identification information, and then pushed to the client as push data.

7. The method according to claim 6, wherein the requested data segment is determined based on the identification information of the at least two data segments in the order of the query request; or The query request also carries indication information for indicating the requested data segment, which is determined based on the indication information.

8. The method according to claim 6, wherein the communication protocol is the HTTP 2.0 protocol; and / or The step of retrieving data from the requested data segment from the object storage device based on the identification information and the object identifier includes: Based on the object identifier, obtain the metadata of the object to which the file to be queried belongs; Based on the metadata and the identification information, the storage location of the data in the requested data segment is determined, and the data is obtained based on the determined storage location.

9. The method according to claim 8, after obtaining the data in the requested data segment from the object storage device based on the identification information and the object identifier, the method further includes: Cache the metadata locally; The step of sequentially obtaining data for each data segment to be pushed based on the identification information includes: For each data segment to be pushed, the storage location of the data in the data segment to be pushed is determined based on the cached metadata and the identification information, and the data in the data segment to be pushed is obtained based on the determined storage location.

10. A data query system, comprising a client and an object storage service terminal, wherein the client and the object storage service terminal communicate based on a communication protocol with data push functionality; The client is used to perform the method described in any one of claims 1-5 above; The object storage service client is used to execute the method described in any one of claims 6-9.

11. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program, when executed, implements the method of any one of claims 1-5 and / or 6-9.

12. A computer storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1-5 and / or 6-9.

Images