Data query method, medium, device and program product based on cloud search service

Abstract

A data query method, medium, equipment and program product based on a cloud search service, relate to the technical field of computers, and the data of all pages included in the file maintained by the cache page of the local cache file is stored, when a query request is received, the first offset information of the target page and the first file identifier to which the query request is directed are determined through the file metadata, the first offset information is used to point to the cache page in the local cache file for storing the pages included in the file, and the data corresponding to the target page is read from the local cache file based on the first offset information and the first file identifier. Since the first offset information for pointing to the cache page in the local cache file for storing the pages included in the file is included in the file metadata, each cache page corresponds to a piece in the local cache file, and the use of hash calculation can be avoided when data is read, thereby improving the performance of data reading.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a data query method, medium, device, and program product based on cloud search services. Background Technology

[0002] In a distributed search system, when accessing an index file stored in remote storage, the index file is downloaded to local storage in blocks of a certain size as a local cache, thereby accelerating the querying of frequently accessed data in the remote storage. Generally, distributed search systems use hash tables and reference counting to cache index files in remote storage. This requires recalculating hash values ​​and adjusting reference counts every time file blocks are allocated and released, leading to increased overall read latency. Summary of the Invention

[0003] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] Firstly, this disclosure provides a data query method based on cloud search services, including:

[0005] In response to a query request, based on the file metadata corresponding to the file maintained by the cloud search service shards, the first offset information and the first file identifier corresponding to the target page targeted by the query request are determined. The file metadata of the file includes the first offset information of the pages included in the file and the first file identifier of the file. The shard stores the data of the pages included in the file through the cache pages of the local cache file. The first offset information is used to point to the cache page in the local cache file used to store the pages included in the file.

[0006] Based on the first offset information corresponding to the target page, the cache page corresponding to the first offset information in the local cache file is accessed, and if the second file identifier associated with the cache page is consistent with the first file identifier, the data of the cache page is read to obtain the data corresponding to the target page.

[0007] Secondly, this disclosure provides a data query device based on cloud search services, including:

[0008] The determination module is configured to, in response to a query request, determine the first offset information and the first file identifier corresponding to the target page targeted by the query request based on the file metadata corresponding to the file maintained by the shards of the cloud search service. The file metadata of the file includes the first offset information of the pages included in the file and the first file identifier of the file. The shard stores the data of the pages included in the file through the cache pages of the local cache file. The first offset information is used to point to the cache page in the local cache file used to store the pages included in the file.

[0009] The reading module is configured to access the cache page in the local cache file corresponding to the first offset information based on the first offset information corresponding to the target page, and read the data of the cache page if the second file identifier associated with the cache page is consistent with the first file identifier, thereby obtaining the data corresponding to the target page.

[0010] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect.

[0011] Fourthly, this disclosure provides an electronic device, comprising:

[0012] A storage device on which computer programs are stored;

[0013] A processing device for executing the computer program in the storage device to implement the steps of the method described in the first aspect.

[0014] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] Based on the above technical solution, the data of all pages included in a file are maintained by storing cache pages in the local cache file. When a query request is received, the first offset information and the first file identifier of the target page for the query request are determined by the file metadata. The first offset information is used to point to the cache page in the local cache file used to store the pages included in the file. Then, based on the first offset information corresponding to the target page, the cache page in the local cache file corresponding to the first offset information is accessed. If the second file identifier associated with the cache page is consistent with the first file identifier, the data of the cache page is read to obtain the data corresponding to the target page. Since the file metadata includes the first offset information corresponding to the cache page in the local cache file used to store the pages included in the file, each cache page is equivalent to a slice in the local cache file. Hash calculations can be avoided when reading data, thereby improving the performance of data reading.

[0016] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0017] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0018] Figure 1 This is an architecture diagram of a cloud search service illustrated according to some embodiments.

[0019] Figure 2 This is a flowchart illustrating a data query method based on a cloud search service, according to some embodiments.

[0020] Figure 3 This is a schematic diagram of the structure of a local cache file, as shown in some embodiments.

[0021] Figure 4 This is a schematic diagram of the structure of a second type of cache page, as shown in some embodiments.

[0022] Figure 5 This is a schematic diagram illustrating the structure of metadata for a local cache file, based on some embodiments.

[0023] Figure 6 This is a schematic diagram of the array structure shown according to some embodiments.

[0024] Figure 7 This is a schematic diagram illustrating data reading according to some embodiments.

[0025] Figure 8 This is a schematic diagram of the structure of a data query device based on cloud search services, as shown in some embodiments.

[0026] Figure 9 This is a schematic diagram of the structure of an electronic device according to some embodiments. Detailed Implementation

[0027] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0028] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0029] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0031] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0032] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0033] The data query method based on cloud search services provided in this disclosure is applicable to cloud search services. The cloud search service referred to in the embodiments of this disclosure can be understood as a one-stop information retrieval and analysis platform, applicable to business scenarios such as full-text search, vector search, hybrid search, AI search, and spatiotemporal retrieval. For example, in a log analysis system, the cloud search service can be used to store and query log data. In an e-commerce platform, the cloud search service can be used to provide product search functionality, offering a fast and accurate search experience. In a big data monitoring system, the cloud search service can be used to store and analyze monitoring data to help users promptly identify potential problems.

[0034] In practical applications, cloud search services can be deployed in a distributed manner across multiple nodes (such as cloud servers). Specifically, cloud search services can be deployed across multiple nodes using a storage-compute separation architecture. The following section, in conjunction with the appendix... Figure 1 The storage and compute separation architecture of the cloud search service is described in detail.

[0035] Figure 1 This is an architecture diagram of a cloud search service illustrated according to some embodiments. For example... Figure 1As shown, the cloud search service comprises multiple compute nodes and storage nodes, which work collaboratively to provide distributed search and analysis capabilities. Compute nodes provide data processing capabilities, while storage nodes deploy remote storage to provide data storage capabilities. Remote storage (also known as centralized storage or shared storage) refers to a storage architecture in a computer network or cluster where multiple servers or virtual machines can access and use the same storage device or file system. Figure 1 As shown, compute nodes 1, 2, and N serve as compute nodes for deploying primary and replica shards. P0, P1, ..., PN represent primary shards deployed on a single compute node, while R0, R1, ..., RN represent the corresponding replica shards. It's important to note that primary shards and their corresponding replica shards can be deployed on different nodes. Furthermore, the number of compute nodes and storage nodes can be elastically scaled up or down according to the needs of the cloud search service.

[0036] In cloud search services, an index is a data structure used to store and search documents. An index can store a massive amount of data that exceeds the hardware limitations of a single node. For example, an index with 1 billion documents might require 1TB of disk space, which may not be available on any single node, or the response time for processing search requests on a single node might be too slow. Therefore, in cloud search services, indexes are divided into multiple shards, such as... Figure 1 As shown, the index can be divided into N shards, P0, P1, ..., PN. Each shard stores a portion of the data in the index. Each shard is essentially a fully functional and independent "index," and each shard is deployed on different nodes.

[0037] Each shard can include a primary shard and replica shards. The primary shard stores the actual index files and is responsible for reading and writing them. The number of primary shards is defined when the index is created and cannot be changed after creation. Replica shards are copies of the primary shards. Each primary shard can have zero or more replica shards. Replica shards copy the index files stored in the primary shard and are responsible for reading them. Replica shards improve the high availability, fault tolerance, and query performance of cloud search services. Replica shards copy the index files of their corresponding primary shards to ensure redundancy and security. When a primary shard fails, the corresponding replica shard can be promoted to a new primary shard to continue providing search services.

[0038] exist Figure 1In the compute-deposit separation architecture shown, the primary shard can store the index file in remote storage. This means that the compute node hosting the primary shard does not store the actual index file; instead, it maintains metadata indicating the storage path of the index file in remote storage. Similarly, in this architecture, the replica shard copies not the index file from the corresponding primary shard, but rather metadata indicating the storage path of the index file in remote storage. In other words, the replica shard also maintains metadata. When the primary or replica shard receives a query request from a client, it uses the metadata to determine the storage path of the data to be queried in remote storage and reads the data from remote storage using that path.

[0039] Therefore, under a storage-compute separation architecture, Figure 1 The compute nodes shown are responsible for data computation and do not actually store index files, while the storage nodes are responsible for actually storing index files. This enables the cloud search service to separate computation and storage, thereby reducing the local storage cost of compute nodes.

[0040] A client can send a write request to any compute node in the cloud search service. The compute node receiving the write request acts as a coordinating node, forwarding the write request to the corresponding primary shard's compute node. The primary shard processes the write request, writes the corresponding index file to the primary shard, and synchronizes the index file to the corresponding replica shard. After confirming successful write, the coordinating node returns a response to the client. Alternatively, while processing a write request, the primary shard can directly write the corresponding index file to remote storage and maintain metadata on the primary shard indicating the storage path of the index file in remote storage. The primary shard can also synchronously synchronize this metadata to the corresponding replica shard. When a primary or replica shard receives a query request from the client, it reads the required field values ​​from remote storage using the metadata, thus enabling the cloud search service to achieve storage-compute separation.

[0041] Of course, to improve the read performance of remote storage, the primary and replica shards of the cloud search service can cache some data in remote storage through local cache files to accelerate reads from remote storage. For example, when a query request is received, the primary and replica shards can directly read the data requested by the query request from the local cache file. If reading the data requested by the query request from the local cache file fails, the data requested by the query request will be read from the remote storage.

[0042] It is worth noting that the primary or replica shard can maintain file metadata corresponding to the index file it is responsible for. This file metadata includes the aforementioned metadata, the first offset information of the pages included in the index file, and the first file identifier of the index file. The primary or replica shard stores the data of the pages included in the file through cache pages in its local cache file. The first offset information is used to point to the cache page in the local cache file used to store the pages included in the file. The first offset information and the first file identifier will be explained in detail later.

[0043] The data query method based on cloud search service provided in the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. Figure 2 This is a flowchart illustrating a data query method based on a cloud search service, according to some embodiments. For example... Figure 2 As shown, this disclosure provides a data query method based on cloud search services, specifically executed by a data query device based on cloud search services, which can be implemented in software and / or hardware. For example... Figure 2 As shown, the method may include the following steps.

[0044] In step 210, in response to the query request, the first offset information and the first file identifier corresponding to the target page targeted by the query request are determined according to the file metadata corresponding to the file maintained by the shards of the cloud search service. The file metadata of the file includes the first offset information of the page included in the file and the first file identifier of the file. The shard stores the data of the page included in the file through the cache page of the local cache file. The first offset information is used to point to the cache page in the local cache file used to store the page included in the file.

[0045] In this embodiment of the disclosure, the shard of the cloud search service can be a shard that receives a query request sent by a client, such as a primary shard or a replica shard. The client can send a query request to any node in the cloud search service. The node receiving the query request acts as a coordinating node, randomly forwarding the query request to the corresponding primary shard or replica shard. When the primary shard or replica shard receives the query request, it determines the first offset information and the first file identifier corresponding to the target page of the query request based on the file metadata maintained by the primary shard or replica shard.

[0046] For example, a query request may include filter conditions, which are conditions used to limit the range of returned results by specifying specific rules during data retrieval. For instance, a filter condition in a query request could be used to indicate that the price of product A is no greater than 500. The target page for the query request could be a page that matches the filter conditions included in the query request.

[0047] It should be understood that the files maintained by the cloud search service in shards refer to the files managed by that shard. Each file maintained by a shard of the cloud search service will have corresponding file metadata maintained on that shard. This file metadata includes the first offset information corresponding to all pages included in the file and the first file identifier of the file. In this embodiment of the disclosure, the files maintained by the cloud search service in shards may refer to index files maintained by shards.

[0048] In this cloud search service, each shard stores the data of all pages in the file it is responsible for maintaining through cache pages in the local cache file. In other words, the local cache file caches data at the page level.

[0049] In some embodiments, the local cache file includes at least one partition file, the partition file includes multiple cache pages, the multiple cache pages include cache pages of a first type, the first type of cache pages are used to store data of the pages included in the file.

[0050] In some embodiments, the partition file further includes a second type of cache page, which is used to store persistent state information corresponding to the first type of cache page.

[0051] Figure 3 This is a schematic diagram illustrating the structure of a local cache file according to some embodiments. For example... Figure 3 As shown, the local cache files can include partition files of type Partition 1, Partition 2, ..., Partition N. Each partition file includes cache pages of type 1 and type 2. For example, each partition file can include N cache pages of type 2, such as PageStateGroup 1, ..., PageStateGroup N, and each partition file can include M cache pages of type 1, such as DataPage 1, ..., DataPage M.

[0052] It should be understood that the first type of cache page serves as the page that actually stores the file data in the partition file, while the second type of cache page serves as the header field in the partition file, used to store persistent state information corresponding to all the first type of cache pages included in the partition file. The persistent state information includes at least the first offset information associated with the first type of cache page and the first file identifier. This persistent state information is used, during a shard restart, based on the persistent state information and the memory mapping mechanism, to map the partition file into the virtual address space of the shard, thereby supporting access to the partition file within the virtual address space.

[0053] Figure 4This is a schematic diagram illustrating the structure of a second type of cache page according to some embodiments. For example... Figure 4 As shown, the second type of cached page, "PageStateGroup 1," includes multiple slots. Slots of type "Data page state slot" are used to store the first offset information (PAGE_INDEX) and the first file identifier (FILE_ID) associated with the first type of cached page. Of course, if the persisted state information includes the page state of the first type of cached page, slots of type "Data pagestate slot" can also be used to record the page state (PAGE_STATE) of the first type of cached page. Each slot can be 64 bytes in size, FILE_ID is 8 bytes, PAGE_INDEX is 4 bytes, PAGE_STATE is 1 byte, and padding is 51 bytes.

[0054] It's important to note that the sizes of both Type 1 and Type 2 cache pages can be configured according to requirements. For example, the size of both types of cache pages can be 256KB. Assuming a partition is 1GB in size, a partition includes 4096 pages, comprising one Type 2 cache page and 4095 Type 1 cache pages. Since each slot in a Type 2 cache page is 64B, one Type 2 cache page consists of 4096 slots. One slot (header pagestate slot 1) stores the state information of the Type 2 cache page, while the remaining 4095 slots store persistent information corresponding to the persistent information of each of the 4095 Type 1 cache pages.

[0055] Of course, local cache files can also have their own metadata. Figure 5 This is a schematic diagram illustrating the structure of metadata for local cached files, based on some embodiments. For example... Figure 5 As shown, the metadata "MetaFile" of the local cache file includes information "partitionSize:long" to describe the size of the partition file, "pageSize:int" to describe the size of the cache page, and "partitionNo:int" to describe the number of partition files.

[0056] In this embodiment, the cache uses cache pages as the granularity, serving as the basic unit for cache allocation, eviction, and access. Accessing a specific LBA (Logical Block Address) in a file allows for retrieving the corresponding cache page and reading via direct addressing, completely avoiding read-write operations and hash calculations. Furthermore, the cache resides on the local disk of the compute node, existing as partition files. The cache can be dynamically expanded or reduced by increasing or decreasing the number of partition files during runtime. Each partition file is divided into multiple cache pages at the cache page level.

[0057] Accordingly, for each file maintained by a shard, corresponding file metadata is maintained on that shard. This file metadata includes at least the first offset information for all pages included in the file and the first file identifier of the file. The first file identifier refers to the file identifier of the file to which the page belongs. If the file is an index file, the first file identifier may include the file name corresponding to the index file. The first offset information is used to point to the cache page in the local cache file used to store the pages included in the file. For example, the first offset information indicates which page in which partition file the data of each page in the file is cached.

[0058] In some embodiments, file metadata records the first offset information of the pages included in the file in the form of an array. The array includes multiple elements for recording the first offset information, and each element corresponds to a page included in the file.

[0059] For example, the file metadata for each file may include an array that records the first offset information for all pages included in the file.

[0060] Figure 6 This is a schematic diagram of the array structure shown according to some embodiments. For example... Figure 6 As shown, the array contains multiple elements, each corresponding to a page in the file. For example, the first element 601 corresponds to the first page of the file, the second element 602 corresponds to the second page, and the third element 603 corresponds to the last page. The value "1" in the first element 601 represents the first offset information corresponding to the first page in the file, indicating that the first page is stored in the first cache page of the partition file. The value "-1" in the second element 602 represents the first offset information corresponding to the second page in the file, indicating that the second page is stored in the -1 cache page of the partition file. The value "30" in the third element 603 represents the first offset information corresponding to the last page in the file, indicating that the last page is stored in the 30th cache page of the partition file.

[0061] It's important to note that the file metadata records the first offset information of all pages included in the file in the form of an array. Regardless of whether a page is actually stored in the local cache file, this array will always contain its first offset. However, pages not actually stored in the local cache file will not have their data retrieved from the local cache file. For example, in... Figure 6 In the array shown, the first offset information corresponding to the second page of the file is "-1". There is no cached page for page -1 in the partition file, so the corresponding cached page will not be read in the partition file.

[0062] It's worth noting that the data stored in the local cache file can come from remote storage. For example... Figure 1 As shown, compute nodes can cache data from remote storage on storage nodes in local cache files on the compute node. Of course, if the cloud search service does not include remote storage, the data stored in the local cache file can come from local files. The local cache file acts as local memory, caching local files, and data can be read from the corresponding cache pages in memory using Direct I / O (unbuffered I / O).

[0063] In step 220, based on the first offset information corresponding to the target page, the cache page corresponding to the first offset information in the local cache file is accessed, and if the second file identifier associated with the cache page is consistent with the first file identifier, the data of the cache page is read to obtain the data corresponding to the target page.

[0064] Here, after obtaining the first offset information and the first file identifier corresponding to the target page, the cache page corresponding to the first offset information in the local cache file can be accessed through the first offset information. For example, Figure 6 As shown, if the target page is the first page in the file, the first cached page in the partition file is accessed. Then, it is checked whether the second file identifier associated with the cached page is the same as the first file identifier. If the second file identifier associated with the cached page is the same as the first file identifier, the data of the cached page is read to obtain the data corresponding to the target page.

[0065] like Figure 4 As shown, the second file identifier (PAGE_INDEX) associated with the cache page is stored in the second type of cache page. The second file identifier associated with the cache page corresponding to the first offset information can be read from the second type of cache page; this second file identifier indicates the file to which the data actually stored in the cache page belongs.

[0066] It should be noted that if the second file identifier is the same as the first file identifier, it means that the file to which the target page belongs, as recorded in the file metadata, is the same as the file to which the target page belongs, and the data corresponding to the target page can be read from the cache page. If the second file identifier is different from the first file identifier, it means that the data stored in the cache page has changed and is inconsistent with the file to which the target page belongs, as recorded in the file metadata. In this case, it is necessary to determine the cache page to which the target page belongs from the file metadata.

[0067] Therefore, by storing data for all pages of a file in a local cache file through cached pages, when a query request is received, the first offset information and first file identifier of the target page for the query request are determined through the file metadata. This first offset information is used to point to the cache page in the local cache file used to store the pages included in the file. Then, based on the first offset information corresponding to the target page, the cache page in the local cache file corresponding to the first offset information is accessed. If the second file identifier associated with the cache page matches the first file identifier, the data of the cache page is read to obtain the data corresponding to the target page. Since the file metadata includes the first offset information corresponding to the cache page in the local cache file used to store the pages included in the file, each cache page is equivalent to a slice in the local cache file. Hash calculations can be avoided during data reading, thereby improving the performance of data reading.

[0068] It is worth noting that in this embodiment of the disclosure, the main shard of the cloud search service can upload the index file to remote storage for storage. Therefore, the computing node where the main shard is located may not mount a local disk or may mount a small number of local disks, thereby saving local storage resources.

[0069] For example, in response to the need for migration of the first primary shard of the cloud search service, a first index number currently used by the first primary shard can be determined. This first index number is used by the first primary shard to determine the filename corresponding to the index file written to it. Then, based on the migration method of the first primary shard, a target step size is determined. Based on the target step size and the first index number, a second index number corresponding to the second primary shard used to replace the first primary shard is determined. The target step size ensures that the second index number is greater than the first index number. Next, during the migration from the first primary shard to the second primary shard, the filename corresponding to the index file written to the second primary shard is determined based on the second index number, so that the filename corresponding to the index file written to the second primary shard is different from the filename of the index file in the first primary shard.

[0070] In this context, the first primary shard of the cloud search service can refer to any primary shard within the cloud search service. The migration of the first primary shard can be triggered by factors such as the node hosting the first primary shard becoming unavailable or a rebalancing of load. For example, if the node hosting the first primary shard exits, the replica shard corresponding to the first primary shard will be promoted to the second primary shard. Alternatively, if the node hosting the first primary shard runs out of disk space or requires load balancing, the first primary shard will migrate from one node to another.

[0071] During the migration of the first primary shard, it will be in a "migrating" or "partially available" state. The first primary shard still exists, but new write operations will be blocked and must wait for the migration to complete. The second primary shard is created on the new node to receive data migrated from the first primary shard. During the migration of the first primary shard, the second primary shard starts from a blank state or a replica shard state, then receives data from the first primary shard. Once data replication is complete, the second primary shard is marked as "active" and begins processing read and write requests. Therefore, when the first primary shard needs to be migrated, the second primary shard can be understood as a new primary shard formed by migrating the old first primary shard.

[0072] In other words, in this embodiment of the disclosure, the second primary shard is the primary shard that takes over the role of the old first primary shard. The second primary shard becomes the primary shard of the shard corresponding to the old first primary shard. The second primary shard is used to take over the working logic of the first primary shard and is responsible for handling read and write requests, while the old first primary shard will be deleted after verification.

[0073] The first index number currently used by the first primary shard refers to the index number currently being used by the first primary shard to determine the filename of the index file written to remote storage. In the cloud search service, the first primary shard maintains a segment counter to record changes in the number of segment files maintained by the first primary shard. Whenever a new segment file is created in the first primary shard, the segment counter increments accordingly. Accordingly, the first index number currently used by the first primary shard could refer to the segment counter value when the first primary shard needs to be migrated.

[0074] It is important to note that in cloud search services, index data is segmented into multiple segment files. Each segment file is a complete and immutable index structure, possessing its own dictionary, inverted index, and document storage data structures. Therefore, in this embodiment, the index file can refer to each segment file.

[0075] When a second primary shard needs to be launched, the engine is used to initialize and run it. Therefore, as examples, the engine used to launch the second primary shard can determine the second index number corresponding to the second primary shard based on the first index number currently used by the first primary shard and the target step size. The target step size is related to the migration method of the primary shard and is used to ensure that the second index number is greater than the first index number.

[0076] For example, the sum of the first index number and the target step size can be used to determine the second index number. For instance, assuming the first index number is 100 and the target step size is 200000, the second index number could be 200100. In this embodiment, when the first primary shard needs to be migrated, the second primary shard uses a larger second index number to determine the filename of the index file stored in remote storage, thereby avoiding conflicts between the filename of the index file written to remote storage by the second primary shard and the filename of the index file written to remote storage by the old first primary shard.

[0077] For example, the engine used to launch the second primary shard can be a read / write engine. In embodiments of this disclosure, a second index number used to determine the file name of the index file can be controlled at the engine level, thereby avoiding the generation of index files with the same file name but different file content.

[0078] It should be understood that the target step size is a dynamic step size, and the target step size will be different depending on the migration method of the first primary shard. In the embodiments of this disclosure, there are two migration methods for the first primary shard: one is that the replica shard corresponding to the first primary shard is upgraded to the second primary shard, and the other is that the first primary shard is directly migrated to another node.

[0079] For example, if the migration method of the first primary shard represents the upgrade of the corresponding replica shard to the second primary shard, the target step size can be calculated using the tenure of the second primary shard. If the migration method of the first primary shard represents a direct migration to another node, the target step size can be determined using the number of migrations already performed on the first primary shard. By using the target step size associated with the migration method of the first primary shard, the second index number can be dynamically adjusted. This allows for the use of a larger second index number to define the filename of the index file in the second primary shard during the migration of the first primary shard, thus avoiding conflicts between the filename of the index file written to remote storage by the second primary shard and the filename of the old index file written to remote storage by the first primary shard.

[0080] It is worth noting that, such as Figure 1As shown, the first primary shard uploads its index file to remote storage. When the first primary shard migrates, the second primary shard generates a new index file and uploads it to remote storage. However, since the old first primary shard continues to write data during the process of the second primary shard replacing the old one, directly using the first index number as the second index number for the second primary shard would result in index files with the same name existing in remote storage. By using a second index number greater than the first index number, the generation of index files with the same name on remote storage can be avoided, ensuring that the index file of the second primary shard can be successfully uploaded to remote storage during the migration of the first primary shard, thus supporting a compute-storage separation architecture.

[0081] After determining the second index number, the filename of the index file written to remote storage by the second primary shard can be determined using this second index number. For example, the second index number can be used as the initial number of the segment counter maintained by the second primary shard. Whenever a new index file is created in the second primary shard, the filename of that index file is determined based on the number recorded in the segment counter. Furthermore, the segment counter count is incremented accordingly. Consequently, the filenames of subsequent index files written to remote storage by the second primary shard will also be determined using a larger second index number.

[0082] It is worth noting that since the second index number is determined by the engine used to start the second primary shard, the second index number of each second primary shard is determined by the engine corresponding to the second primary shard. This is done in a decentralized manner. Furthermore, since the second index number is greater than the first index number, the filename of the index file written to remote storage by the second primary shard will be different from the filename of the index file written to remote storage by the old first primary shard.

[0083] Because the second index number is greater than the first index number, the filename of the index file written to remote storage by the second primary shard will be different from the filename of the index file written to remote storage by the old first primary shard.

[0084] Because the filename of the index file written to the remote storage by the second primary shard is different from the filename of the index data written to the remote storage by the old first primary shard, it is possible to support the first primary shard and its corresponding replica shard to directly query the index file in the remote storage to read data.

[0085] In some feasible implementations, the cloud search service also includes remote storage, where shards of the cloud search service are used to upload files maintained by the shards to the remote storage for storage, and the file metadata also includes metadata indicating the storage path of the pages included in the file in the remote storage.

[0086] Among them, such as Figure 1 As shown, when the cloud search service includes remote storage for storing files for shard maintenance, the shards can upload the files being maintained to the remote storage for storage, thus achieving a storage-compute separation architecture. Correspondingly, the file metadata can record metadata indicating the storage paths of the pages included in the file within the remote storage.

[0087] If accessing the cache page corresponding to the first offset information in the local cache file fails, the data corresponding to the target page can be read from the remote storage based on the metadata corresponding to the target page.

[0088] Here, if it fails to read the data corresponding to the target page from the local cache file based on the first offset information and the first file identifier, it means that the data corresponding to the target page is not cached in the local cache file. In this case, the metadata corresponding to the target page can be obtained from the file metadata, and then the data corresponding to the target page can be read from the remote storage through the metadata corresponding to the target page.

[0089] Therefore, through the above implementation method, files maintained by shards can be stored in remote storage, realizing storage-computation separation. Moreover, if the local cache file does not cache the data of the target page, the data corresponding to the target page can be read by directly accessing the remote storage.

[0090] In some feasible implementations, at least one cache page in an available state is requested from the local cache file, and the data corresponding to the target page read from the remote storage is stored in the cache page. The first offset information and the first file identifier in the file metadata are modified to point to the cache page and the second file identifier associated with the cache page.

[0091] Here, following the above implementation method, the local cache file stores data at the granularity of cache pages. A cache page in an available state can be one that can be allocated to store new data.

[0092] When data corresponding to the target page is read from remote storage, at least one cache page in an available state can be requested from the local cache file. The requested at least one cache page in an available state is used to store the data corresponding to the target page read from remote storage.

[0093] After storing the data corresponding to the target page read from remote storage in the cache page, the first offset information and the first file identifier in the file metadata can be modified to the second offset information and the second file identifier associated with the cache page, thereby forming a one-to-one correspondence between the file metadata and the cache page of the local cache file.

[0094] like Figure 6 As shown, the first offset information corresponding to the second page of the file is "-1". Since there is no cached page for page -1 in the partition file, the corresponding cached page will not be read from the partition file. At this point, the data corresponding to the second page of the file can be read from remote storage. After the data corresponding to the second page of the file read from remote storage is written to the cached page of the local cache file, the first offset information recorded in the second element 602 of the array will be modified to the corresponding second offset information. Assuming that the data corresponding to the second page of the file read from remote storage is written to the 51st cached page of the local cache file, the first offset information "-1" in the second element 602 will be modified to "51". When reading the data corresponding to the target page subsequently, the data corresponding to the target page can be read from the local cache file based on the first offset information "51".

[0095] Following the above implementation, if the data corresponding to the target page cannot be read from the local cache file, a foreground read thread can read the data corresponding to the target page from remote storage based on the metadata of the target page. Simultaneously, an asynchronous thread is started to store the data corresponding to the target page read from the remote storage in the cache page. Then, the data corresponding to the target page is written to disk using msync (forcibly flushing the data already written to memory-mapped disk) in the asynchronous thread.

[0096] Therefore, by storing the data corresponding to the target page read from remote storage in the cache pages included in the local cache file, the cross-media read performance of the cloud search service can be improved in cross-media storage scenarios between remote storage and local disk, enabling the cloud search service to provide read and write performance close to that of local files.

[0097] Figure 7 This is a schematic diagram illustrating data reading according to some embodiments. For example... Figure 7 As shown, the file metadata includes an array that records the first offset information of all pages included in the corresponding file. This array indicates the storage path of each page (e.g., Page1, Page2, ..., PageN) in the local cache file. Furthermore, the file metadata includes metadata indicating the storage path of the pages included in the file in remote storage.

[0098] During data reading, the system first uses the array of file metadata to determine whether the data corresponding to the target page can be read from the local cache file. If the data corresponding to the target page cannot be read, the system accesses the remote storage using the metadata included in the file metadata and reads the data corresponding to the target page from the remote storage.

[0099] In some feasible implementations, a target cache page whose recent access time meets a preset condition can be determined based on the recent access time of the cache pages included in the local cache file. Then, the status of the target cache page is adjusted to an available state, wherein a cache page in an available state can be allocated to store new data.

[0100] Here, the most recently accessed time (LIGHT) of a cached page refers to the last time the cached page was accessed. Whenever a cached page in the local cache file is read, its access time is recorded. The target cached page whose LIGHT meets the preset condition can be the least recently used cached page in the local cache file. For example, the LIGHTs can be sorted chronologically, and the least recently used cached page can be selected as the target cached page. Of course, the target cached page whose LIGHT meets the preset condition can also refer to the top-ranked cached pages.

[0101] It should be understood that the preset conditions are used to characterize the data stored in the target cache page as cold data. A target cache page whose recent access time meets the preset conditions can be understood as a cache page whose data stored in the local cache file is cold data.

[0102] Since the storage space of local cache files is limited, it is necessary to evict cold data stored in the cache pages included in the local cache files. In this embodiment, an asynchronous thread can traverse the linked list of cache pages, determine the popularity of cache pages based on their most recent access time, identify target cache pages whose most recent access time meets preset conditions, and adjust the status of the target cache pages to an available state. Following the above implementation method, cache pages in an available state can be allocated to store data read from remote storage. It should be noted that the asynchronous thread can also move the most popular cache page to the head of the linked list.

[0103] Therefore, by evicting cached pages based on their most recent access time, cold data in local cache files can be released, ensuring the read and write performance of the cloud search service.

[0104] In some feasible implementations, in step 220, the cache page corresponding to the first offset information in the local cache file can be accessed based on the first offset information corresponding to the target page; if the second file identifier associated with the cache page is consistent with the first file identifier, the data of the cache page is read to obtain the data corresponding to the target page; if the data corresponding to the target page is obtained, it is detected whether the first offset information and the first file identifier in the file metadata have changed; if the first offset information and the first file identifier in the file metadata have not changed, it is determined that the data corresponding to the target page obtained is valid.

[0105] Here, following the above implementation method, during the process of reading the data corresponding to the target page based on the first offset information and the first document identifier, the cloud search service will also eliminate cached pages based on the most recent access time. During this process, the cached page corresponding to the first offset information may be eliminated. In order to ensure the accuracy of the data read from the cached page corresponding to the first offset information, in this embodiment of the disclosure, when the data corresponding to the target page is read, it is detected whether the first offset information and the first document identifier in the file metadata have changed. If the first offset information and the first document identifier in the file metadata have not changed, it is confirmed that the data corresponding to the target page read is valid. If the first offset information and the first document identifier in the file metadata have changed, it is determined that the data corresponding to the target page read is invalid, and it is necessary to re-determine the first offset information and the first document identifier corresponding to the target page.

[0106] Detecting whether the first offset information and the first file identifier in the file metadata have changed refers to checking whether the first offset information and the first file identifier in the file metadata before reading have changed relative to the first offset information and the first file identifier in the file metadata after reading. Determining that the data corresponding to the obtained target page is valid means that the data read based on the first offset information and the first file identifier is valid, and the data corresponding to the read target page can be returned to the client.

[0107] It should be understood that, when the data corresponding to the target page is read, checking whether the first offset information and the first file identifier in the file metadata have changed is essentially using optimistic locking to determine whether the data corresponding to the target page is valid. This eliminates the need to lock the cached page or perform hash calculations, thus greatly ensuring read and write performance.

[0108] Therefore, by implementing the above methods, read and write performance is greatly guaranteed by eliminating the need for locking and hash calculations on cached pages.

[0109] In some feasible implementations, a memory mapping mechanism can be used to map local cache files located on the local disk to the fragmented virtual address space. Accordingly, in step 220, based on the first offset information corresponding to the target page, the cache page corresponding to the first offset information can be accessed from the virtual address space, and if the second file identifier associated with the cache page matches the first file identifier, the data of the cache page can be read to obtain the data corresponding to the target page.

[0110] Here, memory mapping (mmap) refers to mapping a file into the virtual address space of a process, establishing a mapping relationship between the physical address of the file and the virtual address space. In this embodiment of the disclosure, the local cache file is stored on the local disk. To accelerate the reading of the local cache file, a memory mapping mechanism can be used to map the local cache file stored on the local disk to the virtual address space corresponding to the fragment.

[0111] Following the above implementation method, the local cache file includes a second type of cache page. The second type of cache page is used to store persistent state information corresponding to the first type of cache page. Based on the persistent state information stored in the second type of cache page and the memory mapping mechanism, the local cache file can be mapped to the virtual address space.

[0112] After determining the first offset information and the first document identifier corresponding to the target page, the cache page corresponding to the first offset information can be accessed from the virtual address space. If the second file identifier associated with the cache page is consistent with the first file identifier, the data of the cache page can be read to obtain the data corresponding to the target page.

[0113] It is worth noting that, through the memory mapping mechanism, local cache files are read in mmap mode, which can support a cache size far exceeding the actual memory of the shards, and can also avoid complex memory space management.

[0114] Therefore, by using local disks and memory as multi-level caches, it is possible not only to accelerate access to remote storage and support high-performance reading of cloud search services, but also to support cache sizes far exceeding the actual memory size of the shards.

[0115] Figure 8 This is a schematic diagram illustrating the structure of a data query device based on cloud search services, according to some embodiments. For example... Figure 8 As shown, this embodiment of the disclosure provides a data query device 800 based on cloud search services, which includes:

[0116] The determination module 801 is configured to, in response to a query request, determine the first offset information and the first file identifier corresponding to the target page targeted by the query request based on the file metadata corresponding to the file maintained by the shards of the cloud search service. The file metadata of the file includes the first offset information of the page included in the file and the first file identifier of the file. The shard stores the data of the page included in the file through the cache page of the local cache file. The first offset information is used to point to the cache page in the local cache file used to store the page included in the file.

[0117] The reading module 802 is configured to access the cache page in the local cache file corresponding to the first offset information based on the first offset information corresponding to the target page, and read the data of the cache page if the second file identifier associated with the cache page is consistent with the first file identifier, thereby obtaining the data corresponding to the target page.

[0118] Optionally, when the cloud search service includes remote storage for storing the files maintained by the shards, the file metadata also includes metadata indicating the storage path of the pages included in the file in the remote storage; the reading module 802 is further configured to:

[0119] If the data corresponding to the target page cannot be read from the local cache file, the data corresponding to the target page is read from the remote storage based on the metadata corresponding to the target page.

[0120] Optionally, the data query device 800 based on cloud search services further includes:

[0121] The request module is configured to request at least one cache page that is in an available state from the local cache file, and to store the data corresponding to the target page read from the remote storage in the cache page;

[0122] The modification module is configured to modify the first offset information and the first file identifier in the file metadata to a second offset information pointing to the cache page and a second file identifier associated with the cache page.

[0123] Optionally, the file metadata records the first offset information of the pages included in the file in the form of an array, the array including multiple elements for recording the first offset information, each element corresponding to a page included in the file.

[0124] Optionally, the data query device 800 based on cloud search services further includes:

[0125] The time determination unit is configured to determine a target cache page whose recent access time satisfies a preset condition based on the recent access time of the cache page included in the local cache file. The preset condition is used to characterize the data stored in the target cache page as cold data.

[0126] The adjustment unit is configured to adjust the state of the target cache page to an available state, wherein the cache page in the available state can be allocated to store new data.

[0127] Optionally, the reading module 802 is specifically configured as follows:

[0128] Based on the first offset information corresponding to the target page, access the cache page in the local cache file that corresponds to the first offset information;

[0129] If the second file identifier associated with the cached page is consistent with the first file identifier, the data of the cached page is read to obtain the data corresponding to the target page;

[0130] If the data corresponding to the target page is obtained, it is detected whether the first offset information and the first file identifier in the file metadata have changed;

[0131] If the first offset information and the first file identifier in the file metadata have not changed, the data corresponding to the obtained target page is determined to be valid.

[0132] Optionally, the local cache file includes at least one partition file, the partition file includes multiple cache pages, the multiple cache pages include cache pages of a first type, the first type of cache pages being used to store data of the pages included in the file.

[0133] Optionally, the partition file further includes a second type of cache page, which is used to store persistent state information corresponding to the first type of cache page. The persistent state information includes at least a first offset information and a first file identifier associated with the first type of cache page. The persistent state information is used to map the partition file to the virtual address space of the partition based on the persistent state information and the memory mapping mechanism when the partition is restarted, so as to support access to the partition file in the virtual address space.

[0134] Optionally, the data query device 800 based on cloud search services further includes:

[0135] The mapping module is configured to map the local cache file located on the local disk to the virtual address space of the fragment through a memory mapping mechanism;

[0136] The reading module 802 is specifically configured as follows:

[0137] Based on the first offset information corresponding to the target page, the cache page corresponding to the first offset information is accessed from the virtual address space, and if the second file identifier associated with the cache page is consistent with the first file identifier, the data of the cache page is read to obtain the data corresponding to the target page.

[0138] The functional logic executed by each functional module in the aforementioned cloud search service-based data query device 800 has been explained in detail in the section on methods, and will not be repeated here.

[0139] The following is for reference. Figure 9 It shows a schematic diagram of the structure of an electronic device (e.g., a server) 900 suitable for implementing embodiments of the present disclosure. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0140] like Figure 9 As shown, electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 901, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 902 or a program loaded from storage device 908 into random access memory (RAM) 903. RAM 903 also stores various programs and data required for the operation of electronic device 900. Processing device 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0141] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0142] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0143] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0144] In some implementations, servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0145] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0146] The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the electronic device to: respond to a query request, determine, based on the file metadata corresponding to the file maintained by the shards of the cloud search service, the first offset information and the first file identifier corresponding to the target page targeted by the query request, wherein the file metadata of the file includes the first offset information of the pages included in the file and the first file identifier of the file, the shard stores the data of the pages included in the file through cache pages of the local cache file, and the first offset information is used to point to the cache page in the local cache file used to store the pages included in the file; based on the first offset information corresponding to the target page, access the cache page in the local cache file corresponding to the first offset information, and if the second file identifier associated with the cache page is consistent with the first file identifier, read the data of the cache page to obtain the data corresponding to the target page.

[0147] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0149] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0150] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0151] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0152] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0153] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0154] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A data query method based on cloud search services, characterized in that, include: In response to a query request, based on the file metadata corresponding to the file maintained by the cloud search service shards, the first offset information and the first file identifier corresponding to the target page targeted by the query request are determined. The file metadata of the file includes the first offset information of the pages included in the file and the first file identifier of the file. The shard stores the data of the pages included in the file through the cache pages of the local cache file. The first offset information is used to point to the cache page in the local cache file used to store the pages included in the file. Based on the first offset information corresponding to the target page, access the cache page in the local cache file that corresponds to the first offset information, and if the second file identifier associated with the cache page is consistent with the first file identifier, read the data of the cache page to obtain the data corresponding to the target page; The file metadata records the first offset information in the form of an array. The array includes multiple elements, each element corresponding to a page and recording the first offset information of that page.

2. The method according to claim 1, characterized in that, The cloud search service also includes remote storage, and the shards of the cloud search service are used to upload the files maintained by the shards to the remote storage for storage; the file metadata also includes metadata indicating the storage path of the pages included in the file in the remote storage; The method further includes: If accessing the cache page corresponding to the first offset information in the local cache file fails, the data corresponding to the target page is read from the remote storage based on the metadata corresponding to the target page.

3. The method according to claim 2, characterized in that, The method further includes: Request at least one cache page that is available from the local cache file, and store the data corresponding to the target page read from the remote storage in the cache page; The first offset information and the first file identifier in the file metadata are modified to the second offset information and the second file identifier associated with the cache page.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Based on the most recent access time of the cached pages included in the local cache file, a target cached page whose most recent access time meets a preset condition is determined. The preset condition is used to characterize the data stored in the target cached page as cold data. The target cache page is adjusted to an available state, wherein the cache page in the available state can be allocated to store new data.

5. The method according to claim 4, characterized in that, The step of accessing the cache page in the local cache file corresponding to the first offset information based on the target page, and reading the data of the cache page to obtain the data corresponding to the target page when the second file identifier associated with the cache page is consistent with the first file identifier, includes: Based on the first offset information corresponding to the target page, access the cache page in the local cache file that corresponds to the first offset information; If the second file identifier associated with the cached page is consistent with the first file identifier, the data of the cached page is read to obtain the data corresponding to the target page; If the data corresponding to the target page is obtained, it is detected whether the first offset information and the first file identifier in the file metadata have changed; If the first offset information and the first file identifier in the file metadata have not changed, the data corresponding to the obtained target page is determined to be valid.

6. The method according to any one of claims 1-3, characterized in that, The local cache file includes at least one partition file, the partition file includes multiple cache pages, the multiple cache pages include cache pages of a first type, the first type of cache pages are used to store data of the pages included in the file.

7. The method according to claim 6, characterized in that, The partition file also includes a second type of cache page, which is used to store persistent state information corresponding to the first type of cache page. The persistent state information includes at least a first offset information and a first file identifier associated with the first type of cache page. The persistent state information is used to map the partition file to the virtual address space of the fragment based on the persistent state information and the memory mapping mechanism when the fragment restarts, so as to support accessing the partition file in the virtual address space.

8. The method according to any one of claims 1-3, characterized in that, The method further includes: The local cache file located on the local disk is mapped to the virtual address space of the segment through a memory mapping mechanism; The step of accessing the cache page in the local cache file corresponding to the first offset information based on the target page, and reading the data of the cache page to obtain the data corresponding to the target page when the second file identifier associated with the cache page is consistent with the first file identifier, includes: Based on the first offset information corresponding to the target page, the cache page corresponding to the first offset information is accessed from the virtual address space, and if the second file identifier associated with the cache page is consistent with the first file identifier, the data of the cache page is read to obtain the data corresponding to the target page.

9. A data query device based on cloud search services, characterized in that, include: The determination module is configured to, in response to a query request, determine the first offset information and the first file identifier corresponding to the target page targeted by the query request based on the file metadata corresponding to the file maintained by the shards of the cloud search service. The file metadata of the file includes the first offset information of the pages included in the file and the first file identifier of the file. The shard stores the data of the pages included in the file through the cache pages of the local cache file. The first offset information is used to point to the cache page in the local cache file used to store the pages included in the file. The reading module is configured to access the cache page in the local cache file corresponding to the first offset information based on the first offset information corresponding to the target page, and read the data of the cache page if the second file identifier associated with the cache page is consistent with the first file identifier, thereby obtaining the data corresponding to the target page; The file metadata records the first offset information in the form of an array. The array includes multiple elements, each element corresponding to a page and recording the first offset information of that page.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processing device, it implements the steps of the method according to any one of claims 1-8.

11. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.

Images