A data query method, electronic device, medium and computer program product

Abstract

The application discloses a data query method, an electronic device, a medium and a computer program product, relates to the technical field of cloud computing, and solves the problem that when a target application program interface is called to perform a paging query, a client side lacks perception of a server side and excessively large single acquisition quantity is requested, resulting in query failure, so that the success rate of paging query data is improved and user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a data query method, electronic device, medium, and computer program product. Background Technology

[0002] In cloud service and other application architectures, clients often execute data query tasks by calling the server's Application Programming Interface (API). For queries that may return massive amounts of data, the industry commonly uses a pagination mechanism, where the client includes the "quantity to be retrieved at one time" (often called the page size or pagination capacity) in the request to retrieve data in batches. However, clients tend to set excessively large pagination capacities when initiating query requests, which can easily lead to request rejection or cause the server to experience a sudden surge in load, resulting in request failure. Summary of the Invention

[0003] This invention provides a data query method, electronic device, medium, and computer program product to at least solve the problem in related technologies where query requests sent by clients to server application programming interfaces carry excessively large amounts of data in a single retrieval, leading to query failures.

[0004] This invention provides a data query method, including: The first query request generated by the client device when the client performs a pagination data retrieval task is intercepted. The first query request includes a first pagination parameter representing the pagination capacity. Obtain the first pagination restriction parameter corresponding to the target application interface of the target server corresponding to the first query request; The first pagination parameter is determined by using the first pagination restriction parameter to obtain the second pagination parameter; The first query request is updated using the second pagination parameter to obtain the second query request; The second query request is sent to the target server to invoke the target application interface to respond to the pagination data retrieval task according to the second pagination parameters.

[0005] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-described data query method.

[0006] The present invention also provides a non-volatile storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described data query method.

[0007] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described data query method.

[0008] This invention intercepts the first query request generated by the client device when performing a paginated data retrieval task. It then adjudicates the first pagination parameter in the first query request based on the first pagination restriction parameter corresponding to the target application interface of the target server, obtaining a second pagination parameter. This second pagination parameter is used to update the first query request, resulting in a second query request. The second query request is then sent to the target server to invoke the target application interface to respond to the paginated data retrieval task according to the second pagination parameter. In other words, by combining the capabilities of the application interface, the quantity of target data requested by the client in a single retrieval is limited. This solves the problem of query failures caused by the client requesting excessively large single retrieval quantities due to a lack of awareness of the server side when calling the application interface for paginated queries. This improves the success rate of paginated data retrieval and enhances the user experience. Attached Figure Description

[0009] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 An architecture diagram of a data query system provided in an embodiment of the present invention; Figure 2 A flowchart of a data query method provided in an embodiment of the present invention; Figure 3 This is a functional configuration diagram of a data query system provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0012] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0013] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] Figure 1 This is an architecture diagram of a data query system provided in an embodiment of the present invention.

[0015] The specific application environment architecture or specific hardware architecture on which the execution of the data query method depends is described here.

[0016] like Figure 1 As shown, the data query method provided in this embodiment of the invention can be applied to a client-server architecture computing system. The data query method provided in this embodiment of the invention can be implemented based on an interception component 102, which can be deployed on the client device 101, or on an interception device other than the client device 101.

[0017] The client device 101 is a terminal device or computing node that initiates the pagination data retrieval request, and its form includes, but is not limited to, a personal computer, smartphone, tablet, IoT device, virtual machine instance, or containerized application runtime environment. The client device 101 runs a client application or intelligent agent program, which generates a first query request containing the first pagination parameters by performing tasks such as data querying.

[0018] Interception component 102 is deployed on the communication path between client device 101 and target server 103 to intercept and process query requests issued by the client. This interception component 102 can exist in different forms, including but not limited to: a software proxy or hook program deployed on client device 101, a gateway service independently deployed on the client side, a proxy server running at the network edge, or a client-side adaptation component of an Application Programming Interface (API) gateway. This interception middleware has the ability to obtain target API restriction parameters, determine pagination parameters, and update query requests.

[0019] Target server 103 is one or more physical servers or a cluster of cloud servers providing API services. Target server 103 has a target application programming interface (API) deployed on it to receive and respond to query requests from clients, returning corresponding batches of data according to the pagination parameters in the request. Target server 103 can connect to a database or other storage system to obtain the actual data.

[0020] The client device 101 is connected to the target server 103 via a network communication link, which may include a local area network, a wide area network, the Internet, a mobile communication network, or the internal network of a cloud service provider. The interception component 102 can be deployed on any node of this link.

[0021] It should be noted that the interception and processing of client requests by the interception component 102 provided in this embodiment of the invention can be transparent to the client device 101. That is, the client device 101 does not need to be aware of the existence of the interception component 102 and still generates and sends requests according to the original logic. The interception component 102 can also run in proxy mode locally on the client device 101. In this case, the client device 101 actively sends the request to the local proxy, which processes it and then forwards it to the target server 103.

[0022] The embodiments of the present invention provide a data query method. The method is described in detail below in conjunction with the execution flow of the data query method.

[0023] Figure 2 A flowchart of a data query method provided in an embodiment of the present invention; Figure 3 This is a functional configuration diagram of a data query system provided in an embodiment of the present invention.

[0024] like Figure 2 As shown, the data query method provided in this embodiment of the invention may include: S201: intercepting a first query request generated by a client device when the client performs a pagination data acquisition task, the first query request including a first pagination parameter characterizing the pagination capacity.

[0025] S202: Obtain the first pagination limit parameter corresponding to the target application interface of the target server corresponding to the first query request.

[0026] S203: Use the first pagination restriction parameter to determine the first pagination parameter, and obtain the second pagination parameter.

[0027] S204: Update the first query request using the second pagination parameter to obtain the second query request.

[0028] S205: Send the second query request to the target server to invoke the target application interface to respond to the pagination data retrieval task according to the second pagination parameters.

[0029] This invention aims to address a core challenge in related technologies: when a client device's application service calls the server's application programming interface (API) to query data, its pagination requests often lack awareness of the backend system's capabilities, leading to a series of problems. For example, when a user submits an open request such as "query all virtual machines," the client, in order to fulfill the request as quickly as possible, may generate an API call with an extremely large page size (pageSize). This is highly likely to fail due to hitting the API call limit, causing backend performance bottlenecks, or exceeding the maximum context length that the client can process at one time (such as exceeding the maximum context length that a large language model can process at one time), ultimately resulting in long user wait times or interrupted interaction, severely impacting system availability.

[0030] To address this, in this embodiment of the invention, the final decision-making power for pagination requests is transferred from the client to an independent interception component for centralized adjudication. A standardized client-side pagination execution process ensures stable, efficient, and controllable data acquisition from multiple sources and heterogeneous APIs. The data query method provided by this embodiment does not rely on special snapshot or cursor capabilities provided by the server. By implementing a series of processes on the client side, it ensures the eventual consistency, reproducibility, and security of pagination results in a dynamic data environment.

[0031] like Figure 3 As shown, the steps provided in this embodiment of the invention can be executed based on a layered system architecture, mainly including a policy layer and an execution layer. The data query method provided in this embodiment of the invention can be applied to scenarios where an intelligent agent, after parsing user intent through natural language interaction, needs to call one or more cloud platform APIs (such as querying virtual machines, disks, logs, and other resources) to obtain a large amount of data.

[0032] Specifically, in this embodiment of the invention, the decision-making power for pagination parameters is centralized. Pagination parameters proposed by the client (such as page size and page number) are only suggestions; the final effective pagination parameters are determined by the strategy layer provided by this embodiment based on a preset strategy. Furthermore, this embodiment also provides a client-side pagination process that encapsulates the complete pagination lifecycle, including cyclic retrieval, data consistency assurance, exception handling, and resource control, into an independent, reusable pagination execution sub-process (or "pagination sub-graph"). This sub-process is responsible for the actual interaction with the backend API.

[0033] The data query method provided in this embodiment of the invention can be adapted to general stateless pagination interfaces based on page number / page size or offset / limit, without requiring the server to provide stateful pagination support such as snapshots or cursors. The data query method provided in this embodiment of the invention is also compatible with APIs that use stateful pagination.

[0034] In specific implementation, for S201, on the communication path between the client device and the target server, an intermediate component (such as a proxy module, gateway component, or interceptor implanted on the client side) captures the original query request issued by the client.

[0035] Paginated data retrieval tasks refer to query tasks that require retrieving large amounts of data from the server in batches, such as fetching a list of virtual machines or operation logs. The first pagination parameter represents the pagination capacity, i.e., the number of data items the client expects to retrieve at a time, such as the originally set page size value.

[0036] For example, when a client wants to execute a query task that may involve returning a massive amount of data, it will generate an original request containing pagination parameters (such as pageSize=1000) according to its own logic. This step intercepts the request after it is sent but before it reaches the target server.

[0037] For S202, the target application programming interface (API) of the target server refers to the server-side API that the first query request will call. The first pagination limit parameter refers to the maximum amount of data that the API can handle or is allowed to retrieve in a single transaction, such as the max_limit value configured in the API gateway, the upper limit of the data volume corresponding to the query timeout threshold on the server side, or the pagination limit of the database, etc.

[0038] This step obtains the service capability boundary information of the target API, that is, "what is the maximum page size that the API can accept". This limit parameter can be read from the pre-configured API metadata or obtained through dynamic interaction with the server or gateway.

[0039] For S203, the ruling refers to comparing or performing logical operations between the first paging parameter and the first paging limit parameter to determine whether the paging capacity originally set by the client is reasonable. For example, when the first paging parameter (e.g., pageSize=10000) is greater than the first paging limit parameter (e.g., max_limit=1000), the ruling result is a more reasonable value (e.g., 1000); when the original parameter is already within the limit range, the ruling result can maintain the original value.

[0040] This step determines the page size desired by the client based on the server's actual capacity, and generates a page size that the target server can handle as the second pagination parameter.

[0041] For S204, updating the first query request using the second pagination parameter includes at least replacing the portion of the first pagination parameter that does not conform to the second pagination parameter with the second pagination parameter, or directly replacing the first pagination parameter with the second pagination parameter. Other parts of the first query request (such as the request address, authentication information, other query conditions, etc.) remain unchanged. The resulting second query request has a pagination capacity that is a reasonable value after adjudication.

[0042] For S205, the second query request is sent to the target server. Upon receiving the request, the target server returns the corresponding batch of data according to the single retrieval quantity specified by the second pagination parameter. Since the second pagination parameter is a reasonable value after adjudication, the request will not be rejected by the target server due to excessive page size or cause the target server to become overloaded.

[0043] The data query method provided in this invention intercepts the first query request generated by the client device when performing a paginated data retrieval task. It then determines the first pagination parameter in the first query request based on the first pagination restriction parameter corresponding to the target application interface of the target server, obtaining a second pagination parameter. The second pagination parameter is used to update the first query request to obtain a second query request, which is then sent to the target server to invoke the target application interface to respond to the paginated data retrieval task according to the second pagination parameter. In other words, by combining the capabilities of the application interface, the method limits the amount of target data requested by the client in a single retrieval, solving the problem of excessively large single retrieval requests due to the client's lack of awareness of the server side when calling the application interface for paginated queries, leading to query failures. This improves the success rate of paginated data queries and enhances the user experience.

[0044] The above embodiments describe how to determine the client's first pagination parameters based on the first pagination restriction parameters corresponding to the target application interface of the target server, thereby obtaining the second pagination parameters. In addition, more strategies can be used to comprehensively determine the second pagination parameters.

[0045] In some optional implementations, the data query method provided in this embodiment of the invention may further include: obtaining a second pagination restriction parameter.

[0046] In S203, the first pagination parameter is determined using the first pagination restriction parameter to obtain the second pagination parameter. This can include: determining the first pagination parameter using the first pagination restriction parameter and the second pagination restriction parameter to obtain the second pagination parameter.

[0047] The steps for obtaining the second pagination restriction parameter may include: if a third pagination restriction parameter exists for the client, then the third pagination restriction parameter is used as the second pagination restriction parameter; if a third pagination restriction parameter does not exist but a fourth pagination restriction parameter exists corresponding to the resource type of the target application interface, then the fourth pagination restriction parameter is used as the second pagination restriction parameter; if neither a third nor a fourth pagination restriction parameter exists, then the default pagination restriction parameter is used as the second pagination restriction parameter.

[0048] In practice, the first pagination parameter is determined by combining the first and second pagination restriction parameters, thus obtaining the second pagination parameter. The second pagination restriction parameter can be derived from restriction parameters from multiple dimensions.

[0049] Specifically Figure 3 The strategy layer is the core component responsible for pagination parameter decision-making in this embodiment of the invention. After receiving the data query request generated by the client, but before the request is actually sent to the target application interface, it intervenes and completes a series of checks and decisions.

[0050] like Figure 3 As shown, a multi-dimensional static upper limit configuration table can be maintained at the strategy layer to trim the page size (pageSize) required by the client. This configuration table contains at least the following levels: (1) Global default value: sets a common default page size and maximum page size for all resources.

[0051] (2) Resource-level definition: Set default values ​​and maximum values ​​for different types of API resources (such as virtual machines, cloud disks, and operation logs). For example, log APIs can usually support larger page sizes than virtual machine information APIs.

[0052] (3) SLA / Tenant-level Override: Allows the above configurations to be overridden based on the Service Level Agreement (SLA) or tenant identity. For example, high-priority tenants may be allowed to use larger page sizes or higher query quotas than regular tenants.

[0053] These three configuration layers (global default value, resource-level definition, and SLA / tenant-level coverage) have a priority-based overlapping relationship, and there are no mandatory size constraints (e.g., it is not mandatory for resource-level definitions to be smaller than the global default value). Resource-level definitions are set separately for different types of API resources (such as virtual machines and operation logs). This is usually based on an assessment of the query complexity of different resources and the characteristics of the backend data volume, ultimately resulting in the second pagination limit parameter.

[0054] In this embodiment of the invention, the first pagination parameter is determined by the first pagination restriction parameter and the second pagination restriction parameter to obtain the second pagination parameter. This may include: using the minimum value among the first pagination parameter, the first pagination restriction parameter, and the second pagination restriction parameter as the second pagination parameter.

[0055] In other words, when a client request is received, the strategy layer calculates the actual page size (actual_page_size) as the second pagination parameter. The calculation logic is as follows: actual_page_size=min(first pagination parameter, first pagination limit parameter, second pagination limit parameter); Here, `min` indicates taking the minimum value. The first pagination parameter is the page size requested by the client. The first pagination limit parameter can be the upper limit declared by the API, which can be the API administrator or API provider's pre-declared capability metadata for a specific API, i.e., the maximum number of return items the API itself can support (max_limit). The second pagination limit parameter can be the upper limit configured by the policy, i.e., the maximum limit value that ultimately takes effect in the static configuration table of the policy layer after priority determination at the aforementioned three levels (global default value, resource-level definition, SLA / tenant-level coverage). The system will take the minimum value among the client-requested page size, the API declared upper limit, and the policy configuration upper limit as the final actual page size.

[0056] To enable the pagination strategy to adapt to the diverse APIs on the backend, an API capability declaration list can be maintained at the strategy layer. System administrators or API providers need to pre-declare the pagination-related capability metadata for each integrated API, including: (1) Pagination interface type: specifying whether the API follows the page number / page size (page / pageSize) mode or the offset / limit number (offset / limit) mode.

[0057] (2) Maximum number of entries (max_limit): The maximum number of entries that the API itself can support.

[0058] (3) Sorting capability: Whether it supports user-specified sorting fields (sort_key), and which fields are supported. This is a prerequisite for achieving stable pagination.

[0059] (4) Total return capability: Whether the API response includes a total data field (total).

[0060] (5) Range filtering capability: Does the API support range queries based on a specific field (such as creation time createdAt) (e.g., createdAt<=X)? This capability is key to implementing the water level mechanism (which will be discussed below).

[0061] During the adjudication phase, the strategy layer will solidify the necessary parameters for the upcoming pagination sub-process based on these capability declarations, such as selecting the correct pagination parameter name, determining a key for stable sorting, and the final page size calculated based on the API cap and strategy cap.

[0062] Based on the above embodiments, the data query method provided by the embodiments of the present invention may further include: pre-injecting pagination suggestion information into the client to guide the client to generate a first query request based on the pagination suggestion information.

[0063] The pagination suggestion information includes at least the fifth pagination restriction parameter.

[0064] To reduce the possibility of clients generating unreasonable pagination requests from the source, in this embodiment of the invention, pagination suggestion information can be injected into the client in advance to guide the client to configure reasonable first pagination parameters in the data query request.

[0065] If applied to agent data querying, pre-injecting pagination suggestions into the client can be done by injecting a piece of guiding text about pagination into the input model's prompts at the initial stage of the interaction with the agent. This guiding text clearly defines the usage specifications of pagination parameters, such as: 1. Recommended page size: clearly informing the agent of a reasonable default page size, such as 10 or 20.

[0066] 2. Absolute upper limit of page size: Set the maximum value that an agent should not exceed under any circumstances, such as 100. This upper limit (denoted as the maximum page size {max_page_size}) can be a static configuration. Better yet, it can be dynamically generated and injected by the strategy layer based on the current context (such as tenant level, system load) to achieve personalized and fine-grained guidance.

[0067] 3. Pagination initiation and continuity: Guide the agent to start requesting from the first page and retrieve data page by page in sequence to meet the user's complete data needs.

[0068] 4. Risk Warning: Clearly warn that setting the page size too large for the smart agent may cause performance issues or request failures.

[0069] This mechanism enables agents to construct more reasonable requests based on given constraints when planning their API call sequence, significantly reducing the frequency of request pruning and correction by subsequent policy layers and improving the efficiency of the entire interaction process. Even if the agent violates these guidelines, subsequent policy and execution layers will still enforce the system's hard constraints.

[0070] Based on the above embodiments, the present invention further describes the steps of interacting with the target server to continuously obtain pagination results.

[0071] In this embodiment of the invention, sending the second query request to the target server in step S205 to call the target application interface to respond to the pagination data acquisition task according to the second pagination parameters may include: repeatedly sending the second query request to the target server until the termination judgment condition in the second query request is met.

[0072] Specifically, starting from the second time the second query request is sent, before each second query request is sent, the target data address parameter in the current second query request is updated based on the returned data of the previous second query request.

[0073] In specific implementation, the termination judgment conditions may include at least one of the following: the number of data entries returned by the current round response is less than the number of single retrievals specified in the second pagination parameter; the current round response contains an identifier field indicating that there is no subsequent data; the cumulative number of data entries retrieved has reached the total amount of data to be retrieved; or the next page positioning identifier returned in the current round response is empty.

[0074] This step can be based on Figure 3 The execution layer is shown in the diagram. The execution layer is the core operational unit of this invention; it exists as an independent, reentrant sub-process responsible for executing the paging tasks determined by the strategy layer. This sub-process ensures the stability and consistency of data acquisition by maintaining a series of internal states and implementing fine-grained control flow.

[0075] Specifically, the pagination subprocess can maintain a context state object during execution, which contains the following key information: (1) Pagination context (page_ctx): records the current pagination progress and mode, such as mode (mode, page or offset), page size (pageSize), current page number or offset (page_index / offset), total number of fetched items (fetched), sort key (sort_key), and watermark for enhancing consistency.

[0076] (2) Seen IDs set: A set used to store unique identifiers of returned data entries. Its main function is to remove duplicates when there is jitter in cross-page data, ensuring the uniqueness of the final result.

[0077] (3) Metrics: Performance data used to monitor and audit the pagination process, such as the number of requested pages, the number of API calls, the average latency, and the number of retries.

[0078] (4) Guards: Global guard restrictions inherited from the strategy layer.

[0079] (5) Overlap number (overlap_k): A small integer (e.g., 1-3) defines the number of additional entries that may overlap with the previous page's data that need to be retrieved when requesting the next page of data. This is used to proactively address offset issues caused by data insertion / deletion.

[0080] In this embodiment of the invention, before sending the second query request, the method may further include: determining whether the status of the pagination data acquisition task triggers a circuit breaker condition; if yes, then ending the pagination data acquisition task; if no, then sending the second query request.

[0081] In practice, to prevent pagination operations from consuming excessive resources or causing users to wait for extended periods, a set of global guardrails can be set at the strategy layer as circuit breakers for the entire pagination task. When the pagination data retrieval task is executed at the execution layer, the global guardrails at the strategy layer can be used to monitor and trigger the circuit breaker.

[0082] The circuit breaker conditions may include at least one of the following: the continuous execution time of the pagination data retrieval task reaches the task deadline (deadline_ms, i.e., the total maximum execution time of the pagination task); the amount of data retrieved by the pagination data retrieval task reaches the data volume limit (max_pages, the maximum number of pages, i.e., the maximum number of pages allowed to be retrieved); the system resources consumed by the pagination data retrieval task reach the preset system resource consumption limit (max_cost, i.e., the estimated or actual resource consumption limit, which can be expressed as the number of API calls); and the access frequency corresponding to the pagination data retrieval task reaches the access frequency limit corresponding to the target application interface (QPS, queries per second, i.e., the access rate limit for the backend API).

[0083] During the pagination process, after each page is retrieved, it is checked whether these guardrails have been touched. Once any condition is met, the pagination process will immediately terminate and return the retrieved data along with a corresponding explanation of the termination.

[0084] Current paginated data retrieval processes suffer from a technical problem: when using traditional offset / page-based pagination queries, the consistency of the result set is difficult to guarantee. Essentially, this type of pagination counts data from the Nth item onwards. If the underlying dataset changes between multiple pagination requests (e.g., new data is inserted, or old data is deleted), the position of the Nth item will shift, causing the client to see duplicate data on the next page or miss some data.

[0085] In this embodiment of the invention, the second query request may further include sorting parameters, which are determined according to the sorting type corresponding to the target application interface.

[0086] The above-mentioned loop execution sends the second query request to the target server, which may include: loop execution sends the second query request to the target server to call the target application interface to sort the target data according to the sorting parameter, and then sequentially obtains the target data corresponding to the second pagination parameter and sends it to the client device.

[0087] Furthermore, the iterative execution of sending the second query request to the target server to call the target application interface to sort the target data according to the sorting parameters, and then sequentially retrieving the target data corresponding to the second pagination parameters and sending it to the client device, may include: after sending the first second query request corresponding to the pagination data retrieval task and receiving the first page of target data sent by the target server, obtaining the maximum and minimum values ​​of the corresponding type of metadata of the first page of target data according to the sorting parameters, as a water level marker; determining the filtering conditions based on the water level marker as the boundary according to the sorting type and the water level marker; starting from the second second query request, before each second query request is sent, updating the target data address parameters in the current second query request according to the returned data of the previous second query request, and injecting the filtering conditions into the second query request.

[0088] This invention introduces a watermark mechanism into the pagination sub-process to solve the problem of data position drift between multiple pagination requests on the client side. Specifically, in the pagination sub-process, the target application interface is called to respond to the pagination data retrieval task according to the second pagination parameters, which may include: 1. Initialization: Based on the decision result of the strategy layer, the pagination context is initialized, including fixing the sort key, page size, etc. 2. First request and snapshot establishment (watermark mechanism): When the first API request is executed (e.g., requesting page 1), a stable sort key must be specified (e.g., createdAt DESC, i.e., sorted in descending order of creation time). The main purpose of this request is not just to retrieve the first page, but to use it to determine a snapshot boundary. Specifically, the boundary value of the sort field is extracted from the data of the first page and locked as the watermark of this pagination session.

[0089] In this embodiment of the invention, the water level serves to freeze the data range, ensuring that the entire pagination session operates on a fixed subset of data. In other words, the water level locks the data visible during this pagination operation within a time window, thus simulating a pseudo-snapshot. Subsequent pagination parameters (page or offset) are calculated only within this subset, unaffected by newly inserted or deleted data. This solves the common problem of result drift during page turning in offset pagination.

[0090] In some optional embodiments of the present invention, the sorting type is sorted by creation time, the sorting parameter is sorted in ascending order by creation time, the water level mark is the earliest creation time in the target data on the first page, and the filtering condition is to filter target data whose creation time is earlier than the water level mark.

[0091] In other words, the target application interface of the target server is required to sort the target data in descending order (createdAt DESC, i.e., sorted by creation time in descending order), and take the earliest createdAt (creation time) on the first page as the watermark. Subsequent requests will include the condition createdAt <= watermark. This ensures that even if new data is inserted (whose creation time createdAt must be later), it will not be included in the current pagination results.

[0092] In some alternative embodiments of the present invention, the sorting type is sorted by creation time, the sorting parameter is sorted in descending order by creation time, the water level mark is the latest creation time of the target data on the first page, and the filtering condition is to filter target data whose creation time is later than the water level mark.

[0093] In other words, the target application interface of the target server can be instructed to sort the target data in ascending order (createdAt ASC, i.e., sorted in ascending order of creation time), and the latest createdAt (creation time) on the first page can be used as the watermark. Subsequent requests will include the condition createdAt>=watermark. This ensures that even if there is updated data later (createdAt), it will only be listed at the end of the target data and will not disrupt the order of the current pagination session.

[0094] It's important to note that the watermark is determined only once during the first request and remains unchanged throughout the entire pagination session. Subsequent pages (page 2, page 3, etc.) still use the same watermark as the snapshot boundary, simply using the page number or offset to continue scrolling within this frozen data range. Continuously updating the watermark during pagination is equivalent to repeatedly moving the snapshot boundary, which can introduce duplicate or missed reads.

[0095] Therefore, when new data is added to the target data, because the water level mechanism adds boundary filtering conditions (such as creation time being earlier than the water level value), the newly inserted data will be directly blocked outside this time window after the first query. The new data will not participate in this pagination session and will not be sent to the client.

[0096] In summary, in this embodiment of the invention, a fixed boundary is established on the first page using a watermark mechanism, locking the dataset for this pagination session. This ensures that pagination calculations are always performed on this fixed subset, preventing duplication, omissions, or disordered order due to real-time changes in the database.

[0097] Furthermore, the call to the target application interface to respond to the pagination data retrieval task according to the second pagination parameter may also include: 3. Loop retrieval: Enter a loop, the termination condition of which is: the required amount of data has been retrieved (such as the Top-N specified by the user, the first N), the end of the data has been reached, or any global guardrail has been triggered.

[0098] 4. Perform the following operations inside the loop: (1) Guardrail check: At the beginning of each iteration, check whether the guardrails such as the total longest execution time (deadline_ms) and the maximum number of pages (max_pages) of the pagination task have been reached. If they are reached, the loop is interrupted. (2) Request construction: Construct the second query request based on the current pagination context (page or offset). The request parameters must include: fixed pagination parameters: such as page=2, pageSize=100; fixed sort key: completely consistent with the first request, such as sort_key=createdAt DESC; fixed watermark filtering conditions: starting from the second request, each request must carry the watermark filtering conditions determined in step 2 (such as createdAt<=watermark_value). This ensures that all subsequent queries operate on an approximately fixed data snapshot, thereby solving the offset drift problem. (3) API call and backoff: Execute the second query request (API call). If a retryable error (such as network timeout, server rate limiting HTTP429) is encountered, a retry with an exponential backoff strategy is executed. For rate limiting errors, the pageSize of subsequent requests can be dynamically reduced selectively.

[0099] The water level mechanism described in the above embodiments cannot solve the problem of data position drift caused by data deletion during page turning. In this regard, the data query method provided by the embodiments of the present invention may further include: from the second time the second query request is sent, before each second query request is sent, a cross-page overlap parameter is injected into the second query request, so that when the target application interface performs the pagination data acquisition task, it backtracks the target data corresponding to the cross-page overlap parameter and adds it to the returned data; from the time the second page target data returned by the target server is received, after each time the current page target data is received, the last first preset number of target data in the previous page is taken out and compared with the first preset number of target data in the current page target data to delete the duplicate target data.

[0100] In this embodiment of the invention, cross-page overlap processing is performed for each API data request. For example, when requesting page n, the target application interface is asked to retrieve the last k duplicate data entries (overlap_k) of the previous page (which can be 1 to 3 entries). The first overlap_k entries of the current page are then compared with the last overlap_k entries of the previous page, and duplicate data is removed, retaining only the data that has not been retrieved. This corrects page misalignment caused by data deletion or offset, avoiding skipping or duplicate retrieval issues. For example, if data is deleted between two requests, causing subsequent data to overshoot, overlap alignment can be used to reposition the data, ensuring continuous pagination.

[0101] Furthermore, the data query method provided in this embodiment of the invention may also include: storing the unique identifier of the target data returned by the target server; starting from receiving the second page of target data returned by the target server, each time the current page of target data is received, comparing the unique identifier of the current page of target data with the stored unique identifier to delete the target data that has been obtained repeatedly.

[0102] In other words, a uniqueness check can be performed on each retrieved paginated data. Using its globally unique identifier (or a parallel deduplication key, such as a unique identifier and version number), the data is compared against the set of known identifiers. If the unique identifier has already appeared, the data is discarded; otherwise, it is added to the final result set and the set is updated. This step can be combined with the cross-page overlap handling described above. First, cross-page overlap handling is performed, followed by global unique identifier checking and deduplication. This addresses cross-page duplication caused by identical sort keys or data jitter, ensuring that the final retrieved data is complete and free of duplicates.

[0103] In this embodiment of the invention, through cross-page overlap alignment and deduplication based on globally unique identifiers, the time complexity is O(k+n), where k is the number of overlapping entries (usually 1~3, constant overhead) and n is the number of valid entries on the current page.

[0104] Combining the watermark mechanism described in the above embodiments, for example, the boundary value of the sorting field can be selected as the watermark on the first page. All subsequent pagination requests are then subject to the condition `createdAt <= watermark (descending order)` or `createdAt >= watermark (ascending order)` to freeze the data range of the entire session and prevent result drift caused by new data. Then, cross-page overlap and deduplication based on globally unique identifiers are performed. To address page-to-page connectivity issues, multiple overlapping records are retrieved and compared, combined with global deduplication, to correct inconsistencies between pages caused by data deletion, parallel sorting, or slight fluctuations. In other words, the watermark mechanism achieves session-level protection, locking the data range of the entire pagination process (preventing new data from jumping the queue). Cross-page overlap and deduplication based on globally unique identifiers provide page-level protection, used to correct gaps between pages (preventing deletion / misalignment / parallel duplication).

[0105] The above 4, the following operations performed inside the loop body may also include: (4) Status update: based on the amount of data returned, update the pagination pointer (page++ or offset+=actual_item_count), and accumulate the number of data items fetched, the number of pages fetched, and other metrics.

[0106] (5) Termination condition judgment: Check whether the amount of data returned this time is less than the requested pageSize. If so, it is very likely that the end of the data has been reached and the loop can be terminated in advance. Top-N early stop: When the upper layer only needs Top-N, the execution layer stops getting data when the cumulative return reaches N.

[0107] Finally, the target application interface responds to the pagination data retrieval task according to the second pagination parameter, and may also include: 5. Returning results: After the loop ends, the retrieved data, a summary of the pagination process (such as total time and total number of pages), and suggestions for the next operation (such as whether loading can continue) are returned to the upper-level caller.

[0108] In summary, the data query method provided by this invention can achieve a high level of data consistency through a combination strategy on the client side without server-side snapshots: (1) Forced stable sorting: The client forces the use of one or more sorting keys that can uniquely determine the data order (such as createdAt DESC, id DESC) in all pagination requests. This avoids data drift caused by the uncertainty of the database's default sorting.

[0109] (2) Watermark mechanism: By locking a query boundary (such as a timestamp) after the first request, all subsequent requests are made within this boundary, which greatly reduces the interference of newly inserted data on the pagination result set and approximates a snapshot of a point in time.

[0110] (3) Overlapping and deduplication based on globally unique identifiers: This mechanism is a direct means of combating pagination misalignment ("jitter") caused by data insertion / deletion. By retrieving several data entries and comparing them with the previous page, the pagination pointer can be accurately located and corrected to ensure that data is neither duplicated nor omitted.

[0111] (4) Stable Sorting and Tie-breaker: To avoid duplicate values ​​of the primary sort key (e.g., sorted by creation time) (i.e., tie-breakers, where two columns have the same creation time), which can lead to unstable ordering across multiple queries, this invention enforces a composite stable sorting. The core of this method is the introduction of a "tie-breaker key": a globally unique secondary sort key used to break the deadlock when primary keys are the same, ensuring absolute sorting stability. In practice, this key is usually a unique identifier for the data entry, such as id. Therefore, the complete sorting expression is: primary sort key DESC, tie-breaker key (id) DESC, which ensures that the order of results under the same filtering conditions is completely reproducible during replay.

[0112] For APIs using offset / limit mode, database performance can drop drastically when the offset value is very large (i.e., "deep page lookup").

[0113] In this regard, the data query method provided by the embodiments of the present invention may further include: if it is detected that the target server fails to perform the pagination data acquisition task, then the second query request is resent after reducing the second pagination parameter in the second query request; after reducing the second pagination parameter, if the target server successfully executes the second query request a second preset number of times, then the second pagination parameter in the second query request is enlarged.

[0114] Specifically, for extremely large datasets, a large query task can be broken down into multiple subtasks based on a partitionable field (such as time or geographic region). These subtasks can perform pagination within their respective partitions and improve overall efficiency through parallel execution.

[0115] Additionally, when the target application interface encounters an HTTP 429 error (i.e., "too many requests, rate throttling by the server"), or when the proportion of request timeouts continuously exceeds a certain threshold (denoted as θ), adaptive pagination is executed. This can halve the current page size (pageSize), but it will not be less than the preset minimum value (page_min) to reduce the pressure on a single request.

[0116] Furthermore, the waiting time before each retry can be increased in multiples of 2, and a certain amount of random jitter can be introduced to prevent multiple requests from being re-initiated at the same time.

[0117] After the system returns to normal, for example, if no errors occur within the second preset number of times (which can be 5) of the statistics window, and 95% of the response latency of the interface is lower than the target threshold, the second pagination parameter can be gradually increased, such as gradually increasing the page size. Each time, a fixed step size (Δ) can be added until the maximum allowed value (page_max) is reached.

[0118] In addition, a global token bucket mechanism can be used to control the request rate (QPS), thereby avoiding rate limiting or timeouts triggered again due to instantaneous traffic spikes.

[0119] In conventional offset pagination, if a user jumps directly to page 1000, the system must sequentially flip through 999 pages to locate it, which is highly inefficient. Specifically, imagine a database as a book without a table of contents, where each page is dynamically changing. When a client requests page 1000 (i.e., skipping the first 9990 records and retrieving 10 records, offset=9990), the database cannot jump directly to that position. It must meticulously count from the first record up to the 9990th, discarding all previous records before retrieving the remaining 10. As a result, the further down the page (the deeper the page number), the more counting and discarding work the database needs to perform, leading to a sharp performance drop.

[0120] In this regard, the data query method provided by the embodiments of the present invention may further include: caching the query conditions in the second query request of the executed pagination data acquisition task as index information. In S204, updating the first query request using the second pagination parameters to obtain the second query request may include: if the pagination data acquisition task and the executed pagination data acquisition task satisfy the query condition similarity condition, then updating the first query request according to the second pagination parameters and the index information to obtain the second query request.

[0121] The process of updating the first query request based on the second pagination parameters and index information to obtain the second query request may include: determining the target page number of the target data to be obtained by the pagination data retrieval task; obtaining the target index information corresponding to the target page number from the cached index; adding the target index information to the first query request; and replacing the first pagination parameter in the first query request with the second pagination parameter to obtain the second query request.

[0122] In other words, the interception component uses the query conditions in the second query request of the pagination data retrieval task, which has already been executed, as index information. This index information can be added when sending the pagination query request to the target server, allowing the target application interface on the target server to quickly locate the target page and avoid wasting a lot of time on page navigation. This index information is the anchor point, equivalent to a bookmark in database pagination. During the query process, if page 950 has been queried before, the system places a bookmark (anchor point) there. When the user suddenly requests to view page 1000, the system will not start counting from page 1, but will directly find the bookmark for page 950 and count 50 pages forward from page 950. This greatly saves the time spent counting the previous 949 pages.

[0123] Specifically, during the query process, information about certain anchor pages can be recorded for specific combinations of conditions (filters, sorting methods, page sizes). The anchor can be a key field of the first piece of data on that page (such as (createdAt, id)), or it can be the offset value corresponding to that page. When the user jumps to a deeper page (for example, page p), the system does not linearly flip from page 1, but instead first looks for a cached anchor page q that is closest to p and where q < p. Then it continues to catch up from page q until it reaches page p. This can significantly reduce the cost of repeated scans.

[0124] In some optional implementation manners of the embodiments of the present invention, the index information may include key fields (such as createdAt, id), which is equivalent to recording on the bookmark that "the first piece of data on page 950 was created on October 1, 2025, and the ID is 12345". Then, during the next query, it can directly tell the database "help me find data with a time earlier than October 1, 2025 and an ID less than 12345". Technically, this is equivalent to instantly converting an inefficient offset query into an efficient cursor query.

[0125] In some optional implementation manners of the embodiments of the present invention, the index information may include an offset (offset value), for example, directly record that the offset corresponding to page 950 is 9490.

[0126] Thus, when the user requests page p (such as page 1000), the interception component does not directly send a query request, but first looks in the cache for an anchor q that is closest to page 1000 and in front of it (such as the system finds the anchor for page 950 stored previously). The interception component发起请求从 page 950 的锚点位置开始发起请求,连续向后请求(或计算偏移量),数出剩下的 50 页,直到凑齐第 1000 页的数据返回给用户。由此,可以显著降低数据库重复扫描的数据量。

[0127] In the embodiments of the present invention, to enable the system to correctly establish and use index information, it is only allowed to be used when the current query request and the cached index information meet the query condition similarity condition. For example, when the search keywords are the same (both searching for the North China region, running), the sorting method is the same (both sorted in reverse order by creation time), and the page size per page is exactly the same, the index information is allowed to be used.

[0128] At this time, in the interception component, the query condition similarity condition is cached together with the index information, or can be used as an index for the above index information, so as to record the subsequent generated index information.

[0129] It should be noted that there seems to be an unclear part in the translation of where "发起请求从 page 950 的锚点位置开始发起请求" might need further clarification in the original Chinese for a more accurate translation. But following the requirements, this is the best attempt based on the provided text.Because the underlying data on the target server is dynamically added and deleted—for example, the first record on page 950 yesterday might be on page 940 today because data was deleted earlier. If the old index data is used continuously, the result will be misaligned (data drift).

[0130] Therefore, in some optional implementations, the query similarity condition can be having the same water level marker; the water level marker is the maximum or minimum value in the metadata corresponding to the sorting parameters and the first page of target data returned by the target server after sorting according to the sorting parameters in the second query request. That is to say, by binding the index information to the water level (i.e., the time snapshot boundary at the time of query) described in the above embodiments, only queries with the same water level can share the index information.

[0131] The data query method provided in this embodiment of the invention may further include: deleting the index information after the cached index information has reached a preset time. That is, by setting the time-to-live (TTL) of the index information, the index information is only valid for a short period of time (such as a few minutes), and is automatically destroyed and retrieved again after the timeout, thus ensuring data accuracy.

[0132] To improve the user experience of paginated data query, the data query method provided in this embodiment of the invention may further include: sending explanatory information to the user when the requirements of the first query request cannot be met.

[0133] Specifically, when the strategy layer prunes the agent's request, or when the pagination process is terminated early due to a guardrail, an explanatory text can be generated and appended to the returned result.

[0134] For example, when the second pagination parameter is less than the first pagination parameter, meaning the page size is truncated, the explanatory text could be: To ensure system performance, the number of items returned per page for this request has been adjusted to actual_page_size items. Currently, fetched items are displayed, totaling pages. You can choose to continue fetching the next page, or narrow your query by specifying items, regions, time ranges, etc.

[0135] If the returned data is not the entire target data, the explanatory text can be: You requested a large amount of data. The system has retrieved the first N results for you as an overview. You can continue to retrieve the remaining data, or view it by region / time period as needed.

[0136] By attaching explanatory text to the returned results, users can understand the reason in a timely manner when the query results do not meet their expectations, which improves the user experience and can also guide the agent or user to conduct more accurate and efficient subsequent queries.

[0137] The data query method provided in this embodiment of the invention may further include: imposing boundary constraints on the client, wherein the types of boundary constraints include pagination request interception, pagination decision-making power revocation, and pre-guidance.

[0138] Taking the client as an intelligent agent as an example, in order to ensure the mandatory execution of the governance strategy, the system design imposes strict restrictions on the capabilities of the intelligent agent: (1) Tool interface (Schema) design: The API tool definition exposed to the intelligent agent may not contain writable parameters such as pageSize, or even if it is exposed, the execution layer must force the decision to be made through the strategy layer, and the input of the intelligent agent is only used as a reference.

[0139] (2) Separation of decision-making power: The core decision-making power of the pre-trained model is limited to "whether to continue to get the next page" and "how to narrow the query range by adding filtering conditions", and cannot directly control the underlying parameters of pagination.

[0140] (3) Pre-guidance: As described in the above embodiments, a behavioral framework is set in advance through prompt words to form a constraint system that combines soft and hard approaches.

[0141] To ensure the healthy operation of paginated queries, guarantee data consistency, and provide complete result traceability, the data query method provided in this embodiment of the invention may further include: monitoring the status parameters of the target data returned by the target application interface of the target server through a telemetry interface. The status parameters may include at least one of traffic and efficiency indicators, latency and performance indicators, error and resilience indicators, and policy and guardrail intervention indicators.

[0142] The traffic and efficiency metrics can include at least one of the following: number of pages scanned, number of returned data items, and number of API calls. The number of pages scanned (pages_scanned) refers to the total number of pages successfully retrieved in a single pagination session. The number of returned data items (items_returned) refers to the total number of valid data items finally returned to the client after all consistency processing (deduplication). The number of API calls (api_calls) refers to the total number of requests initiated to the target application interface, including all retries; the ratio of this metric to the number of pages scanned reflects the efficiency of the target application interface calls.

[0143] Latency and performance metrics can include at least one of the following: average page latency, P95 page latency, and deep pagination hits. Average page latency (avg_page_latency) refers to the average response time (from request to response) for a single pagination request to the target application interface. P95 page latency (p95_latency) refers to the response time of 95% of pagination requests being below this value, and is a key indicator for measuring user experience and system stability. Deep pagination hits (deep_offset_hits) refer to the number of times the offset or page number of a pagination request exceeds a preset threshold, used to identify deep pagination scenarios that put high pressure on the backend database performance.

[0144] Error and resilience metrics can include at least one of the following: error rate and retry count. Error rate (error_rate) is the percentage of failed calls (non-2xx status codes) to the total number of calls to the target application interface. Retry count refers to the total number of retries triggered by recoverable errors (such as network jitter, 429 rate limiting). A surge in this metric typically indicates instability of the target application interface service or traffic exceeding its capacity.

[0145] Policy and guardrail intervention metrics can include at least one of the following: clamp events, fallback rate, and guardrail trigger count. Clamp events (clamp_events) are the number of times the policy layer performs clamp operations because the page size requested by the agent exceeds the limit. This metric can be used to evaluate the effectiveness of upstream guidance mechanisms for clients. Fallback rate (fallback_rate) refers to the frequency at which the system degrades to the basic pagination mode due to limitations in the target application interface (such as lack of support for stable sorting). Guardrail trigger count (guardrail_triggered_count) refers to the number of times pagination terminates prematurely due to reaching global guardrails such as the maximum page size or longest execution time.

[0146] By setting dynamic thresholds and alarm rules for these metrics (for example, triggering an alarm when the P95 page latency exceeds 2 seconds for 5 consecutive minutes, or when the error rate suddenly increases by 30%), operations and maintenance personnel can promptly perceive the system's health status and quickly respond to potential problems.

[0147] To facilitate the review and auditing of pagination results, especially for tracking and diagnosing occasional data inconsistencies, and particularly when specific events that may compromise data consistency are detected, corresponding logs can be generated.

[0148] The data query method provided in this embodiment of the invention may further include: generating a paginated session event log based on a second query request, wherein the information in the paginated session event log may include: session identifier, query context, and event details.

[0149] The session identifier can include a session ID and a request ID. The session ID (session_id) is used to uniquely identify a complete end-to-end pagination session. The request ID (request_id) is used to identify a single target application interface call request within the session, for precise location.

[0150] The query context can include: filter hash, sort key, and watermark. The filter hash (filters_hash) is the value obtained by hashing the filtering conditions (such as the WHERE clause) of the target application interface query, used to aggregate all queries targeting the same subset of data. The sort key (sort_keys) is the complete sorting expression used in this session (e.g., createdAt DESC, id DESC). The watermark is the watermark value determined during the initial request of this session.

[0151] The event details (event_type) can include: event type and event details. The event type (event_type) describes the category of the inconsistent event, such as DUPLICATE_DETECTED, OVERLAP_MISMATCH, MISSING_SEQUENCE, etc.

[0152] The event details (event_details) are a JSON object containing specific context, which may include parameters such as: overlap count, list of duplicate IDs, missing sequence range, overlap difference details, and timestamp. The overlap count (overlap_k) is the configured number of cross-page overlaps. The list of duplicate IDs (duplicated_ids) consists of duplicate data IDs identified and discarded during the idempotent deduplication process. The missing sequence range (missing_sequence_ranges) is the range of data gaps detected when the sort key is continuous (such as sequence numbers). The overlap difference details (overlap_diff) describe the specific differences between the end of the previous page and the beginning of the current page in the overlapping portion. The timestamp is the precise time the event occurred.

[0153] The monitoring and logging mechanism provided by this invention can provide protection for the following application scenarios: Online fault location: When a user reports data corruption or missing data, all relevant audit logs can be quickly retrieved through the session ID (session_id), accurately reconstructing each step of decision-making and data processing in the pagination process, thereby quickly locating the root cause of the problem as either client logic, API behavior, or the underlying data itself.

[0154] Offline consistency replay: Engineers can use information such as filter hashes, sort keys, and watermarks in the audit logs to accurately replay any historical pagination session in the test environment to stably reproduce and debug complex boundary conditions and concurrency issues.

[0155] Performance bottleneck analysis: By analyzing the long-term trends of metrics such as deep offset hits (deep_offset_hits) and P95 page latency (p95_latency), we can identify which API endpoints or query patterns have performance bottlenecks, thus providing data-driven decision-making basis for database index optimization, API transformation, or capacity planning.

[0156] Security and Compliance Audit: The complete paginated session event log provides an immutable record of data access, detailing when, through what queries, and what range of data was retrieved, meeting the audit requirements for data access behavior in highly compliant industries such as finance and healthcare.

[0157] In summary, this invention provides a complete closed-loop governance solution encompassing pre-event guidance, in-event adjudication, and robust post-event execution. This solution enables effective control over large-scale API pagination retrieval for intelligent agents on the client side, without relying on special server-side support. It not only resolves system performance and stability issues that may arise from unreasonable pagination requests but also significantly improves the accuracy and reproducibility of pagination results in dynamic data environments through a refined consistency guarantee mechanism. This makes it particularly suitable for private cloud and cloud management platform scenarios with stringent requirements for security, stability, and resource control.

[0158] Based on the above embodiments, this embodiment of the invention takes the client as an intelligent agent as an example to provide an execution example of a paginated data acquisition task in a practical application scenario.

[0159] Use Case: A user issues a command to an agent deployed on a private cloud management platform: query all running virtual machines in a specific region, sorted in descending order of creation time. The agent needs to call the platform's virtual machine application programming interface ( / vms API) to complete this task. This API is a standard, stateless paging interface based on offset / limit patterns.

[0160] The data query method provided in this embodiment of the invention follows the logic of pre-event guidance, in-event adjudication, and post-event robust execution, involving three major stages: system prompt injection, strategy-level adjudication, and execution-level operation.

[0161] Phase 1: Pre-initiated guidance - Injecting constraints via system prompts.

[0162] At the beginning of the session between the user and the agent, the interception component provided in this embodiment of the invention injects guiding text such as pagination suggestions into the model. For example, based on the current context (e.g., the user is a VIP tenant) and the global configuration, the following text is generated: "When pagination is required to query data, the suggested page size (limit) is 50. Under no circumstances should the page size exceed 200. Please start the request from the first page (offset=0) and ensure that you use createdAt DESC for sorting."

[0163] The above text is integrated into a complete system prompt and provided to the model.

[0164] After receiving a user command, the agent generates a preliminary API call plan based on its understanding of the command and the injected system hints. Thanks to guidance, it is more likely to generate a reasonable request, such as: GET / vms?filter=region:north,status:running&sort=createdAt:DESC&offset=0&limit=50. Even if the agent generates an unreasonable request (such as limit=500), subsequent processes will force correction.

[0165] Phase Two: In-Process Adjudication - Intelligent adjudication and interpretation at the strategy level.

[0166] Before the API request (first query request) generated by the agent is actually sent, it must be intercepted and adjudicated by the policy layer.

[0167] 1. Request interception: The strategy layer acts as middleware on the request path and captures the first query request mentioned above.

[0168] 2. Static Limit Determination: The policy layer loads a static configuration for virtual machine resources, which defines the maximum number of returned entries (max_limit) as 100. Based on the policy layer, the limit=50 in the agent's first query request is compared with the configured max_limit=100, and the minimum value is taken. The final determined actual page size actual_limit is 50.

[0169] Since actual_limit equals the agent's suggested value, no pruning occurred this time, and no explanatory information needs to be generated.

[0170] 3. API Capability Negotiation: The strategy layer queries the pre-registered API capability list to confirm that the virtual machine application interface supports offset / limit mode, supports sorting by creation time (createdAt), and that its response body does not contain a total data field (total). It confirms that createdAt is a valid timestamp field, suitable for subsequent watermark mechanisms.

[0171] 4. Cost and latency guardrail settings: The strategy layer adds a set of guardrail parameters to this pagination task based on the global configuration: deadline_ms: 60000 (total time not exceeding 60 seconds), max_pages: 200 (maximum 200 pages to be retrieved), max_api_calls: 300 (maximum 300 API calls).

[0172] 5. Task Issuance: The strategy layer packages the pagination parameters (actual_limit=50), the fixed sorting key (createdAt DESC, id DESC, with a unique ID automatically appended as the parallel resolution key), the guardrail configuration, and the API capability metadata into a pagination task context and passes it to the execution layer.

[0173] Phase 3: Post-event robust execution - client-side pagination sub-process at the execution layer.

[0174] The execution layer is the unit that actually interacts with the backend API. It uses a strict loop control flow to ensure the stability and consistency of data acquisition.

[0175] 1. Initialization: The execution layer creates a pagination session and initializes the internal state: page_ctx = {mode: "offset", page_size: 50, offset: 0, fetched: 0, sort_key: "createdAt DESC, idDESC", watermark: null}, seen_ids = new Set(), metrics = {......}.

[0176] 2. Initial Request and Water Level Establishment: The execution layer sends a second query request to initiate the first API call: GET / vms?filter=......&sort=createdAt:DESC,id:DESC&offset=0&limit=50.

[0177] Suppose the target application interface successfully returned 50 virtual machine data entries. The execution layer iterates through these 50 data entries and finds the minimum value of the `createdAt` field (because it is sorted in descending order), for example, `2025-08-01T12:00:00Z`.

[0178] Set this minimum value as the watermark for this pagination session: page_ctx.watermark = '2025-08-01T12:00:00Z'.

[0179] Simultaneously, the unique IDs of these 50 data entries are stored in the seen IDs (seen_ids) set, and the data is stored in the result set. Update status: page_ctx.offset = 50, page_ctx.fetched = 50.

[0180] 3. Looping to retrieve (taking page 2 as an example): (1) Guardrail check: Check whether the total time spent, number of pages retrieved, etc. exceed the limit. Assume that the limit is not exceeded.

[0181] (2) Request Construction: Construct the next API request. The key is that the request must include the filter condition derived from the watermark. The request becomes: GET / vms?filter=region:north,status:running,createdAt:<='2025-08-01T12:00:00Z'&sort=createdAt:DESC,id:DESC&offset=50&limit=50. This createdAt condition ensures that even if a new virtual machine is created (whose createdAt will be greater than the watermark value), it will not interfere with the result set of this pagination.

[0182] (3) API Call and Backoff: Execute the API call. If the API returns an HTTP 429 (rate limiting) response, the execution layer will pause and wait for an exponentially increasing time (e.g., 200ms, 400ms, 800ms...) before retrying. Simultaneously, it may temporarily reduce the limit for the next request to 25 to alleviate server load.

[0183] (4) Data consistency processing: Assume that the API successfully returned 50 new data entries.

[0184] (5) Handling cross-page overlap: By setting the request limit to 53 and recording the cross-page overlap number (overlap_k) as 3 when requesting, after receiving the data of this page, the first 3 data of this page are compared with the last 3 data of the previous page. If it is found that the data has been shifted forward due to data deletion, the starting position of the valid data is adjusted to avoid data omission.

[0185] Deduplication: Iterate through the newly acquired 50 data entries. For each data entry, check if its unique identifier exists in the seen identifiers (seen_ids) set. If it exists, this data entry is a duplicate (possibly due to typified sort keys or data jitter), and it is discarded. If it does not exist, add its unique identifier to the seen identifiers (seen_ids) set, and append the data to the final result set.

[0186] (6) Status update: Assume that 48 valid data entries are obtained after deduplication. Update status: pagination context offset (page_ctx.offset) += 50 (the increase of offset is still calculated according to the requested limit, not the actual number returned), number of fetched (page_ctx.fetched) += 48.

[0187] (7) Termination condition judgment: Check whether the amount of data returned this time (50 records) is less than the requested limit (50 records). If it is not less than the limit, the loop continues.

[0188] 4. Loop Termination and Return: This loop continues until an API call returns fewer than 50 data entries, indicating that the end of the data has been reached.

[0189] Alternatively, the loop may terminate prematurely when the number of fetched pages (page_ctx.fetched) reaches the user-specified Top-N, or when the number of fetched pages (metrics.pages) reaches the maximum number of pages (max_pages) bar.

[0190] After the loop ends, the execution layer returns all deduplicated and consistent data result sets, along with an execution summary (such as "358 data entries retrieved, time taken 5.3 seconds, total 8 pages"), to the upper-level caller (agent).

[0191] If the query was terminated due to a barrier, the summary will include an explanation, such as: "Due to the large amount of data, to ensure system stability, we have retrieved the first 10,000 records for you. For complete data, please try narrowing your query scope, such as adding a condition that 'creation time' is within the last month."

[0192] Through the above implementation methods, the data query method provided by this invention builds a powerful and extremely robust pagination governance layer on the client side without changing the existing application programming interface, effectively solving the three core problems of performance, stability and data consistency in the intelligent agent calling scenario.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0194] Embodiments of the present invention also provide a data query apparatus, comprising: an interception unit, configured to intercept a first query request generated by a client device when performing a paginated data retrieval task based on the client, the first query request including a first pagination parameter representing pagination capacity; an acquisition unit, configured to acquire a first pagination restriction parameter corresponding to a target application interface of a target server corresponding to the first query request; a ruling unit, configured to rule on the first pagination parameter using the first pagination restriction parameter to obtain a second pagination parameter; an update unit, configured to update the first query request using the second pagination parameter to obtain a second query request; and a sending unit, configured to send the second query request to the target server to invoke the target application interface to respond to the paginated data retrieval task according to the second pagination parameter.

[0195] In this embodiment of the invention, the acquisition unit can also be used to acquire a second pagination restriction parameter. The adjudication unit adjudicates the first pagination parameter using the first pagination restriction parameter to obtain the second pagination parameter, including: adjudicating the first pagination parameter using the first pagination restriction parameter and the second pagination restriction parameter to obtain the second pagination parameter. The step of acquiring the second pagination restriction parameter includes: if a third pagination restriction parameter corresponding to the client exists, then the third pagination restriction parameter is used as the second pagination restriction parameter; if a third pagination restriction parameter does not exist but a fourth pagination restriction parameter corresponding to the resource type of the target application interface exists, then the fourth pagination restriction parameter is used as the second pagination restriction parameter; if neither the third nor the fourth pagination restriction parameter exists, then the default pagination restriction parameter is used as the second pagination restriction parameter.

[0196] In this embodiment of the invention, the adjudication unit uses the first pagination restriction parameter and the second pagination restriction parameter to adjudicate the first pagination parameter to obtain the second pagination parameter, which may include: using the minimum value among the first pagination parameter, the first pagination restriction parameter, and the second pagination restriction parameter as the second pagination parameter.

[0197] The data query device provided in this embodiment of the invention may further include: a guidance unit, used to pre-inject pagination suggestion information into the client to guide the client to generate a first query request based on the pagination suggestion information; wherein the pagination suggestion information includes at least a fifth pagination restriction parameter.

[0198] In this embodiment of the invention, the sending unit sends a second query request to the target server to call the target application interface to respond to the pagination data acquisition task according to the second pagination parameters. This may include: repeatedly sending the second query request to the target server until the termination judgment condition in the second query request is met; wherein, starting from the second time the second query request is sent, before each second query request is sent, the target data address parameter in the current second query request is updated according to the return data of the previous second query request.

[0199] In this embodiment of the invention, the data query device provided may further include: a monitoring unit, configured to determine whether the status of the pagination data acquisition task triggers a circuit breaker condition before sending the second query request; if yes, then terminate the pagination data acquisition task; if no, then trigger the sending unit to send the second query request; wherein the circuit breaker condition includes at least one of the following: the continuous execution time of the pagination data acquisition task reaches the task deadline, the amount of data acquired by the pagination data acquisition task reaches the data amount limit, the system resources consumed by the pagination data acquisition task reach the preset system resource consumption limit, and the access frequency corresponding to the pagination data acquisition task reaches the access frequency limit corresponding to the target application interface.

[0200] In this embodiment of the invention, the second query request may further include a sorting parameter, which is determined according to the sorting type corresponding to the target application interface; the sending unit cyclically sends the second query request to the target server, which may include: cyclically sending the second query request to the target server to call the target application interface to sort the target data according to the sorting parameter, and then sequentially obtaining the number of target data corresponding to the second pagination parameter and sending them to the client device.

[0201] In this embodiment of the invention, the sending unit cyclically sends a second query request to the target server to call the target application interface to sort the target data according to the sorting parameters, and then sequentially obtains the target data corresponding to the second pagination parameters and sends it to the client device. This may include: after sending the first second query request corresponding to the pagination data acquisition task and receiving the first page of target data sent by the target server, obtaining the maximum and minimum values ​​of the metadata of the corresponding type of the first page of target data according to the sorting parameters, and using them as water level markers; determining the filtering conditions based on the sorting type and the water level markers; and before each second query request is sent, updating the target data address parameters in the current second query request according to the returned data of the previous second query request, and injecting the filtering conditions into the second query request.

[0202] The data query apparatus provided in this embodiment of the invention may further include: a cross-page processing unit, configured to inject cross-page overlap parameters into the second query request before each second query request is sent, starting from the second second query request, so that when the target application interface performs the paginated data acquisition task, it backtracks the number of target data corresponding to the cross-page overlap parameters and adds them to the returned data; starting from receiving the second page of target data returned by the target server, after each current page of target data is received, it retrieves the last first preset number of target data in the previous page and compares it with the first preset number of target data in the current page of target data to delete duplicate target data.

[0203] The data query device provided in this embodiment of the invention may further include: a deduplication unit, used to store the unique identifier of the target data returned by the target server; starting from receiving the second page of target data returned by the target server, each time the current page of target data is received, the unique identifier of the current page of target data is compared with the stored unique identifier to delete the duplicate target data.

[0204] The data query device provided in this embodiment of the invention may further include: a parameter adjustment unit, configured to, if the target server fails to execute the pagination data acquisition task, reduce the second pagination parameter in the second query request and resend the second query request; and if the target server successfully executes the second query request a second preset number of times after reducing the second pagination parameter, increase the second pagination parameter in the second query request.

[0205] The data query apparatus provided in this embodiment of the invention may further include: an indexing unit for caching query conditions in the second query request of an executed paginated data retrieval task as index information; and an updating unit for updating the first query request using the second pagination parameters to obtain the second query request, which may include: if the paginated data retrieval task and the executed paginated data retrieval task satisfy the query condition similarity condition, then updating the first query request according to the second pagination parameters and the index information to obtain the second query request.

[0206] In this embodiment of the invention, the adjudication unit updates the first query request according to the second paging parameter and index information to obtain the second query request, which may include: determining the target page number of the target data to be obtained by the paging data acquisition task; obtaining the target index information corresponding to the target page number from the cached index; adding the target index information to the first query request; and replacing the first paging parameter in the first query request with the second paging parameter to obtain the second query request.

[0207] In this embodiment of the invention, the indexing unit can also be used to: delete the index information after the cached index information has reached a preset time.

[0208] For a description of the features in the embodiment corresponding to the data query device, please refer to the relevant description in the embodiment corresponding to the data query method, which will not be repeated here.

[0209] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data query method embodiments.

[0210] Embodiments of the present invention also provide a non-volatile storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data query method embodiments when running.

[0211] In one exemplary embodiment, the aforementioned non-volatile storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0212] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data query method embodiments.

[0213] Embodiments of the present invention also provide another computer program product, including a non-volatile storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data query method embodiments.

[0214] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0215] The foregoing has provided a detailed description of the data query method, electronic device, medium, and computer program product provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A data query method, characterized in that, include: The first query request generated by the client device when the client performs a pagination data retrieval task is intercepted. The first query request includes a first pagination parameter representing the pagination capacity. Obtain the first pagination restriction parameter corresponding to the target application interface of the target server corresponding to the first query request; The first pagination parameter is determined by using the first pagination restriction parameter to obtain the second pagination parameter; The first query request is updated using the second pagination parameter to obtain the second query request; The second query request is sent to the target server to invoke the target application interface to respond to the pagination data retrieval task according to the second pagination parameters.

2. The data query method according to claim 1, characterized in that, Also includes: Get the second pagination limit parameters; The first pagination parameter is determined using the first pagination restriction parameter to obtain the second pagination parameter, including: The first pagination parameter is determined by using the first pagination restriction parameter and the second pagination restriction parameter to obtain the second pagination parameter; The steps for obtaining the second pagination restriction parameter include: If a third pagination restriction parameter exists for the client, then the third pagination restriction parameter shall be used as the second pagination restriction parameter; If the third pagination restriction parameter does not exist, but a fourth pagination restriction parameter corresponding to the resource type of the target application interface exists, then the fourth pagination restriction parameter shall be used as the second pagination restriction parameter. If the third pagination restriction parameter and the fourth pagination restriction parameter do not exist, then the default pagination restriction parameter shall be used as the second pagination restriction parameter.

3. The data query method according to claim 2, characterized in that, The first pagination parameter is determined using the first pagination restriction parameter and the second pagination restriction parameter to obtain the second pagination parameter, including: The minimum value among the first pagination parameter, the first pagination restriction parameter, and the second pagination restriction parameter is used as the second pagination parameter.

4. The data query method according to claim 1, characterized in that, Also includes: Pagination suggestion information is pre-injected into the client to guide the client to generate the first query request based on the pagination suggestion information; The pagination suggestion information includes at least the fifth pagination restriction parameter.

5. The data query method according to claim 1, characterized in that, Sending the second query request to the target server to invoke the target application interface to respond to the pagination data retrieval task according to the second pagination parameters includes: The process of sending the second query request to the target server is repeated until the termination condition in the second query request is met. Specifically, starting from the second time the second query request is sent, before each second query request is sent, the target data address parameter in the current second query request is updated based on the returned data of the previous second query request.

6. The data query method according to claim 5, characterized in that, Before sending the second query request, the following is also included: Determine whether the status of the pagination data retrieval task triggers the circuit breaker condition; If so, then the pagination data retrieval task ends; If not, then execute the sending of the second query request; The circuit breaker conditions include at least one of the following: the continuous execution time of the pagination data acquisition task reaches the task deadline, the amount of data acquired by the pagination data acquisition task reaches the data volume limit, the system resources consumed by the pagination data acquisition task reach the preset system resource consumption limit, and the access frequency corresponding to the pagination data acquisition task reaches the access frequency limit corresponding to the target application interface.

7. The data query method according to claim 5, characterized in that, The second query request also includes sorting parameters, which are determined according to the sorting type corresponding to the target application interface; The process of repeatedly sending the second query request to the target server includes: The process involves repeatedly sending the second query request to the target server, calling the target application interface to sort the target data according to the sorting parameters, and then sequentially retrieving the target data corresponding to the second pagination parameters and sending it to the client device.

8. The data query method according to claim 7, characterized in that, The process involves repeatedly sending the second query request to the target server, invoking the target application interface to sort the target data according to the sorting parameters, and then sequentially retrieving the target data corresponding to the second pagination parameters and sending it to the client device, including: After sending the first second query request corresponding to the pagination data acquisition task and receiving the first page of target data sent by the target server, the maximum and minimum values ​​of the metadata of the corresponding type of the first page of target data are obtained according to the sorting parameters and used as water level markers; The filtering criteria, with the water level mark as the boundary, are determined based on the sorting type and the water level mark. Starting from the second time the second query request is sent, before each second query request is sent, the target data address parameter in the current second query request is updated according to the returned data of the previous second query request, and the filtering conditions are injected into the second query request.

9. The data query method according to claim 8, characterized in that, The sorting type is sorted by creation time, the sorting parameters are sorted in ascending order by creation time, the water level mark is the earliest creation time of the target data on the first page, and the filtering condition is to filter the target data whose creation time is earlier than the water level mark.

10. The data query method according to claim 8, characterized in that, The sorting type is sorted by creation time, the sorting parameters are sorted in descending order by creation time, the water level mark is the latest creation time of the target data on the first page, and the filtering condition is to filter the target data whose creation time is later than the water level mark.

11. The data query method according to claim 5, characterized in that, Also includes: Starting from the second time the second query request is sent, before each second query request is sent, the cross-page overlap parameter is injected into the second query request, so that when the target application interface performs the pagination data acquisition task, it backtracks the target data corresponding to the cross-page overlap parameter and adds it to the returned data. Starting from receiving the second page of target data returned by the target server, each time the target data of the current page is received, the last first preset number of target data in the previous page is taken out and compared with the first preset number of target data in the target data of the current page to delete the duplicate target data.

12. The data query method according to claim 5, characterized in that, Also includes: A unique identifier for storing the target data returned by the target server; Starting from receiving the second page of target data returned by the target server, each time the target data of the current page is received, the unique identifier of the target data of the current page is compared with the stored unique identifier to delete the target data that has been obtained repeatedly.

13. The data query method according to claim 5, characterized in that, Also includes: If the target server is detected to have failed to execute the pagination data retrieval task, the second pagination parameter in the second query request is reduced and the second query request is resent. After reducing the second pagination parameter, if the target server successfully executes the second query request a second preset number of times, then the second pagination parameter in the second query request is increased.

14. The data query method according to claim 5, characterized in that, Also includes: The query conditions in the second query request of the already executed pagination data retrieval task are cached as index information; The first query request is updated using the second pagination parameter to obtain the second query request, including: If the pagination data retrieval task and the already executed pagination data retrieval task meet the query condition similarity condition, then the first query request is updated according to the second pagination parameters and the index information to obtain the second query request.

15. The data query method according to claim 14, characterized in that, The first query request is updated based on the second pagination parameters and the index information to obtain the second query request, including: Determine the target page number of the target data to be acquired by the pagination data acquisition task; The target index information corresponding to the target page number is obtained from the cached index; The target index information is added to the first query request, and the first pagination parameter in the first query request is replaced with the second pagination parameter to obtain the second query request.

16. The data query method according to claim 14, characterized in that, The similarity condition for the query criteria is that they have the same water level marker; The water level marker is the maximum or minimum value in the metadata corresponding to the sorting parameters on the first page returned by the target server after sorting according to the sorting parameters in the second query request.

17. The data query method according to claim 14, characterized in that, Also includes: The index information is deleted after the cached index information has been cached for a preset time.

18. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data query method as described in any one of claims 1 to 17 when executing the computer program.

19. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the data query method as described in any one of claims 1 to 17.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data query method as described in any one of claims 1 to 17.

Images