An adaptive paging data pulling method for external API

Abstract

This invention provides an adaptive paginated data retrieval method for external APIs. It employs a unified abstraction of multiple pagination methods combined with runtime feature-driven adaptive optimization. Through unified pagination capability modeling and adapter encapsulation, it standardizes the processing of all types of pagination mechanisms. By real-time collection of request end-to-end features and dynamic decision-making by a strategy engine, it adaptively adjusts pagination parameters and request rhythm. Without altering the original specifications of the external API, it balances the efficiency, stability, and reliability of batch data retrieval, while significantly reducing the access and maintenance costs for multi-API scenarios.

Description

Technical Field

[0001] This invention relates to the field of API paging technology, and in particular to an adaptive paging data retrieval method, electronic device, and computer-readable storage medium for external APIs. Background Technology

[0002] With the widespread adoption of enterprise-level microservice architectures and SaaS services, cross-system data interaction has become a core aspect of digital business. Many enterprise business systems need to connect to external third-party open APIs (such as e-commerce platforms, financial institutions, government services, and industrial internet platforms) to complete batch data retrieval, synchronization, and integration. To avoid issues like interface timeouts, service overload, and data transmission interruptions caused by a single request returning massive amounts of data, external APIs generally employ pagination mechanisms to return batch data in segments. Paginated data retrieval has thus become a core foundational capability for cross-system API data interaction.

[0003] Currently, there is no unified standard for pagination mechanisms in internal and external APIs within the industry. The mainstream implementations fall into four categories:

[0004] Page number pagination: This method uses a combination of page / pageSize parameters to locate the data range by page number + number of items per page, and is the most common pagination method.

[0005] Offset-based pagination: This method uses a combination of offset and limit parameters to locate data based on the data offset plus the number of records retrieved at one time. It is suitable for batch retrieval scenarios where the data sorting is fixed.

[0006] Cursor-based pagination: Using the combination of cursor / nextToken / after / first parameters, the cursor marker returned from the previous page is used to locate the starting position of the next page of data. It is suitable for high-stability retrieval scenarios with massive amounts of data and real-time data changes.

[0007] Custom private pagination: Some vendors design non-standard pagination protocols based on their own business needs, using custom parameter names, pagination rules, and data return formats.

[0008] However, the existing pagination data retrieval technologies for external APIs in the aforementioned industries are all developed around fixed pagination logic for a single interface, lacking unified abstraction, encapsulation, and dynamic adaptation capabilities. This results in a core flaw that cannot be avoided in scenarios involving multiple API connections and high-concurrency batch retrieval.

[0009] 1. Lack of unified pagination abstraction leads to extremely poor code reusability and high integration and maintenance costs.

[0010] Existing systems typically develop a separate set of pagination logic for each external API, writing independent parameter construction, state update, and termination judgment code for different pagination methods. When a new API is added or the pagination rules of an existing API change, all pagination logic needs to be redeveloped or modified, resulting in high code redundancy. In multi-API scenarios, development and maintenance costs increase linearly, making it impossible to reuse core pagination capabilities.

[0011] 2. Fixed pagination strategy cannot adapt to real-time status fluctuations of external APIs.

[0012] Existing solutions generally employ static pagination strategies with fixed page sizes, fixed request frequencies, and fixed concurrency, which cannot be dynamically adjusted based on the real-time operational status of external APIs. When the performance of external API interfaces fluctuates or the service load increases, a fixed large page size can lead to request timeouts and service anomalies; when the API triggers rate limiting rules, a fixed request frequency can lead to continuous request failures or even IP blocking; while a conservative fixed small page size and low request frequency can result in inefficient batch data retrieval, failing to fully utilize the API's service capabilities.

[0013] 3. Lack of a unified fault tolerance and backoff mechanism results in low reliability of fetching tasks.

[0014] The existing solution's exception handling logic is deeply coupled with a single API, lacking unified fault tolerance capabilities for timeouts, rate limiting, and service anomalies. When consecutive request failures occur, there is no standardized retry backoff or circuit breaker degradation mechanism, which can easily lead to the interruption of the entire batch fetching task due to a single request failure, failing to guarantee the continuity and reliability of fetching massive amounts of data.

[0015] 4. The pagination logic is deeply coupled with the business logic, resulting in poor system scalability.

[0016] The existing solution couples the core pagination logic, such as pagination parameter construction, request sending, status update, and data parsing, with the upper-layer business logic. When pagination rules, API access addresses, or rate limiting strategies change, the business code needs to be modified synchronously, which can easily lead to business logic failures and results in extremely poor system scalability and maintainability. Summary of the Invention

[0017] To address the technical problems existing in the prior art, the present invention provides the following technical solution:

[0018] On the one hand, an adaptive pagination data retrieval method for external APIs is provided, characterized by the following steps:

[0019] S1. Receive data query requests initiated by upper-layer services or clients and perform legality verification;

[0020] S2. Based on the target external API identifier, read the corresponding pagination capability metadata model, identify the pagination methods supported by the API, and automatically match the corresponding pagination adapter;

[0021] S3. Complete the initialization of the selected paging adapter, load the historical running statistics of the corresponding API as the initial strategy reference, and initialize the unified internal paging state at the same time.

[0022] S4. Use the pagination adapter to convert the current unified internal pagination state into the HTTP request parameters required by the target external API, construct a standardized HTTP request, and embed the task identifier and pagination state identifier.

[0023] S5. Send a pagination request to the target external API, record the timestamp and status information of the entire request chain, monitor the request status, and trigger the basic retry mechanism to handle network anomalies.

[0024] S6. Receive API response or capture request exception, complete the initial parsing of response data, and collect and record the original characteristics of this request.

[0025] S7. Based on the original features collected in this request, update the statistical features corresponding to the target API through a sliding window of a preset size;

[0026] S8. Using the original features and statistical features as input, the adaptive strategy engine completes automated decision-making and dynamically optimizes the core pagination parameters for the next round of requests. The optimized parameters are then synchronized to the unified internal pagination status.

[0027] S9. Complete the full parsing of response data and extraction of valid business data, and update the unified internal pagination status according to the response content through the pagination adapter;

[0028] S10. Combining the API response content with the constraints of the query task context, determine whether pagination is still needed. If pagination is still needed, return to step S4 to enter the next round of pagination request loop. If pagination is not needed, enter the task closing stage.

[0029] S11. Summarize all the business data retrieved, verify data integrity, and generate a statistical report for this retrieval task.

[0030] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described above.

[0031] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above method.

[0032] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0033] 1. A unified abstraction of multiple pagination methods completely solves the core problems of poor reusability and code redundancy in existing technologies.

[0034] This invention abstracts and encapsulates all pagination mechanisms, including page number type, offset type, cursor type, and custom type, through a standardized pagination capability metadata model, a unified adapter interface, and an internal pagination state representation. Upper-layer business processes only need to operate on the unified pagination state and adapter interface, without needing to care about the specific implementation details of the underlying pagination.

[0035] This design achieves consistent pagination processing for multiple external APIs. The core pagination retrieval, strategy optimization, and exception handling logic are 100% reusable. When adding new APIs, there is no need to rewrite the core pagination logic; only metadata configuration or a minimal adapter logic needs to be implemented. The code reusability rate is improved by more than 90%, completely solving the problems of interface-by-interface development and high code redundancy in existing technologies.

[0036] 2. Runtime feature-driven adaptive pagination strategy to address the pain point that fixed strategies cannot adapt to interface fluctuations.

[0037] This invention constructs a full-link feature acquisition system and an adaptive strategy engine. Based on runtime features such as API real-time response time, error rate, and rate limiting, it dynamically adjusts pagination size, concurrency, request interval, and retry strategy, achieving real-time optimization of pagination strategy: when API service performance is good, it automatically improves fetch efficiency and makes full use of API service capabilities; when API performance fluctuates, rate limits are applied, or timeouts occur, it automatically degrades and avoids such situations to ensure request success rate.

[0038] Verified in real-world business scenarios, this design can reduce API request failure rates by more than 80%, increase the overall throughput of batch data retrieval by more than 50%, and completely avoid issues such as rate limiting and blocking and task interruption caused by fixed strategies, thus significantly improving the stability and efficiency of data retrieval.

[0039] 3. A scalable architecture decoupled from external APIs significantly reduces the cost of accessing and maintaining multiple API scenarios.

[0040] This invention adopts a layered and decoupled architecture design, which completely decouples the core pagination logic, the adaptive strategy engine and the pagination rules of external APIs. When adding a new API or changing the pagination rules of an existing API, only the metadata configuration or the adapter implementation needs to be modified, without changing the core fetch logic and business code.

[0041] This design shortens the new interface access cycle in multi-API scenarios from the traditional days to hours, reduces subsequent maintenance costs by more than 70%, and avoids the risk of business code modification due to changes in pagination rules, greatly improving the scalability and maintainability of the system.

[0042] 4. A standardized fault-tolerance mechanism across the entire data chain significantly improves the reliability of batch data retrieval.

[0043] This invention constructs a standardized fault tolerance system covering all scenarios including timeouts, rate limiting, service anomalies, and network anomalies. Through unified retry backoff, circuit breaker degradation, and breakpoint resumption mechanisms, it avoids interruptions to the entire data fetching task caused by a single request anomaly, achieving continuity and reliability in the batch fetching of massive amounts of data. Simultaneously, the end-to-end feature collection and statistics system enables observability and traceability of the entire API fetching process, facilitating problem localization and performance optimization, and solving the problems of insufficient fault tolerance and coupling between existing anomaly handling and single APIs. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the API pagination mechanism provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the implementation process of the method provided in the embodiments of the present invention. Detailed Implementation

[0047] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0048] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0049] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0050] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0051] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0052] This invention proposes a pagination data retrieval method that combines unified abstraction of multiple pagination modes with runtime feature-driven adaptive optimization. By unifying pagination capability modeling and adapter encapsulation, it achieves standardized processing of all types of pagination mechanisms. Through real-time collection of request end-to-end features and dynamic decision-making by a strategy engine, it enables adaptive adjustment of pagination parameters and request rhythm. Without changing the original specifications of external APIs, it balances the efficiency, stability, and reliability of batch data retrieval, while significantly reducing the access and maintenance costs of multi-API scenarios.

[0053] This invention adopts a layered and decoupled cloud-native architecture design, consisting of six core modules from top to bottom. All modules interact through standardized interfaces to achieve capability reuse and flexible expansion, fully covering all technical modules and process nodes in the original document. The architecture layers and core responsibilities are shown in Table 1 below:

[0054] Architecture layering Core responsibilities and included modules Upper-layer business access layer It receives data query requests from upper-layer business / clients, encapsulates them into standardized query task contexts, and manages the task lifecycle. Pagination unified adaptation layer The core implementation provides a unified abstraction for multiple pagination methods, including an API pagination capability metadata model, a standardized pagination adapter interface, and implementations for multiple types of pagination adapters. Adaptive Strategy Engine Layer The core decision-making center of this invention includes a feature processing module, a strategy decision-making module, a threshold configuration management module, and a rule engine execution unit. Request Execution and Feature Acquisition Layer Responsible for constructing and sending pagination HTTP requests, completing end-to-end data collection for request / response / errors, and outputting raw feature data. Pagination status management layer Maintain a unified internal paging state, complete the updating, validation, and lifecycle management of the paging state, and decouple it from the adapter. underlying HTTP communication layer It provides basic HTTP request sending, timeout control, TLS encryption, and connection pool management capabilities, offering standardized communication capabilities to upper layers.

[0055] Table 1

[0056] This invention first models the pagination capabilities of various external APIs in a unified manner, recording their pagination method types and pagination parameter mapping relationships, and defining a pagination adaptation interface. Different pagination mechanisms, such as page number type, offset type, and cursor type, are encapsulated into a unified internal pagination state representation. Based on this, during the pagination fetching process, the response time, number of returned records, and error types (including timeout, rate limiting, and service exceptions) of each request are continuously collected. An adaptive strategy engine dynamically adjusts the unified internal pagination parameters (such as page size, request rhythm, retry, and backoff strategies), which are then converted by specific adapters into the actual request parameters required by each API. Compared with existing fixed-strategy, interface-by-interface implementations, this invention achieves unified processing and adaptive optimization for multiple pagination methods without changing existing API specifications. It significantly reduces timeout and rate limiting risks while ensuring the stability of external interfaces, improves the overall throughput and reliability of batch data fetching, and reduces the cost of multi-API access and maintenance.

[0057] The functions of the core modules of the system of the present invention will be described in detail below:

[0058] 1. API pagination capability metadata model

[0059] This invention first establishes a unified model for the pagination capabilities of all external APIs, converting non-standardized pagination rules into standardized metadata configurations. The core fields of the model include: API unique identifier, pagination method type, pagination request parameter mapping relationship, response pagination field mapping relationship, supported maximum / minimum pagination size, API rate limiting quota rules, timeout threshold, historical runtime statistics, and custom extended configurations. This model forms the foundation for unified pagination adaptation; all external API pagination rules can be standardized and described using this model.

[0060] 2. Standardized paging adapter interface

[0061] like Figure 1 As shown, this invention defines a standardized adapter interface that is independent of the pagination method. All pagination logic is implemented through this interface. Upper-layer business logic and core processes only depend on this interface and do not care about the specific pagination implementation at the underlying level. The core methods of the interface include:

[0062] init(): Adapter initialization, loads API pagination metadata, and sets initial pagination parameters;

[0063] buildRequestParam(): Converts the uniform internal pagination state into the HTTP request parameters required by the target API;

[0064] updatePageState(): Updates the unified internal pagination state based on the API response content;

[0065] isEnd(): Determines whether the pagination retrieval termination condition has been met based on the API response content and task constraints.

[0066] Upon receiving the API request, the pagination adapter is selected based on the API request before proceeding with the subsequent pagination execution process.

[0067] Based on this standardized adapter interface, this invention implements four types of universal adapters, covering all mainstream paging methods:

[0068] Page number adapter: Adapts to page / pageSize and other page number types, and enables standardized processing of page number auto-increment and page size adjustment;

[0069] Offset adapter: Adapts to offset-based pagination such as offset / limit, and enables standardized processing of offset accumulation and adjustment of the number of items retrieved;

[0070] Cursor adapter: Adapts to cursor-type pagination such as cursor / nextToken / after, and realizes standardized processing of cursor parsing and updating;

[0071] Custom adapter: Adapts to vendor-specific non-standard paging protocols, implements custom logic by inheriting standardized interfaces, and reuses the entire set of core fetching and adaptive capabilities.

[0072] 3. Standardize the internal pagination status representation

[0073] This invention abstracts the core states of all pagination methods into a unified internal pagination state, independent of the pagination type. This unified state serves as the core data carrier for the entire workflow. Core fields include: current pagination position, pagination size, current cursor value, cumulative number of data fetched, cumulative number of requests, number of consecutive successful requests, number of consecutive failed requests, current concurrency, current request interval, task start time, and cumulative execution time. This unified state completely decouples the upper-level workflow from the underlying pagination method; all strategy adjustments and state updates are based on this unified state.

[0074] 4. Adaptive Strategy Engine Core Rule System

[0075] The strategy engine of this invention makes dynamic decisions based on real-time request characteristics and sliding window statistical characteristics through a predefined rule system. The core rules include pagination size adjustment rules, concurrency adjustment rules, request interval adjustment rules, retry and backoff rules, and exception circuit breaker rules. All rules can be flexibly adjusted through the threshold configuration module to adapt to the characteristics of different APIs.

[0076] like Figure 2 As shown, the adaptive pagination data retrieval process of this invention is as follows:

[0077] Step 1: Business Request Access and Query Task Context Encapsulation

[0078] Send API request:

[0079] The system receives and parses API requests. Upper-layer business logic or clients initiate data query requests, which include: the target external API identifier, query conditions, the expected data range to be retrieved (time interval, maximum number of records, etc.), and acceptable latency or timeout thresholds. The system encapsulates this request into a unified "query task context," which serves as the input for the subsequent pagination retrieval process.

[0080] Therefore, when an upper-layer business or client initiates a data query request, the request parameters include: a unique identifier for the target external API, data query conditions, the expected data range to be retrieved (time interval, maximum number of records to retrieve, etc.), acceptable latency / timeout thresholds, and maximum error count constraints. The system encapsulates this request into a standardized query task context, serving as the sole input carrier for the entire subsequent process. Simultaneously, it assigns a unique task ID to the task, initializes the task lifecycle state, and completes the task's validity verification.

[0081] Step 2: Target API pagination metadata matching and pagination adapter selection

[0082] The system identifies the pagination methods supported by the API based on its pre-configuration or metadata, and selects from the following adapters:

[0083] Page number adapter: Suitable for page number-based pagination such as page / pageSize;

[0084] Offset adapter: Suitable for offset-based pagination such as offset / limit;

[0085] Cursor adapter: Suitable for cursor-type pagination such as cursor / nextToken / after / first;

[0086] Custom adapters: Suitable for vendor-proprietary or non-standard paging protocols.

[0087] This choice allows subsequent processes to rely solely on a unified adapter interface, without needing to know the specific pagination parameters.

[0088] For example, based on the unique identifier of the target API in the request, the system reads the pagination capability metadata model corresponding to the API from the metadata configuration library, and identifies the core configurations supported by the API, such as pagination methods, parameter mapping rules, and constraint thresholds.

[0089] Based on the API-based pagination method type, the system automatically matches and instantiates the corresponding pagination adapter: page number pagination matches the page number adapter, offset pagination matches the offset adapter, cursor pagination matches the cursor adapter, and non-standard private pagination matches the custom adapter. Through the standardized interface of the adapter, the entire subsequent process only relies on the unified interface, without needing to concern itself with the implementation details of the underlying pagination parameters.

[0090] Step 3: Paging adapter initialization and initial strategy loading

[0091] After the adapter is selected, the system completes the adapter initialization process:

[0092] Based on the API pagination metadata, set the initial pagination parameters for the adapter: pageIndex=1 for page number type, offset=0 for offset type, and cursor=null for cursor type. Set the default initial page size for all types.

[0093] Load the historical statistics for this API, including the recent response time distribution, error rate, rate limiting triggering, and optimal page size range, as a reference for the initial strategy;

[0094] Initialize the unified internal paging state, synchronize the initial parameters to the unified state model, and initialize the task's statistical indicator counters.

[0095] For example, setting initial paging parameters for the selected adapter, such as:

[0096] Page number type: pageIndex = 1, pageSize = default value;

[0097] Offset type: offset = 0, limit = default value;

[0098] Cursor type: cursor = null, pageSize = default value.

[0099] At the same time, load historical statistics related to this API (recent response time distribution, error rate, rate limiting, etc.) as a reference for the initial strategy.

[0100] Step 4: Pagination Request Parameter Conversion and HTTP Request Construction

[0101] The adapter calls the buildRequestParam() method to convert the current unified internal pagination state into the actual HTTP request parameters required by the target external API, including the encapsulation of pagination parameters, query conditions, authentication information, etc.

[0102] Based on the converted parameters, the system constructs standardized HTTP requests, sets basic timeout, request headers, and request methods, and embeds the current task ID and pagination status identifier into the request context for subsequent end-to-end tracing and feature collection.

[0103] Once the page is constructed, a pagination request is sent. The pagination adapter converts the current internal pagination state into specific HTTP request parameters. The system records the request start time and sends this pagination request to the target external API, waiting for the return result.

[0104] Step 5: Sending pagination requests and recording end-to-end state

[0105] The system sends the pagination request to the target external API through the underlying HTTP communication layer, and records the entire chain information such as the request start timestamp, request parameters, and pagination status. If concurrent fetching capability is enabled, the system initiates multiple non-overlapping pagination requests simultaneously based on the current concurrency configuration, thereby improving fetching efficiency.

[0106] During the request sending process, the system monitors the request status in real time. If a basic network anomaly occurs, the basic retry mechanism is immediately triggered, and the anomaly information is recorded for subsequent feature collection.

[0107] Step 6: Request Response / Exception Handling and Raw Feature Collection

[0108] Upon receiving an API response or encountering a request exception, the system performs unified processing of the response / exception, and simultaneously calculates and records the original characteristics of this request. All characteristics are bound to the task ID and the unique identifier of this request. The original characteristics include:

[0109] Performance characteristics: actual response time of this request, DNS resolution time, TCP connection time, and server processing time;

[0110] Data characteristics: the number of valid data records returned in this response, and the size of the data body;

[0111] Status characteristics: HTTP response status code, rate limiting related fields in the response header (remaining request quota, reset time, etc.);

[0112] Anomaly characteristics: error type (timeout, rate limiting, 4xx client error, 5xx server error, network anomaly, etc.), error details, and the node where the anomaly occurred.

[0113] If the request receives a normal response, the system performs format validation and preliminary parsing of the response data; if the request is abnormal, the system records the corresponding error flag according to the abnormality type, providing input for subsequent strategy decisions.

[0114] For example, upon receiving a response or encountering an error, the system calculates and records the original characteristics of this pagination request, including:

[0115] Actual response time;

[0116] Returns the number of records;

[0117] HTTP status codes;

[0118] Error type (e.g., timeout, 5xx service error, etc.);

[0119] Rate limiting related response header fields (if present).

[0120] Simultaneously, based on the original characteristics of the most recent N requests, the statistical characteristics are updated, including:

[0121] Average response time and quantile response time of the most recent N requests (e.g., P95).

[0122] Error rate of the most recent N requests;

[0123] Recent consecutive successes and consecutive failures.

[0124] Step 7: Updating the statistical features of the sliding window

[0125] Based on the raw features collected in this request, the system updates the statistical features corresponding to this API using a sliding window of a preset size (default window size N=20 requests) for subsequent adaptive strategy decisions. These statistical features include:

[0126] Response time statistics: average response time of the most recent N requests, P95 / P99 quantile response time, response time volatility;

[0127] Error statistics: Overall error rate of the most recent N requests, rate limiting error rate, timeout error rate, and service exception error rate;

[0128] Continuity statistics: number of recent consecutive successful requests, number of recent consecutive failed requests;

[0129] Efficiency statistics: number of valid data entries retrieved per unit time, average retrieval efficiency per request.

[0130] All statistical features are updated in real time and synchronized to the historical statistical information database of API pagination metadata for reference in the initial strategy of subsequent tasks.

[0131] Step 8: Adaptive strategy decision and optimization of next round of request parameters

[0132] The system takes the original features and sliding window statistical features as input, and through the rule system of the adaptive strategy engine, completes automated decision-making and dynamically optimizes the core parameters of the next round of requests. The core decision-making logic includes:

[0133] Concurrency adjustment rules

[0134] Concurrency Improvement: When the following three conditions are met simultaneously: "the average response time of the most recent N requests is lower than the first latency threshold T_low", "the error rate of the most recent N requests is lower than the first error rate threshold E_low", and "the number of consecutive successes is greater than or equal to the preset value K_success", the current concurrency will be increased by one level, up to a maximum of the preset maximum concurrency C_max.

[0135] Concurrency reduction: When either of the following conditions is triggered, "the average response time of the most recent N requests is higher than the second latency threshold T_high" or "the error rate of the most recent N requests is higher than the second error rate threshold E_high", the current concurrency will be reduced by one level, and the minimum concurrency will not be lower than the preset minimum concurrency C_min.

[0136] Page size adjustment rules

[0137] Page size increase: When the interface response performance is good and no errors occur, the page size is increased by a preset step size, not exceeding the maximum page size supported by the API, thereby improving the efficiency of single request fetching.

[0138] Page size reduction: When timeouts, excessively long response times, or server errors occur, the page size is reduced by a preset ratio to decrease the server load on a single request and improve the success rate of requests.

[0139] Request intervals and backoff rules

[0140] If the API returns rate limiting information, or if the remaining request quota within the current time window is nearly exhausted, the request interval will be increased exponentially according to the preset linear backoff strategy to avoid triggering stricter rate limiting rules.

[0141] If continuous service anomalies or timeouts occur, the request interval will be adjusted according to the exponential backoff strategy, and the circuit breaker detection will be triggered. When the number of consecutive failures reaches the threshold, the requests will be temporarily suspended to prevent invalid requests from continuously consuming resources.

[0142] Retry strategy optimization: The number of retries and the retry interval are dynamically adjusted according to the error type. Idempotent requests support automatic retries, while non-idempotent requests are only retried if the data is not actually sent after a timeout, thus avoiding data duplication.

[0143] After the strategy decision is made, the system will synchronize the optimized parameters to the unified internal paging status as input for the next round of requests.

[0144] Example: Using raw features and statistical features as input, a rule-based algorithm based on thresholds and sliding windows automatically determines the concurrency and request interval for the next round of requests, including but not limited to:

[0145] The concurrency level will be increased by one level (e.g., from C to C+1, but not exceeding the preset upper limit C_max) when the following conditions are met simultaneously:

[0146] The average response time of the most recent N requests is lower than the first latency threshold T_low;

[0147] The error rate of the most recent N requests is lower than the first error rate threshold E_low;

[0148] The number of recent consecutive successes is greater than or equal to the preset value K_success.

[0149] Request interval adjustment rules (when to "increase request interval"): If API rate limiting occurs, or if the remaining request quota in the current time window is close to being exhausted, the request interval time will be increased exponentially according to the preset linear backoff strategy.

[0150] Step 9: Response Data Parsing and Pagination Status Update

[0151] For requests with normal responses, the system completes full parsing of the response data. Simultaneously, the paging adapter calls the `updatePageState()` method to update the unified internal paging state based on the response content. The specific update rules are as follows:

[0152] Page number pagination: Increment the current page number by 1 and update the cumulative number of retrieved items simultaneously;

[0153] Offset-based pagination: Update the current offset to "original offset + number of records returned this time", and update the cumulative number of records retrieved simultaneously;

[0154] Cursor-based pagination: Parse the nextCursor / endCursor cursor values ​​from the response body, update the cursor fields to a unified state, and synchronously update the cumulative number of retrieved records.

[0155] Meanwhile, the system completes a full traversal and parsing of the JSON structure of the response data: for object type nodes, it accesses the child nodes corresponding to each field in turn, updates the path according to the field name, and then recursively calls the traversal logic; for non-target array nodes, it accesses each element in the array in turn, updates the path in the form of wildcards or indices, and then continues to traverse, finally extracting all valid business data and verifying data integrity.

[0156] After the current node and its subordinate target arrays are processed, the system continues to traverse downwards along the JSON structure: for object type nodes, the system accesses the child nodes corresponding to each field in turn, updates the path according to the field name, and then recursively calls the traversal logic; for non-target array nodes, the system accesses each element in the array in turn, updates the path in the form of wildcards or indices, and then continues traversal.

[0157] Update the pagination status; the pagination adapter updates the "current position" based on the response content:

[0158] Page number format: Page number + 1;

[0159] Offset type: offset = offset + number of records returned this time;

[0160] Cursor type: Resolve the nextCursor or endCursor from the response and update the cursor.

[0161] The updated result will serve as input for the next round of pagination requests.

[0162] Step 10: Pagination termination condition judgment and loop execution

[0163] The system calls the isEnd() method from the adapter, and, in conjunction with task constraints, determines whether to continue pagination. Termination conditions include:

[0164] The external API returns information that clearly indicates there is no more data (e.g., returning an empty array, hasNext=false, no next page cursor);

[0165] The cumulative number of records retrieved has reached the maximum number of records retrieved as set in the business request.

[0166] The cumulative time consumed by the task has reached the maximum time threshold set by the business logic.

[0167] The cumulative number of errors has reached the maximum error threshold set by the business logic, triggering task termination.

[0168] If the judgment result is "pagination still needs to continue": the process returns to step 4, and based on the updated pagination status and optimized strategy parameters, enters the next round of pagination request loop;

[0169] If the judgment result is "Do not continue pagination": the process enters the task closing stage and stops the pagination request loop.

[0170] Step 11: Task completion and result summary return

[0171] After the system completes the pagination retrieval, it summarizes all the retrieved business data, verifies the integrity and continuity of the data, and generates a statistical report for this retrieval task, including core indicators such as the total number of retrieved records, the total number of requests, the request success rate, the average response time, and the total time consumed.

[0172] Finally, the aggregated business data and task statistics report are returned to the upper-level business caller or subsequent processing module, completing the entire process of adaptive pagination data retrieval. At the same time, all feature data of this task are synchronized to the historical statistics information database for continuous optimization of the API pagination strategy.

[0173] This invention also features a "mode degradation and fault-tolerance fallback mechanism." By supplementing the design with a smooth degradation mechanism, when an external API experiences a serious anomaly or the adaptive strategy engine temporarily fails, it can automatically switch to a fixed-policy safety mode, completing data retrieval based on preset conservative paging parameters, thus preventing the entire retrieval function from failing. Simultaneously, a task breakpoint resumption mechanism is designed. When a task is unexpectedly interrupted, retrieval can resume from the interrupted position based on a unified internal paging state, without having to start from the beginning, ensuring the continuity of massive data retrieval.

[0174] Therefore, this invention first models the pagination capabilities of various external APIs in a unified manner, recording their pagination method types and pagination parameter mapping relationships, and defining a pagination adaptation interface. Different pagination mechanisms, such as page number type, offset type, and cursor type, are encapsulated into a unified internal pagination state representation. Based on this, during the pagination fetching process, the response time, number of returned records, and error types (including timeout, rate limiting, and service exceptions) of each request are continuously collected. An adaptive strategy engine dynamically adjusts the unified internal pagination parameters (such as page size, request rhythm, retry, and backoff strategies), which are then converted by specific adapters into the actual request parameters required by each API. Compared with existing fixed-strategy, interface-by-interface implementations, this invention achieves unified processing and adaptive optimization for multiple pagination methods without changing existing API specifications. It can significantly reduce timeout and rate limiting risks while ensuring the stability of external interfaces, improve the overall throughput and reliability of batch data fetching, and reduce the cost of multi-API access and maintenance.

[0175] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An adaptive pagination data retrieval method for external APIs, characterized in that, Includes the following steps: S1. Receive data query requests initiated by upper-layer services or clients and perform legality verification; S2. Based on the target external API identifier, read the corresponding pagination capability metadata model, identify the pagination methods supported by the API, and automatically match the corresponding pagination adapter; S3. Complete the initialization of the selected paging adapter, load the historical running statistics of the corresponding API as the initial strategy reference, and initialize the unified internal paging state at the same time. S4. Use the pagination adapter to convert the current unified internal pagination state into the HTTP request parameters required by the target external API, construct a standardized HTTP request, and embed the task identifier and pagination state identifier. S5. Send a pagination request to the target external API, record the timestamp and status information of the entire request chain, monitor the request status, and trigger the basic retry mechanism to handle network anomalies. S6. Receive API response or capture request exception, complete the initial parsing of response data, and collect and record the original characteristics of this request. S7. Based on the original features collected in this request, update the statistical features corresponding to the target API through a sliding window of a preset size; S8. Using the original features and statistical features as input, the adaptive strategy engine completes automated decision-making and dynamically optimizes the core pagination parameters for the next round of requests. The optimized parameters are then synchronized to the unified internal pagination status. S9. Complete the full parsing of response data and extraction of valid business data, and update the unified internal pagination status according to the response content through the pagination adapter; S10. Combining the API response content with the constraints of the query task context, determine whether pagination needs to continue; if pagination still needs to continue, return to step S4 to enter the next round of pagination request loop. If further pagination is not required, proceed to the task completion phase. S11. Summarize all the business data retrieved, verify data integrity, and generate a statistical report for this retrieval task.

2. The adaptive pagination data retrieval method for external APIs according to claim 1, characterized in that, In step S2, the pagination adapter is divided into four types of general adapters: page number adapter, offset adapter, cursor adapter, and custom adapter. Among them, the page number adapter adapts to page number pagination of type page / pageSize, the offset adapter adapts to offset pagination of type offset / limit, the cursor adapter adapts to cursor pagination of type cursor / nextToken / after, and the custom adapter adapts to vendor-specific non-standard pagination protocols. The standardized adapter interface includes at least the init initialization method, the buildRequestParam parameter constructor method, the updatePageState state update method, and the isEnd termination judgment method.

3. The adaptive pagination data retrieval method for external APIs according to claim 1, characterized in that, In step S2, the API pagination capability metadata model is a standardized metadata configuration. The core fields include at least the API unique identifier, pagination method type, pagination request parameter mapping relationship, response pagination field mapping relationship, pagination size upper and lower limits, API rate limiting quota rules, timeout threshold, historical operation statistics information, and custom extended configuration.

4. The adaptive pagination data retrieval method for external APIs according to claim 1, characterized in that, In step S6, the performance characteristics collected include the actual response time of this request, DNS resolution time, TCP connection time, and server processing time. Data characteristics include the number of valid data records returned in this response and the size of the data body; status characteristics include the HTTP response status code and rate-limiting related response header fields. The abnormal characteristics include the error type, error details, and the node where the abnormality occurred; In step S7, the default size of the sliding window is 20 requests. Among the statistical features, the response time statistics include the average response time of the most recent N requests, the P95 / P99 quantile response time, and the response time volatility. Error statistics include the overall error rate, rate limiting error rate, timeout error rate, and service exception error rate of the most recent N requests; continuity statistics include the number of most recent consecutive successful requests and the number of most recent consecutive failed requests; fetch efficiency statistics include the number of valid data records fetched per unit time and the average fetch efficiency per request.

5. The adaptive pagination data retrieval method for external APIs according to claim 1, characterized in that, In step S8, the decision logic of the adaptive strategy engine includes a concurrency adjustment rule, which is specifically as follows: When the following three conditions are met simultaneously: the average response time of the most recent N requests is lower than the first latency threshold T_low, the error rate of the most recent N requests is lower than the first error rate threshold E_low, and the number of recent consecutive successes is greater than or equal to the preset value K_success, the current concurrency level will be increased by one level, up to a maximum of the preset maximum concurrency level C_max. When either the average response time of the most recent N requests exceeds the second latency threshold T_high or the error rate of the most recent N requests exceeds the second error rate threshold E_high, the current concurrency level will be reduced by one level, not lower than the preset minimum concurrency level C_min.

6. The adaptive pagination data retrieval method for external APIs according to claim 1, characterized in that, In step S8, the decision logic of the adaptive strategy engine also includes page size adjustment rules, request interval and backoff rules, and retry strategy optimization rules, wherein: The pagination size adjustment rules are as follows: when the interface response performance is good and no errors occur, the pagination size is increased by a preset step size, not exceeding the maximum pagination size supported by the API; when timeouts, excessively long response times, or server errors occur, the pagination size is reduced by a preset ratio. The request interval and backoff rules are as follows: when API rate limiting or the remaining request quota is nearly exhausted, the request interval time is increased exponentially according to the preset linear backoff strategy; when continuous service anomalies or timeouts occur, the request interval is adjusted according to the exponential backoff strategy, and circuit breaker detection is triggered at the same time. The retry strategy optimization rules are as follows: dynamically adjust the number of retries and the retry interval according to the error type; idempotent requests support automatic retries; and non-idempotent requests only retries when the timeout occurs and the data has not actually been sent.

7. The adaptive pagination data retrieval method for external APIs according to claim 1, characterized in that, In step S9, the paging adapter executes differentiated paging state update rules based on the paging method type, specifically: Page number pagination: Increment the current page number by 1 and update the cumulative number of data entries retrieved simultaneously; Offset-based pagination: Updates the current offset to the sum of the original offset and the number of records returned this time, and synchronously updates the cumulative number of data records retrieved; Cursor-based pagination: Parse the cursor value of the next page from the response body, update it to the unified internal pagination status, and synchronously update the cumulative number of data records fetched; Meanwhile, a full traversal and parsing of the JSON structure of the response data is performed. For object type nodes, the child nodes corresponding to each field are accessed in turn, the path is updated, and the traversal logic is recursively called. For non-target array nodes, each element in the array is accessed in turn, the path is updated in the form of wildcards or indices, and the traversal continues. Finally, valid business data is extracted and its integrity is verified.

8. The adaptive pagination data retrieval method for external APIs according to claim 1, characterized in that, In step S10, the termination condition for pagination fetching includes any one of the following: The external API returned information that clearly indicated no more data, including returning an empty array, hasNext=false, and no next page cursor; The total number of records retrieved has reached the maximum number of records retrieved as set in the query task context. The cumulative time consumed by the task has reached the maximum time threshold set in the query task context. The cumulative number of errors has reached the maximum error threshold set in the query task context.

9. The adaptive pagination data retrieval method for external APIs according to claim 1, characterized in that, The method also includes a smooth degradation and fault tolerance fallback mechanism, specifically: when a serious external API anomaly or an adaptive strategy engine failure is detected, it automatically switches to a fixed strategy security mode and completes data retrieval based on preset conservative paging parameters; at the same time, a task breakpoint resume mechanism is set up so that when a task is unexpectedly interrupted, the retrieval task continues to be executed from the interrupted position based on a unified internal paging state, without having to start from the beginning.

10. An electronic device, characterized in that, The electronic device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 9.

Images