Data paging query method, device and equipment
By maintaining the pagination table information in real time and pre-obtaining the row key list of adjacent page numbers, the performance bottleneck and user experience issues of data pagination queries under massive data are resolved, and a fast and convenient pagination query service is achieved.
Patent Information
- Application Number
- CN202511815162.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-04
- Publication Date
- 2026-02-27
AI Technical Summary
In scenarios involving massive amounts of data, existing data pagination query methods suffer from problems such as long query times, unresponsive user interfaces, and page crashes. In particular, when processing massive amounts of data in relational databases, performance bottlenecks are obvious, and prohibiting deep pagination queries cannot meet users' page-jumping needs.
The pagination table information is maintained in real time, including a list of row keys for adjacent pages next to the previous query page number, and sent to the front end. When a user clicks on any adjacent page number, the server quickly finds all the data based on the pre-obtained list of row keys, enabling fast navigation.
It provides a convenient and efficient pagination query service, greatly improving the user experience and allowing users to quickly jump to the selected page number with a near-instantaneous response.
Smart Images
Figure CN121579544A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] Embodiments of the present application relate to the technical field of computers, and are applicable to the financial field, such as application scenarios of banks and the like, and in particular to a data paging query method, device and equipment. BACKGROUND
[0002] In the era of massive data, more and more data is generated by individuals and enterprises, and when a user wants to view data in a front-end interface, the data is generally displayed in the form of pages.
[0003] When the data set viewed by the user is very large, such as several million or even tens of millions of records, the user will encounter problems such as long query time, non-response of the user interface, page crash and the like when performing paging query, which seriously affects the user experience.
[0004] In related technologies, there are mainly two kinds of optimization methods for deep paging:
[0005] The first optimization method is only applicable to a relational database, and uses a subquery for optimization. The subquery first locates a data range according to an index, and the main query performs query according to the data range condition obtained by the subquery, so that the data set to be processed by the main query is greatly reduced, avoiding scanning of the entire data set.
[0006] The second optimization method is to prohibit deep paging, that is, only the previous page and the next page are provided for query, and page jumping is not allowed.
[0007] However, the first optimization method is only applicable to a relational database, and the data capacity of the relational database is limited, which is not suitable for massive data processing, and there is a significant performance bottleneck when processing massive data.
[0008] The second optimization method directly avoids deep paging query from the aspects of business and technology, and only provides the previous page and the next page for page turning query, which is effective in some scenarios. However, in some specific scenarios, the demand for page jumping often exists. SUMMARY
[0009] To solve the problems in the prior art, embodiments of the present application provide a data paging query method, device and equipment, which maintain page table information in real time, the page table information includes a row key list of a first number of adjacent page codes adjacent to a previous query page code of preloading, and adjacent page codes of a target page code maintained in the page table information are sent to the front end as page codes allowed for paging selection by the user.
[0010] Since the row key list of adjacent pages has been obtained in advance, when a user clicks on any adjacent page, the server can quickly find all the data based on the row key list, enabling a rapid jump to the selected page and obtaining a near-instantaneous response. This provides users with a convenient and efficient pagination query service, greatly improving the user experience.
[0011] The specific technical solutions of this application embodiment are as follows:
[0012] On the one hand, embodiments of this application provide a data pagination query method, including:
[0013] A data pagination query method, the method comprising:
[0014] In response to a query request for a target page number, the row key list of the target page number is obtained based on the pagination table information, and the full data corresponding to each row key in the row key list is queried; wherein, the pagination table information includes a row key list of a first number of adjacent page numbers adjacent to the previously queried page number;
[0015] Obtain the row key list of adjacent page numbers of the target page number, update the pagination table information based on the obtained row key list, and store the updated pagination table information;
[0016] The front end is fed back the full data and the queryable page number, where the queryable page number is the adjacent page number of the target page number in the updated pagination table information, so that the front end can display multiple full data entries of the target page number and pagination information, including the queryable page number.
[0017] Furthermore, the method also includes:
[0018] In response to a data query request carrying query conditions, the total number of data that meet the query conditions is retrieved from the first database;
[0019] Calculate the total number of pages based on the total number of data points;
[0020] The total number of pages is sent to the front end so that the front end displays pagination information containing the total number of pages.
[0021] Furthermore, the method also includes:
[0022] Based on the first database, determine a list of row keys for the homepage page numbers that meet the query conditions;
[0023] Based on the row key list of the homepage page numbers, query the full data of the homepage from the second database;
[0024] The full data of the homepage is fed back to the front end so that the front end can display the full data of the homepage at the homepage page number.
[0025] Further, obtaining the row key list of adjacent page numbers of the target page number includes:
[0026] Determine the target row key in the row key list of the target page number from the previous query;
[0027] Based on the target row key, a sequential query is performed according to the secondary index of the second database to obtain a list of row keys for adjacent page numbers of the target page number; the secondary index is adapted to the query conditions.
[0028] Furthermore, the fact that the secondary index is adapted to the query conditions indicates that the index fields of the secondary index are a subset of the set of condition fields in the query conditions.
[0029] Further, the step of sequentially querying according to the secondary index of the second database to obtain the row key list of adjacent page numbers of the target page number includes:
[0030] A candidate row key list is obtained by sequentially querying the secondary index of the second database.
[0031] Query the full candidate data corresponding to each candidate row key in the candidate row key list from the second database;
[0032] By filtering out the candidate row keys corresponding to the full data that do not meet the query conditions from the candidate row key list, a row key list of adjacent page numbers of the target page number is obtained.
[0033] Further, updating the pagination table information based on the acquired row key list includes:
[0034] In response to the query frequency of the adjacent page numbers in the pagination table information satisfying the frequent query condition, a row key list of a second number of adjacent page numbers adjacent to the current page number is obtained, and the pagination table information is updated based on the obtained row key list; the second number is greater than the first number.
[0035] Furthermore, the storage of the updated pagination table information includes:
[0036] The updated pagination table information is stored in the backend Redis database;
[0037] The step of responding to a query request for a target page number by obtaining the row key list of the target page number based on the pagination table information includes:
[0038] In response to a query request for a target page number, the Redis database is accessed, and a list of row keys for the target page number contained in the current pagination table information is obtained from the Redis database.
[0039] Furthermore, the storage of the updated pagination table information includes:
[0040] The updated pagination table information is sent to the front end so that the front end stores the updated pagination table information in the front end storage space. When the front end responds to a query request for the target page number, it accesses the front end storage space and retrieves the row key list of the target page number contained in the current pagination table information from the front end storage space.
[0041] Furthermore, the first database is an ElasticSearch database, and the second database is an HBase database.
[0042] On the other hand, embodiments of this application also provide a data pagination query device, the device comprising:
[0043] The query module is used to respond to a query request for a target page number, obtain the row key list of the target page number based on the pagination table information, and query the full data corresponding to each row key in the row key list; wherein, the pagination table information includes the row key list of a first number of adjacent page numbers adjacent to the previous queried page number;
[0044] The acquisition module is used to acquire a list of row keys of adjacent page numbers of the target page number, update the pagination table information based on the acquired list of row keys, and store the updated pagination table information.
[0045] The feedback module is used to feed back the full data and the queryable page number to the front end. The queryable page number is the page number adjacent to the target page number in the updated pagination table information, so that the front end can display multiple full data entries of the target page number and pagination information, including the queryable page number.
[0046] On the other hand, embodiments of this application also provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described method.
[0047] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.
[0048] Finally, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method.
[0049] Using the embodiments of this application, pagination table information is maintained in real time. The pagination table information includes a row key list of a first number of neighboring page numbers adjacent to the previous queried page number. The neighboring page numbers of the target page number maintained in the pagination table information are sent to the front end as page numbers that allow users to select pages.
[0050] Since the row key list of adjacent pages has been obtained in advance, when a user clicks on any adjacent page, the server can quickly find all the data based on the row key list, enabling a rapid jump to the selected page and obtaining a near-instantaneous response. This provides users with a convenient and efficient pagination query service, greatly improving the user experience.
[0051] To make the above and other objects, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0053] Figure 1 The diagram shown is a flowchart of a data pagination query method according to an embodiment of this application.
[0054] Figure 2 The diagram shown is a flowchart illustrating a process by which the server determines pagination information based on query conditions in an embodiment of this application.
[0055] Figure 3 The diagram shown is a flowchart illustrating a method for querying the full data of the homepage based on a first database in an embodiment of this application.
[0056] Figure 4 The diagram shown is a flowchart illustrating one method for obtaining a list of row keys for adjacent page numbers of a target page in an embodiment of this application.
[0057] Figure 5 The diagram shown is another flowchart illustrating the process of obtaining the row key list of adjacent page numbers of the target page number in an embodiment of this application.
[0058] Figure 6 The diagram shown is a structural schematic of a data pagination query device according to an embodiment of this application;
[0059] Figure 7 The diagram shown is a structural schematic of a computer device in an embodiment of this application.
[0060] Explanation of symbols in the attached drawings:
[0061] 601. Query module;
[0062] 602. Acquisition Module;
[0063] 603. Feedback Module;
[0064] 702. Computer equipment;
[0065] 704, Processor;
[0066] 706. Memory;
[0067] 708. Drive mechanism;
[0068] 710. Input / Output Module;
[0069] 712. Input devices;
[0070] 714. Output devices;
[0071] 716. Presentation equipment;
[0072] 718. Graphical User Interface;
[0073] 720. Network interface;
[0074] 722. Communication link;
[0075] 724. Communication bus. Detailed Implementation
[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0077] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.
[0078] This application provides the method operation steps as described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.
[0079] It should be noted that the acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.
[0080] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.
[0081] To address the problems existing in the prior art, embodiments of this application provide a data pagination query method, such as... Figure 1 As shown, the data pagination query method may include:
[0082] S101: In response to a query request for a target page number, obtain the row key list of the target page number based on the pagination table information, and query the full data corresponding to each row key in the row key list; wherein, the pagination table information includes a row key list of a first number of neighboring page numbers adjacent to the previously queried page number.
[0083] The data pagination query method provided in this application can be applied to devices that communicate with client devices, such as servers, which can also be understood as backend devices. The client device is a device running frontend software, used to receive query requests initiated by users, generate pages, and display the queried data to the user through the pages.
[0084] For example, the client device receives a query request, forwards the query request to the server, the server performs a data query based on the query request, and then sends the relevant data to the front end.
[0085] In some embodiments of this application, the data to be queried can be stored in a database, and the server accesses the database to perform the data query. For example, each full record in the database is stored in rows, corresponding to a rowkey. When performing a paginated query, the list of rowkeys corresponding to the target page number is determined, and all full records for the target page number can be found.
[0086] S102: Obtain the row key list of adjacent page numbers of the target page number, update the pagination table information based on the obtained row key list, and store the updated pagination table information.
[0087] In this embodiment, a pagination table is maintained and updated in real time. The pagination table includes row key information for a first number of neighboring pages adjacent to the previous queried page number.
[0088] That is, each time a data query is performed for a specific page number, not only is the full data corresponding to that specific page number sent to the client device, but also the row key list of the first number of adjacent pages adjacent to that specific page number is obtained.
[0089] For example, adjacent pages can be adjacent in descending order of page number, adjacent pages can be adjacent in ascending order of page number, or both of the above can be included.
[0090] Taking a first quantity of 5, and page numbers arranged in ascending order, as an example, if a user initiates a query request for page 10, the server quickly retrieves all data corresponding to page 10 from the database based on the row key list for page 10 contained in the current pagination table information. Furthermore, the server obtains the row key list for each page from page 11 to page 15, updates the pagination table information, and stores the updated pagination table information.
[0091] S103: Feed back the full data and the queryable page number to the front end. The queryable page number is the page number adjacent to the target page number in the updated pagination table information, so that the front end can display multiple full data entries of the target page number and pagination information, including the queryable page number.
[0092] In this embodiment, the full data corresponding to the target page number is fed back to the front end, so that the front end displays multiple full data entries for the target page number.
[0093] In addition, by sending the searchable page numbers (i.e., the adjacent page numbers in the updated pagination table information) to the front end, the front end can display the adjacent page numbers. Continuing the previous example, the searchable page numbers are "pages 11-15". The row key list of these page numbers has been pre-obtained, meaning that the aforementioned adjacent page numbers are the page numbers that the user can directly select next time, and the server can quickly retrieve the corresponding full data. The user can quickly jump to any adjacent page number.
[0094] As can be seen, in this embodiment, the pagination table information is maintained in real time. The pagination table information contains a row key list of a first number of neighboring page numbers adjacent to the previous queried page number. The neighboring page numbers of the target page number maintained in the pagination table information are sent to the front end as the page numbers that allow users to select pages.
[0095] Since the row key list of adjacent pages has been obtained in advance, when a user clicks on any adjacent page, the server can quickly find all the data based on the row key list, enabling a rapid jump to the selected page and obtaining a near-instantaneous response. This provides users with a convenient and efficient pagination query service, greatly improving the user experience.
[0096] In some embodiments of this application, Figure 1 The query steps shown are for non-first queries initiated by users. For first queries initiated by users, query conditions are usually included, and the server can query the total number of data that meet the conditions.
[0097] See Figure 2 This is a flowchart illustrating how a server determines pagination information based on query criteria, including the following steps:
[0098] S201: In response to a data query request carrying query conditions, query the total number of data that meet the query conditions from the first database.
[0099] In this embodiment, the first database does not need to store all the data, but only the index information of the data, such as the row key information of the data and the information of fields related to the query conditions. Among them, the row key information represents the row key corresponding to the data when the data is stored in another database.
[0100] For example, query conditions may include user ID, date, etc., so the first database can store the user ID, date, row key information, etc. corresponding to each piece of data.
[0101] Specifically, in some embodiments of this application, an architecture supporting massive data storage and querying can be constructed by combining different databases based on their characteristics.
[0102] For example, the first database stores the row keys of the data and information about the query fields related to the query conditions; the second database stores the full data, which is queried using the row keys.
[0103] Accordingly, the first database should be one with strong global search capabilities and support for massive data storage; the second database should be one that supports real-time reading and writing of massive amounts of data and strong consistency.
[0104] For example, this embodiment uses Elasticsearch as the first database and HBase as the second database. HBase is a distributed storage system that provides high performance, massive storage, scalability, real-time read / write capabilities, and data retrieval based solely on the range of rowkey and primary key. Elasticsearch (ES) is a distributed, highly scalable, and real-time search and data analysis engine developed using Java. The choice of the first and second databases is merely an example, and this embodiment does not limit the scope of the application.
[0105] For the first query request initiated by a user, the system retrieves the total number of matching data points from the first database based on the query criteria. The first database possesses strong retrieval capabilities, thus enabling it to quickly retrieve the total number of data points matching the query criteria.
[0106] S202: Calculate the total number of pages based on the total number of data.
[0107] Since the number of data entries per page can be preset, the server can calculate the total number of pages based on the total number of data entries.
[0108] S203: Send the total number of pages to the front end so that the front end displays pagination information containing the total number of pages.
[0109] In this embodiment, the pagination information includes the total number of pages containing data that meets the query conditions. This makes it easier for users to view the total number of data entries and pages, allowing them to intuitively understand the size of the data and improving the user experience. It also provides a basis for users to choose to view data in reverse order.
[0110] In some embodiments of this application, for the first query initiated by a user, the list of row keys corresponding to the homepage data can be queried based on the first database. See [link to specific examples]. Figure 3 This is a flowchart illustrating the process of querying the full data of the homepage based on the first database, including the following steps:
[0111] S301: Determine a list of row keys for the homepage page numbers that meet the query conditions based on the first database.
[0112] In this embodiment, querying the row key list of all data that meets the query conditions based on the first database would require a lot of time and computing resources. However, retrieving the row key list of only the homepage page number data that meets the query conditions based on the first database would only require very little time and computing resources.
[0113] Therefore, as one possible implementation of this application, for the first query initiated by the user, the row key list of the homepage page number can be obtained based on the first database.
[0114] S302: Retrieve all data from the first page from the second database using the row key list based on the first page number.
[0115] As mentioned above, the first database does not store all the data, but rather the row key information corresponding to the all data. After determining the row key list for the homepage page number, the server then calls the second database to query the all data corresponding to each row key. The collection of these all data is the homepage all data.
[0116] S303: Feed back the full data of the homepage to the front end so that the front end can display the full data of the homepage at the homepage page number.
[0117] In this embodiment, for the initial query request carrying query conditions, the first database can be invoked to retrieve the row key information of the homepage page number that meets the conditions, and then the second database can be invoked to retrieve the full homepage data, which is then sent to the front end. By making reasonable use of the characteristics of different databases and combining the first and second databases to query homepage data, the response speed of providing the user with the full homepage data is improved.
[0118] In some embodiments of this application, after a user initiates an initial query request, the front end displays the full data of the homepage to the user. The user usually wants to view data for other pages, and can then initiate a paginated query request for the target page by clicking "Queryable Pages".
[0119] Understandably, if the row key information were retrieved from the primary database for every subsequent pagination query request from a user, the primary database would need to store the row key information for every row of data that initially matches the query criteria. In scenarios with massive amounts of data, this would consume a significant amount of resources and significantly reduce response time. Alternatively, for each pagination query request, complex condition filtering would need to be re-executed in the primary database, which would also result in very slow response times.
[0120] Based on the above considerations, in this embodiment, the row key information of data for other page numbers can be found from the second database using the secondary index built into the second database.
[0121] See Figure 4 A flowchart illustrating the process of obtaining a list of row keys for adjacent pages of a target page includes the following steps:
[0122] S401: Determine the target row key in the row key list for the target page number.
[0123] S402: Based on the target row key, perform a sequential query according to the secondary index of the second database to obtain a list of row keys for adjacent page numbers of the target page number; the secondary index is adapted to the query conditions.
[0124] In this embodiment, the index fields included in the secondary index can be configured to match the query conditions in the user's query request as closely as possible.
[0125] For example, indexed fields may include fields such as "User ID", "Date", "Department", and "Years of Service". Users can use any combination of these fields as query conditions to retrieve row key information in the second database based on the corresponding secondary index.
[0126] In addition, in a second database, such as HBase, the data is arranged naturally in a specific order, usually based on the time the data was generated.
[0127] For example, using "User ID" and "Date" as index fields for a secondary index, with the specific content of the fields being "User001" and "Year xx Month", a query in the HBase database based on this secondary index will retrieve the data generated by "User001" in "Year xx Month". This data is sorted according to the time of generation, and each piece of data corresponds to a row key, which is also arranged in order.
[0128] Therefore, in this embodiment, when the query conditions are determined, a secondary index that matches the query conditions is selected, and the row key list of adjacent page numbers of the target page number is queried in the second database.
[0129] Based on the target row key in the row key list of the target page number, a sequential query is performed using the secondary index of the second database to obtain the row key list of neighboring page numbers. The target row key can be the last row key in the target page number's row key list, thus ensuring that the row key list of the neighboring page numbers queried subsequently is adjacent to the target page number's row key list.
[0130] In some embodiments of this application, for adjacent page numbers, since the row keys are arranged sequentially, the list of row keys queried may only contain the start rowkey and end rowkey of the adjacent page number.
[0131] In some embodiments of this application, the page number to be pre-recorded and the corresponding row key list can be determined based on the position of the target page number selected by the user this time in the adjacent page number of the previously selected target page number.
[0132] For a pre-loaded list of page keys, if the selected target page number is still within the range of adjacent pages, it is not necessary to retrieve the list of page keys again. Instead, based on the last page key obtained in the previous query, the list of page keys for XY pages should be retrieved sequentially. Here, X is the first quantity, and Y represents the number of pages whose adjacent pages are still within the range of adjacent pages of the new target page.
[0133] For example, if the user's last selected target page number was "page 3", and the first number is 5, the adjacent page numbers included in the current pagination table information include "pages 4-8". If the user selects "page 7" this time, since "page 8" is still within the adjacent page number range of the new target page number, it is not necessary to retrieve the row key list for that page number again.
[0134] Correspondingly, the preloading process can be based on the last row key of "page 8", and then perform a sequential query according to the secondary index of the HBase database, with the number of pages queried being 5-1=4.
[0135] In this embodiment, based on HBase's secondary index, the row key list of neighboring pages of the current target page is preloaded. Since HBase is suitable for massive data storage and fast scanning, it can quickly respond to retrievals of such ordered key values without needing to re-execute complex condition filtering for each pagination query. Furthermore, each preloading only requires scanning a small amount of row key information as needed, enabling rapid preloading of the row key list and subsequent updating of the pagination table information.
[0136] In some embodiments of this application, the matching of a secondary index with query conditions indicates that the index field of the secondary index is a subset of the set of condition fields in the query conditions.
[0137] Specifically, if the secondary index of the second database does not completely correspond to the query conditions, for example, if the secondary index does not support some fields, then the indexed fields of the secondary index are only a subset of the set of condition fields in the query conditions. In this case, the query can still be performed based on the secondary index, but the queried data needs to be verified.
[0138] For example, see Figure 5 Another flowchart for obtaining the row key list of adjacent pages to the target page number includes the following steps:
[0139] S501: Perform a sequential query based on the secondary index of the second database to obtain a list of candidate row keys.
[0140] For a detailed description of this step, please refer to S401, and will not be repeated here.
[0141] S502: Query the candidate full data corresponding to each candidate row key in the candidate row key list from the second database.
[0142] When the indexed fields of a secondary index are only a subset of the set of condition fields in the query conditions, the full data corresponding to the candidate row key list queried based on the secondary index may not satisfy the query conditions.
[0143] Therefore, the full data corresponding to the candidate row key list is obtained as candidate full data, and further verification is performed to determine whether it meets the query conditions.
[0144] S503: Filter out the candidate row keys corresponding to the candidate full data that do not meet the query conditions from the candidate row key list to obtain the row key list of the neighboring pages of the target page number.
[0145] For each candidate full dataset, determine whether it meets the query criteria and filter out the candidate row keys corresponding to the candidate full dataset that do not meet the query criteria.
[0146] Since some candidate row keys have been deleted, the list of row keys obtained may not be sufficient to reach the number of row keys corresponding to the adjacent page numbers. In this case, we can return to the step of sequential querying based on the secondary index of the second database to further scan and obtain candidate row keys, and then filter the newly obtained candidate row keys until the number of row keys in the list of obtained row keys meets the requirements.
[0147] In this embodiment, since some row keys are deleted, the method of storing the start row key and the end row key is not suitable. It is necessary to store the undeleted row keys corresponding to each adjacent page number separately.
[0148] In this embodiment, when the index fields of the secondary index are only a subset of the condition fields in the query conditions, although it is necessary to query the full candidate data corresponding to the candidate row key list from the first database and determine whether the query conditions are met, the computational cost and time consumption for performing condition judgments on a small number of row key lists are also small, since the row key list preloaded each time is small. Compared to the method of re-executing complex condition filtering of all data based on the first database for each pagination query, the speed of preloading row keys is still significantly improved. Furthermore, it reduces the requirements for the secondary index; even when the secondary index of the second database does not support some fields, this scheme can still be used for row key preloading.
[0149] In some embodiments of this application, updating the pagination table information based on the obtained row key list may specifically include: in response to the query frequency of the adjacent page numbers in the pagination table information satisfying the frequent query condition, obtaining a row key list of a second number of adjacent page numbers adjacent to the current page number, and updating the pagination table information based on the obtained row key list; the second number is greater than the first number.
[0150] For example, the front end can count the frequency of users clicking on adjacent page numbers in the pagination table information and send the statistical results to the back end.
[0151] As described above, each time a user clicks on a nearby page, the backend updates the pagination table information and generates a new nearby page. The nearby page can also be understood as the currently searchable page.
[0152] If a user frequently clicks on adjacent page numbers, the user's intention may be to quickly browse data further down the page.
[0153] Based on this, if the candidate determines that the frequency of user queries for adjacent page numbers meets the frequent query condition, such as the query frequency exceeding once per second, the number of preloaded page numbers is increased from the first number to the second number.
[0154] This involves obtaining a list of row keys for the second number of adjacent pages to the current page number, and updating the pagination table information based on this list.
[0155] As a result, the number of adjacent page numbers included in the pagination table information also increases accordingly, and the number of searchable page numbers displayed to the user on the front end also increases accordingly, which helps to speed up the user's page navigation and improve the user experience.
[0156] In some embodiments of this application, the updated pagination table information can be stored in a backend Redis database. Accordingly, in response to a query request for a target page number, obtaining the row key list of the target page number based on the pagination table information includes: in response to a query request for the target page number, accessing the Redis database and obtaining the row key list of the target page number contained in the current pagination table information from the Redis database.
[0157] Among them, Redis is a high-performance open-source key-value store system that can quickly respond to server reads. Storing pagination table information in the backend Redis database can improve the response speed of pagination queries.
[0158] In some embodiments of this application, the server can send updated pagination table information to the front end, so that the front end stores the updated pagination table information in its front-end storage space. Thus, when the front end responds to a query request for a target page number, it accesses the front-end storage space and retrieves the row key list of the target page number contained in the current pagination table information.
[0159] In this embodiment, the pagination table information can be directly stored in the front-end storage space. When the front-end receives a query request for the target page number, it can directly access the front-end storage space without having to obtain the row key list of the target page number through the back-end server, which further improves the response speed of pagination queries.
[0160] In some embodiments of this application, for each secondary index of the second database, both an ascending secondary index and a descending secondary index can be set. The ascending secondary index supports sorting by time in ascending order, while the descending secondary index supports sorting by time in descending order.
[0161] For example, assuming the query condition is user ID + date range, the secondary index fields can include user ID and a `data_dt` field, where `data_dt` is an 8-digit date represented as yyyymmdd. For an ascending order secondary index, it's built using user ID + `data_dt`, with the `data_dt_re` field taking the value yyyymmdd; for an inverted order secondary index, it's built using user ID + `data_dt_re`, where the `data_dt_re` field takes the value 99999999-yyyymmdd. Therefore, using an inverted order secondary index for retrieval will prioritize data with larger dates, achieving the effect of reversing the order.
[0162] In this embodiment, when the query conditions meet the reverse order query, the row key list of the neighboring page numbers of the target page number is obtained. Specifically, this may include: obtaining the row key list of the neighboring page numbers of the target page number based on a reverse order secondary index that is compatible with the query conditions.
[0163] For example, query conditions include: querying a user's log data for the past week. When preloading the row key list, the row key list of adjacent page numbers can be found in the second database based on the appropriate inverted secondary index.
[0164] As can be seen, in this embodiment, two sets of secondary indexes, one in ascending order and one in descending order, are set for the same query condition. This allows for efficient range scanning based on the corresponding secondary indexes to quickly obtain the target data, regardless of whether the query is requested to be from the earliest to the latest time or from the latest to the earliest time.
[0165] Furthermore, in second databases, such as HBase, data storage is essentially arranged in ascending lexicographical order according to the rowkey. This embodiment generates a new descending field, data_dt_re, through a mathematical transformation (99999999 - data_dt), and builds a descending index based on this. Since the value of data_dt_re is the opposite of the original date data_dt, when data is stored in ascending order of data_dt_re, its physical arrangement is exactly equivalent to the descending order of the original dates. This design cleverly utilizes HBase's inherent sorting mechanism, achieving native support for descending date queries without introducing complex calculation logic or external sorting tools, ensuring query efficiency and system simplicity.
[0166] Furthermore, in paginated queries, users most frequently access the first and last pages. In this embodiment, the ascending index can quickly locate and return the data of the first page in sequence, while the descending index can equally efficiently locate and return the data of the last page. This effectively solves the technical problem of drastic performance degradation when accessing later pages in traditional deep pagination queries.
[0167] Based on the same inventive concept, embodiments of this application also provide a pagination query device, such as... Figure 6 As shown, the device includes:
[0168] The query module 601 is used to respond to a query request for a target page number, obtain the row key list of the target page number based on the pagination table information, and query the full data corresponding to each row key in the row key list; wherein, the pagination table information includes a row key list of a first number of adjacent page numbers adjacent to the previous queried page number;
[0169] The acquisition module 602 is used to acquire a list of row keys of adjacent page numbers of the target page number, update the pagination table information based on the acquired list of row keys, and store the updated pagination table information.
[0170] Feedback module 603 is used to feed back the full data and the queryable page number to the front end. The queryable page number is the page number adjacent to the target page number in the updated pagination table information, so that the front end can display multiple full data entries of the target page number and pagination information, including the queryable page number.
[0171] Furthermore, the query module 601 is also used for:
[0172] In response to a data query request carrying query conditions, the total number of data that meet the query conditions is retrieved from the first database;
[0173] Calculate the total number of pages based on the total number of data points;
[0174] The total number of pages is sent to the front end so that the front end displays pagination information containing the total number of pages.
[0175] Furthermore, the query module 601 is also used for:
[0176] Based on the first database, determine a list of row keys for the homepage page numbers that meet the query conditions;
[0177] Based on the row key list of the homepage page numbers, query the full data of the homepage from the second database;
[0178] The full data of the homepage is fed back to the front end so that the front end can display the full data of the homepage at the homepage page number.
[0179] Furthermore, the acquisition module 602 is specifically used for:
[0180] Determine the target row key in the row key list of the target page number from the previous query;
[0181] Based on the target row key, a sequential query is performed according to the secondary index of the second database to obtain a list of row keys for adjacent page numbers of the target page number; the secondary index is adapted to the query conditions.
[0182] Furthermore, the fact that the secondary index is adapted to the query conditions indicates that the index fields of the secondary index are a subset of the set of condition fields in the query conditions.
[0183] Furthermore, the acquisition module 602 is specifically used for:
[0184] A candidate row key list is obtained by sequentially querying the secondary index of the second database.
[0185] Query the full candidate data corresponding to each candidate row key in the candidate row key list from the second database;
[0186] By filtering out the candidate row keys corresponding to the full data that do not meet the query conditions from the candidate row key list, a row key list of adjacent page numbers of the target page number is obtained.
[0187] Furthermore, the acquisition module 602 is specifically used for:
[0188] In response to the query frequency of the adjacent page numbers in the pagination table information satisfying the frequent query condition, a row key list of a second number of adjacent page numbers adjacent to the current page number is obtained, and the pagination table information is updated based on the obtained row key list; the second number is greater than the first number.
[0189] Furthermore, it also includes a storage module for:
[0190] The updated pagination table information is stored in the backend Redis database;
[0191] The step of responding to a query request for a target page number by obtaining the row key list of the target page number based on the pagination table information includes:
[0192] In response to a query request for a target page number, the Redis database is accessed, and a list of row keys for the target page number contained in the current pagination table information is obtained from the Redis database.
[0193] Furthermore, it also includes a sending module for:
[0194] The updated pagination table information is sent to the front end so that the front end stores the updated pagination table information in the front end storage space. When the front end responds to a query request for the target page number, it accesses the front end storage space and retrieves the row key list of the target page number contained in the current pagination table information from the front end storage space.
[0195] Furthermore, the first database is an ElasticSearch database, and the second database is an HBase database.
[0196] Since the principle of the above-mentioned device in solving the problem is similar to that of the above-mentioned method, the implementation of the above-mentioned device can refer to the implementation of the above-mentioned method, and the repeated parts will not be described again.
[0197] like Figure 7 The diagram shows a schematic of a computer device according to an embodiment of this application. The apparatus in this application can be the computer device in this embodiment, executing the methods described above. The computer device 702 may include one or more processors 704, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 702 may also include any memory 706 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, the memory 706 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 702. In one case, when the processor 704 executes associated instructions stored in any memory or combination of memories, the computer device 702 can perform any operation of the associated instructions. The computer device 702 also includes one or more drive mechanisms 708 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.
[0198] Computer device 702 may also include an input / output module 710 (I / O) for receiving various inputs (via input device 712) and providing various outputs (via output device 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface (GUI) 718. In other embodiments, the input / output module 710 (I / O), input device 712, and output device 714 may be omitted, and the device may function solely as a computer device within a network. Computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.
[0199] Communication link 722 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.
[0200] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method.
[0201] This application also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the steps of the above-described method.
[0202] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0203] It should also be understood that, in the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects have an "or" relationship.
[0204] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0205] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0206] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, apparatuses, or units, or they may be electrical, mechanical, or other forms of connection.
[0207] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.
[0208] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0209] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0210] This application uses specific embodiments to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this application should not be construed as a limitation of this application.
Claims
1. A data pagination query method, characterized in that, The method includes: In response to a query request for a target page number, the row key list of the target page number is obtained based on the pagination table information, and the full data corresponding to each row key in the row key list is queried; wherein, the pagination table information includes a row key list of a first number of adjacent page numbers adjacent to the previously queried page number; Obtain the row key list of adjacent page numbers of the target page number, update the pagination table information based on the obtained row key list, and store the updated pagination table information; The front end is fed back the full data and the queryable page number, where the queryable page number is the adjacent page number of the target page number in the updated pagination table information, so that the front end can display multiple full data entries of the target page number and pagination information, including the queryable page number.
2. The method according to claim 1, characterized in that, The method further includes: In response to a data query request carrying query conditions, the total number of data that meet the query conditions is retrieved from the first database; Calculate the total number of pages based on the total number of data points; The total number of pages is sent to the front end so that the front end displays pagination information containing the total number of pages.
3. The method according to claim 2, characterized in that, The method further includes: Based on the first database, determine a list of row keys for the homepage page numbers that meet the query conditions; Based on the row key list of the homepage page numbers, query the full data of the homepage from the second database; The full data of the homepage is fed back to the front end so that the front end can display the full data of the homepage at the homepage page number.
4. The method according to claim 3, characterized in that, The method of obtaining the row key list of adjacent page numbers of the target page number includes: Determine the target row key in the row key list of the target page number from the previous query; Based on the target row key, a sequential query is performed according to the secondary index of the second database to obtain a list of row keys for adjacent page numbers of the target page number; the secondary index is adapted to the query conditions.
5. The method according to claim 4, characterized in that, The fact that the secondary index is adapted to the query conditions indicates that the index fields of the secondary index are a subset of the set of condition fields in the query conditions.
6. The method according to claim 4 or 5, characterized in that, The step of sequentially querying based on the secondary index of the second database to obtain the row key list of adjacent page numbers of the target page number includes: A candidate row key list is obtained by sequentially querying the secondary index of the second database. Query the full candidate data corresponding to each candidate row key in the candidate row key list from the second database; By filtering out the candidate row keys corresponding to the full data that do not meet the query conditions from the candidate row key list, a row key list of adjacent page numbers of the target page number is obtained.
7. The method according to claim 1, characterized in that, Updating the pagination table information based on the obtained row key list includes: In response to the query frequency of the adjacent page numbers in the pagination table information satisfying the frequent query condition, a row key list of a second number of adjacent page numbers adjacent to the current page number is obtained, and the pagination table information is updated based on the obtained row key list; the second number is greater than the first number.
8. The data pagination query method according to claim 1, characterized in that, The updated pagination table information includes: The updated pagination table information is stored in the backend Redis database; The step of responding to a query request for a target page number by obtaining the row key list of the target page number based on the pagination table information includes: In response to a query request for a target page number, the Redis database is accessed, and a list of row keys for the target page number contained in the current pagination table information is obtained from the Redis database.
9. The data pagination query method according to claim 1, characterized in that, The updated pagination table information includes: The updated pagination table information is sent to the front end so that the front end stores the updated pagination table information in the front end storage space. When the front end responds to a query request for the target page number, it accesses the front end storage space and retrieves the row key list of the target page number contained in the current pagination table information from the front end storage space.
10. The method according to any one of claims 3-9, characterized in that, The first database is an ElasticSearch database, and the second database is an HBase database.
11. A data pagination query device, characterized in that, The device includes: The query module is used to respond to a query request for a target page number, obtain the row key list of the target page number based on the pagination table information, and query the full data corresponding to each row key in the row key list; wherein, the pagination table information includes the row key list of a first number of adjacent page numbers adjacent to the previous queried page number; The acquisition module is used to acquire a list of row keys of adjacent page numbers of the target page number, update the pagination table information based on the acquired list of row keys, and store the updated pagination table information. The feedback module is used to feed back the full data and the queryable page number to the front end. The queryable page number is the page number adjacent to the target page number in the updated pagination table information, so that the front end can display multiple full data entries of the target page number and pagination information, including the queryable page number.
12. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.
14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.