Data paging merging method and device in distributed environment and electronic equipment

Abstract

The invention relates to a data paging merging method and device in a distributed environment and electronic equipment, and belongs to the technical field of distributed data processing.The method comprises the steps that the sorting field value of the last piece of data in the last page of query data is obtained; determining the initial data of the next page of the retrieval node by taking the sorting field value as a screening condition; sorting the initial data according to a sorting field, and determining target data of a retrieval node according to the number of pieces of query data of a next page; and determining the next page of data according to the target data. According to the method and the device, the technical problems of high server pressure, high network bandwidth consumption and low global sorting efficiency caused by the fact that data paging and merging often need to be subjected to full query and sorting in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data processing, and in particular to a method, device and electronic equipment for merging data pages in a distributed environment. Background Art

[0002] In today's big data era, with the explosive growth of data volume and the increasing demand for cross-regional data interaction, the traditional centralized data processing and transmission mode can no longer meet the requirements of efficient, secure and real-time data processing. Specifically, the traditional mobile data method, which is to transmit data from scattered source points to central processing nodes for unified processing, not only faces high data transmission costs, but also increases the risk of data leakage during transmission. At the same time, due to network delays and data synchronization issues, data updates are often delayed, making it difficult to meet users' high requirements for data real-time performance.

[0003] In order to overcome the above challenges, the industry has gradually turned to a distributed data processing model that adopts mobile computing (or edge computing). In this model, data is no longer centrally transmitted to a single processing center, but is allowed to be initially processed and analyzed at the source or close to the source of the data. Data nodes in various places follow a unified data access interface standard to provide efficient data retrieval services. By building a highly integrated and flexible data retrieval platform, the platform can access and integrate data retrieval services from different regions in real time, and realize rapid data collection and intelligent display. This data paging and merging method in a distributed environment not only significantly reduces the cost and risk of data transmission, but also improves the efficiency of data processing by reducing the amount of data transmission. At the same time, since the data is initially processed locally or at the edge node, the impact of network latency on data processing speed is reduced, ensuring the real-time and accuracy of the data. Ultimately, users can obtain a comprehensive, accurate and almost latency-free data view through the platform, providing strong data support for decision-making and business operations.

[0004] Although the data paging and merging method in a distributed environment effectively alleviates the challenges brought by centralized data processing through the mobile computing model, the current mainstream sorting and merging method still exposes many shortcomings in practical applications. Specifically, during paging processing, this method directly sends the page number and page size as query conditions to all data retrieval service nodes, requiring each node to perform a full query and sort from page 1 to the target page. As the paging page number increases, this processing method gradually reveals the following significant problems: 1. Sharp increase in server memory pressure: Since each retrieval service node needs to query and cache all data from page 1 to the target page, the amount of data required to be cached increases linearly as the page number increases, which puts tremendous pressure on the server's memory resources. This not only limits the maximum amount of data that the system can process, but may also cause service interruptions due to memory overflow.

[0005] 2. Huge network bandwidth consumption: As the amount of data returned by the retrieval service node increases, the amount of data transmitted over the network also increases dramatically, placing higher demands on network bandwidth. This not only increases the delay in data transmission, but may also affect the response speed and stability of the overall system due to network congestion.

[0006] 3. Inefficient global sorting: After the results returned by all retrieval nodes are aggregated to the query initiation node, global sorting is required to extract the data of the target page. However, as the amount of data increases, the memory and computing time required for global sorting also increase significantly, causing the sorting process to become a performance bottleneck. This not only prolongs the user's waiting time, but may also affect the processing of other concurrent queries due to resource exhaustion. Summary of the invention

[0007] In view of this, it is necessary to provide a data paging merging method, device and electronic device in a distributed environment to solve the technical problems in the prior art that data paging merging often requires full query and sorting, resulting in high server pressure, high network bandwidth consumption and low global sorting efficiency.

[0008] In order to solve the above problems, the present invention provides a data paging merging method in a distributed environment, comprising: Get the sort field value of the last piece of data in the previous page of query data of the retrieval node; Using the sorting field value as a screening condition, determining the initial data of the next page of the retrieval node; Sort the initial data according to the sorting field, and determine the target data of the retrieval node according to the number of data items to be queried on the next page; The next page of data is determined according to the target data.

[0009] In a possible implementation, obtaining the sort field value of the last piece of data in the previous page of query data of the retrieval node includes: Get the number of queried data on the previous page, and determine the data on the previous page based on the number of queried data on the previous page; According to the number of data items queried on the previous page, obtain the sort field value of the last data item in the previous page of data queried by the retrieval node.

[0010] In a possible implementation, obtaining the sort field value of the last piece of data in the previous page of data queried by the retrieval node according to the number of pieces of data queried on the previous page includes: According to the number of data items queried on the previous page, the search node obtains the data on the previous page; Partially sort the data on the previous page using the sorting field to obtain the sorted data, and obtain the sorting field value of the last record of the sorted data; The sort field value is cached in a cache system.

[0011] In a possible implementation, the sorting field includes at least one of quantity, business-specific field, ID, price, creation time, update time, expiration time, user name, tag or category.

[0012] In a possible implementation, using the sorting field value as a screening condition to determine the initial data of the retrieval node includes: Taking the sorting field value as a screening condition, data in the search node that is greater than or equal to the sorting field value is obtained to determine the initial data of the search node.

[0013] In a possible implementation, determining the next page of data according to the target data includes: Get the target data of the retrieval node; According to the sorting field, the target data is globally sorted to obtain globally sorted data; Compare the globally sorted data with the data on the previous page, remove duplicate data, and obtain deduplicated data; Determine the next page of data based on the number of query data items and deduplication data on the next page.

[0014] In a possible implementation, after obtaining the next page of data, the following is further included: Store the next page of data in the cache system.

[0015] In a second aspect, the present invention further provides a data paging merging device in a distributed environment, comprising: A sorting field value determination module is used to obtain the sorting field value of the last piece of data in the previous page of query data of the retrieval node; An initial data determination module, used to determine the initial data of the next page of the retrieval node based on the sort field value as a screening condition; A target data determination module is used to sort the initial data according to the sorting field and determine the target data of the retrieval node according to the number of query data on the next page; The next page determination module is used to determine the next page data according to the target data.

[0016] In a third aspect, the present invention further provides an electronic device, comprising: a processor and a memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the data page merging method in the distributed environment are implemented.

[0017] In a fourth aspect, the present invention further provides a computer-readable storage medium storing one or more programs, wherein the one or more programs can be executed by one or more processors to implement the steps in the data paging merging method in a distributed environment as described above.

[0018] The beneficial effects of the present invention are as follows: the sorting field value of the last data in the previous page of queried data is used as a screening condition, and the scope of the next page of data retrieval is dynamically adjusted according to the sorting field, which significantly reduces the data transmission volume and server memory consumption during the data paging and merging process in a distributed environment, and improves the efficiency of global sorting, thereby achieving faster query response speed and higher system performance. At the same time, after local sorting with the same sorting field, the data returned by the retrieval node to the initiating node is determined, avoiding the problem of data omission after screening. Furthermore, this solution is not only suitable for scenarios such as paging query of sub-libraries and sub-tables, paging merging of cross-regional data center services, and paging merging of cross-data services, but also has no specific requirements for data format and order, showing its wide applicability and significant practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A method flow chart of an embodiment of a method for merging data pages in a distributed environment provided by the present invention; Figure 2 A schematic diagram of an embodiment of paging merging in a data paging merging method under a distributed environment provided by the present invention; Figure 3 A schematic diagram of another embodiment of paging merging in the data paging merging method under a distributed environment provided by the present invention; Figure 4 A flowchart of an embodiment of step S104 in the data paging merging method in a distributed environment provided by the present invention; Figure 5 It is a schematic diagram of an embodiment of a data page merging device in a distributed environment provided by the present invention; Figure 6 It is a schematic diagram of the operating environment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0021] In a distributed environment, the traditional paging method is usually: to obtain the first page (1 to 10 items) of data, you need to request all search service nodes A and B at the same time. Search service nodes A and B obtain the first page (1 to 10 items) of data according to the search conditions. The initiating node F gets the data returned by the search nodes A and B, sorts the 20 items, takes 1 to 10 of them, and returns these 10 items as the first page of data to the user. When querying the second page, it is a little different. It is not directly querying the second page (11 to 20 items) of the search nodes A and B. Because the data on the first page is obtained after the data is globally sorted on the initiating node F, directly obtaining the second page (11 to 20 items) of the search nodes A and B will miss the 10 items that were not obtained on the previous page. At this time, the traditional paging method is to obtain 1 to 20 data items on the A and B search nodes, and then initiate node F to globally sort these 40 data items, and then obtain the 11th to 20th data items as the second page of data and return them to the user.

[0022] Obviously, in a distributed system, a significant drawback of the traditional paging mechanism is that as the page number increases, the amount of data that each retrieval node needs to pull increases dramatically, and the amount of data that needs to be globally sorted on the initiating node also increases significantly. This trend not only increases the consumption of network bandwidth, but also greatly increases the computing burden of the server, posing a severe challenge to system performance.

[0023] A specific embodiment of the present invention discloses a method for merging data pages in a distributed environment. Figure 1 ,include: S101, obtaining the sorting field value of the last piece of data in the previous page of query data of the retrieval node; S102, using the sorting field value as a screening condition, determining the initial data of the next page of the retrieval node; S103, sorting the initial data according to the sorting field, and determining the target data of the retrieval node according to the number of query data on the next page; S104: Determine the next page of data according to the target data.

[0024] In this embodiment, the sorting field value of the last data in the previous page of queried data is used as the screening condition, and the scope of the next page of data retrieval is dynamically adjusted according to the sorting field, which significantly reduces the data transmission volume and server memory consumption during the data paging and merging process in a distributed environment, improves the efficiency of global sorting, and thus achieves faster query response speed and higher system performance. At the same time, after local sorting with the same sorting field, the data returned by the retrieval node to the initiating node is determined, avoiding the problem of data omission after screening. Furthermore, this solution is not only suitable for scenarios such as paging query of sub-libraries and sub-tables, paging merging of cross-regional data center services, and paging merging of cross-data services, but also has no specific requirements for data format and order, showing its wide applicability and significant practical value.

[0025] It should be noted that the query data on the previous page is adjacent to the query data on the current page. The sorting field value of the last data on the previous page is used as the filter condition for the next page, thereby reducing the amount of data to be retrieved on the next page, reducing unnecessary data transmission and sorting operations, reducing the pressure on server memory and network bandwidth, and improving query retrieval efficiency.

[0026] In a specific embodiment of the present invention, the method of obtaining the first page of data is consistent with the traditional paging method, such as Figure 2 As shown in the figure, to obtain the first page (1 to 10 items) of data, it is necessary to request all search service nodes A and B at the same time. Search service nodes A and B obtain the first page (1 to 10 items) of data according to the search conditions. The initiating node F obtains the data returned by the search nodes A and B, sorts the 20 items, takes 1 to 10 of them, and returns these 10 items as the first page of data to the user.

[0027] Different from traditional paging, after the initiating node returns the data, it is necessary to record the value of the sorting field (such as creation time) of the last record in the 1st to 10th data queried by the retrieval nodes A and B.

[0028] It should be noted that the sorting field includes at least one of quantity, business-specific field, ID, price, creation time, update time, expiration time, user name, tag or category. The sorting field value is the value corresponding to the sorting field.

[0029] In some embodiments, the determining the initial data of the retrieval node using the sort field value as a screening condition includes: Taking the sorting field value as a screening condition, data in the search node that is greater than or equal to the sorting field value is obtained to determine the initial data of the search node.

[0030] In this embodiment, if Figure 3As shown in the figure, the sorting field (such as creation time) is used as the filtering condition, and the sorting field value of the last record in the previous page of data of the retrieval node is used as the filtering value (such as 2024-03-05 17:05:03). This method can exclude the data that has been selected in the previous page. As the page number increases, the paging performance is greatly improved compared to the traditional paging method.

[0031] In some embodiments, the next page of data is determined based on the target data. Figure 4 ,include: S401, obtaining target data of a search node; S402, globally sorting the target data according to the sorting field to obtain globally sorted data; S403, comparing the globally sorted data with the data on the previous page, removing duplicate data, and obtaining deduplicated data; S404: Determine the next page of data based on the number of query data items and deduplicated data.

[0032] In a specific embodiment, it is assumed that there is data to be retrieved on the retrieval node A as shown in Table 1: Table 1: Data to be retrieved for retrieval node A

[0033] The data to be retrieved on the retrieval node B is shown in Table 2: Table 2: Data to be retrieved for retrieval node B

[0034] S01: Initiating node, initiating paging query (N=1) The user requests data and sends the request to the initiating node F, which forwards the query request to the retrieval nodes A and B.

[0035] S02: Sorting query is performed inside all search nodes The search nodes A and B are first sorted locally according to the field ID, and then the Nth (N=1) page of data is obtained. The query sorting results are shown in Table 3: Table 3: Query ranking results for retrieving nodes A and B

[0036] S03: The initiating node collects the data returned by all the retrieval nodes The initiating node F collects and retrieves the data returned by nodes A and B.

[0037] S04: The initiating node aggregates and sorts the data The initiating node F performs global sorting of the data by field ID. The globally sorted data is shown in Table 4: Table 4: Global sorting results

[0038] S05: Take N-1 pages of data from the Redis cache for deduplication Get the previous page of data from the Redis cache to deduplicate. Currently, this is page 1 of data, so this step is skipped.

[0039] S06: Merge and remove duplicate data After deduplication, the data is reduced from 20 to 18, as shown in Table 5: Table 5: Deduplication and merging results

[0040] S07: Get the Nth page of data and return it to the user Take out the Nth (N=1) page of data and return it to the user.

[0041] S08: Cache the Nth page of result data to Redis Cache the result data of the previous step into Redis, using the user ID + query condition as the key and the JSON representation of the result data as the value. For example: User001_:[{"ID": "1", "Name": "Beijing"},{"ID": "2","Name": "Tianjin"}......] Note: In this example, the query condition is empty, so the Key is User001_ S09: Cache the last piece of data in the result data to Redis Get the sorting field (ID) value of the last data in the result page and cache it in Redis.

[0042] The result page data is shown in Table 6: Table 6: Results page data table

[0043] So the data to be cached is {ID = 10, Name = Fujian}.

[0044] Cache the record in Redis. Use user ID + ORDER + page number + query condition as the key. So the ID data stored in Redis is: User001ORDER_1: 10 S10: Initiating node, initiating paging query (page N, N>1) When the user requests the second page of data, N = 2.

[0045] S11: Get the last piece of data in the N-1 pages of data for each query node according to the query conditions You need to obtain the sort field value of the last data item in the query result of the previous page of the retrieval node. Get it from the Redis cache based on the key value. In this example, the value obtained is 10.

[0046] User001ORDER_1: 10 S12: Add the last piece of data on page N-1 as a filter condition and initiate a paging data query to all search nodes At this time, the value of the sorting field (ID) obtained in the previous step is used as a screening condition to initiate a paging data query to all search nodes. At this time, the process enters step S02 again.

[0047] S02: Sorting query is performed inside all search nodes Note: The difference between the method shown in the present invention and the traditional method is that the sorting field is used as a screening condition to narrow the data retrieval scope and avoid excessive memory, network and computing overhead during deep paging.

[0048] On the retrieval nodes A and B, local sorting is first performed according to the field ID, and then the data is obtained.

[0049] At this time, the pseudo code for retrieving node A is: select * from A where ID>= 10 order by ID limit 10 offset 0; The filtered data is shown in Table 7: Table 7: Search results for node A

[0050] The pseudo code for retrieving node B at this time is: select * from B where ID>= 10 order by ID limit 10 offset 0; The filtered data are as follows: Table 8: Search Node B Screening Results XXXXX

[0051] S03: The initiating node collects the data returned by all the retrieval nodes The initiating node F collects and retrieves the data returned by nodes A and B.

[0052] S04: The initiating node aggregates and sorts the data The initiating node F performs global sorting of the data by field ID. The globally sorted data is as follows: Table 9: Global sorting results

[0053] S05: Take N-1 pages of data from the Redis cache for deduplication Take the previous page of data cached in step S08 from the Redis cache User001_:[{"ID": "1", "Name": "Beijing"},{"ID": "2", "Name": "Tianjin"}......] The specific data are as follows: Table 10: Previous page data

[0054] Remove the data from the previous page in the current page. The data to be removed is {ID: 10, Name: Fujian}. The data after removal is as follows: Table 11: Deduplication results

[0055] S06: Merge and remove duplicate data After deduplication, the data is reduced from 19 to 16.

[0056] S07: Get the Nth page of data and return it to the user Take out the Nth (N=1) page of data and return it to the user.

[0057] To query the next page, continue to repeat step S10 Based on the above-mentioned data paging merging method in a distributed environment, the embodiment of the present invention also provides a data paging merging device in a distributed environment. Figure 5 ,include: The sorting field value determination module 510 is used to obtain the sorting field value of the last piece of data in the previous page of query data; An initial data determination module 520 is used to determine the initial data of the next page of the retrieval node based on the sort field value as a screening condition; The target data determination module 530 is used to sort the initial data according to the sorting field and determine the target data of the retrieval node according to the number of query data on the next page; The next page determination module 540 is used to determine the next page data according to the target data.

[0058] like Figure 6As shown, based on the above-mentioned data paging merging method in a distributed environment, the present invention also provides an electronic device, which can be a computing electronic device such as a mobile terminal, a desktop computer, a notebook, a palm computer, and a server. The electronic device includes a processor 610, a memory 620, and a display 630. Figure 6 Only some components of the electronic device are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0059] In some embodiments, the memory 620 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory 620 may also be an external storage electronic device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 620 may also include both an internal storage unit of the electronic device and an external storage electronic device. The memory 620 is used to store application software and various types of data installed in the electronic device, such as program codes installed in the electronic device. The memory 620 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 620 stores a data paging merging program 640 in a distributed environment, and the data paging merging program 640 in the distributed environment can be executed by the processor 610, thereby realizing the data paging merging method in a distributed environment of each embodiment of the present application.

[0060] In some embodiments, the processor 610 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes stored in the memory 620 or process data, such as executing a data paging merging method in a distributed environment.

[0061] In some embodiments, the display 630 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 630 is used to display information of the data paging and merging electronic device in the distributed environment and to display a visual user interface. The components 610-630 of the electronic device communicate with each other through a system bus.

[0062] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0063] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A data paging merging method in a distributed environment, characterized in that: include: Get the sort field value of the last piece of data in the previous page of query data of the retrieval node; Using the sorting field value as a screening condition, determining the initial data of the next page of the retrieval node; Sort the initial data according to the sorting field, and determine the target data of the retrieval node according to the number of data items to be queried on the next page; The next page of data is determined according to the target data.

2. The data paging merging method in a distributed environment according to claim 1, characterized in that: The step of obtaining the sorting field value of the last piece of data in the previous page of query data of the retrieval node includes: Get the number of queried data on the previous page, and determine the data on the previous page of the retrieval node based on the number of queried data on the previous page; According to the number of data items queried on the previous page, obtain the sort field value of the last data item in the previous page of data queried by the retrieval node.

3. The data paging merging method in a distributed environment according to claim 1, characterized in that: The step of obtaining the sorting field value of the last piece of data in the previous page of data queried by the retrieval node according to the number of pieces of data queried on the previous page includes: According to the number of data items queried on the previous page, the search node obtains the data on the previous page; Partially sort the data on the previous page using the sorting field to obtain the sorted data, and obtain the sorting field value of the last record of the sorted data; The sort field value is cached in a cache system.

4. The data paging merging method in a distributed environment according to claim 1, characterized in that: The sorting field includes at least one of quantity, business-specific field, ID, price, creation time, update time, expiration time, user name, tag or category.

5. The data paging merging method in a distributed environment according to claim 1, characterized in that: The determining of the initial data of the retrieval node by using the sorting field value as the screening condition includes: Taking the sorting field value as a screening condition, data in the search node that is greater than or equal to the sorting field value is obtained to determine the initial data of the search node.

6. The data paging merging method in a distributed environment according to claim 1, characterized in that: The step of determining the next page of data according to the target data includes: Get the target data of the retrieval node; According to the sorting field, the target data is globally sorted to obtain globally sorted data; Compare the globally sorted data with the data on the previous page, remove duplicate data, and obtain deduplicated data; Determine the next page of data based on the number of query data items and deduplication data on the next page.

7. The data paging merging method in a distributed environment according to claim 1, characterized in that: After getting the next page of data, it also includes: Store the next page of data in the cache system.

8. A data paging merging device in a distributed environment, characterized in that: include: A sorting field value determination module is used to obtain the sorting field value of the last piece of data in the previous page of query data of the retrieval node; An initial data determination module, used to determine the initial data of the next page of the retrieval node based on the sort field value as a screening condition; A target data determination module is used to sort the initial data according to the sorting field and determine the target data of the retrieval node according to the number of query data on the next page; The next page determination module is used to determine the next page data according to the target data.

9. An electronic device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps of the data page merging method in a distributed environment as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the data page merging method in a distributed environment as described in any one of claims 1-7.

Images