Multi-source data set query processing method and device

Abstract

The invention discloses a multi-source data set query processing method and device. The method comprises the steps that data set query parameters conforming to user habits are determined according to historical multi-source data set query processing records of a user; the data set query parameters comprise a screening condition parameter, a sorting parameter and a paging parameter; based on an edge calculation mode, screening out a specified data source node from the plurality of data source nodes by utilizing the screening condition parameters and a preset bloom filter of each data source node; reading data from a specified data source node; sorting the read data by using the sorting parameters to obtain a sorted data set; and performing paging processing on the sorted data set by using the paging parameter to obtain a paged data set. The multi-source data set query processing efficiency can be improved, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for querying and processing multi-source datasets. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] In the civil aviation service fee settlement system, data may originate from multiple sources, such as its own application system or other application systems. This necessitates filtering and viewing data from all sources, similar to the functionality of a database system. However, using traditional database methods requires storing all data from all sources into the database first, followed by user filtering and viewing. This repetitive operation is time-consuming, resulting in a poor user experience and extremely low efficiency, failing to meet practical needs. Furthermore, other application systems only provide interfaces for querying data and do not allow direct connection to their databases, making data manipulation via database connections impractical. Summary of the Invention

[0004] This invention provides a method for querying and processing multi-source datasets to improve the efficiency of multi-source dataset query processing and enhance user experience. The method includes:

[0005] Based on the user's historical multi-source dataset query processing records, determine the dataset query parameters that conform to the user's habits; the dataset query parameters include filter parameters, sorting parameters, and pagination parameters;

[0006] Based on the edge computing model, the specified data source node is selected from multiple data source nodes by using the filtering condition parameters and the preset Bloom filter of each data source node.

[0007] Read data from the specified data source node;

[0008] The read data is sorted using sorting parameters to obtain a sorted data set;

[0009] The sorted data set is paginated using pagination parameters to obtain the paginated data set.

[0010] This invention also provides a multi-source dataset query processing device to improve the efficiency of multi-source dataset query processing and enhance user experience. The device includes:

[0011] The parameter determination module is used to determine the dataset query parameters that conform to the user's habits based on the user's historical multi-source dataset query processing records; the dataset query parameters include filter parameters, sorting parameters, and pagination parameters;

[0012] The filtering module is used to filter a specified data source node from multiple data source nodes based on edge computing mode, using filtering condition parameters and a preset Bloom filter for each data source node; and to read data from the specified data source node.

[0013] The sorting module is used to sort the read data using sorting parameters to obtain a sorted data set.

[0014] The pagination module is used to perform pagination processing on the sorted data set using pagination parameters to obtain the paginated data set.

[0015] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described multi-source dataset query processing method.

[0016] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multi-source dataset query processing method.

[0017] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described multi-source dataset query processing method.

[0018] In this embodiment of the invention, the first step is to determine dataset query parameters that conform to user habits based on the user's historical multi-source dataset query processing records. These parameters include filtering parameters, sorting parameters, and pagination parameters. This eliminates the need for additional user operations, improving processing efficiency and user experience. Then, based on an edge computing model, the specified data source node is selected from multiple data source nodes using the filtering parameters and a pre-set Bloom filter for each data source node. This avoids obtaining unnecessary data from unknown sources, saving computational and storage resources. Finally, data is read from the specified data source node, sorted using the sorting parameters, and then paginated using the pagination parameters to obtain a paginated dataset. Compared to existing solutions, this method significantly improves the efficiency of multi-source dataset query processing and saves time. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0020] Figure 1 This is a flowchart illustrating the multi-source dataset query processing method in an embodiment of the present invention;

[0021] Figure 2 This is a specific example diagram of the multi-source dataset query processing method in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the sorting process in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the multi-source dataset query processing device in an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0026] In the civil aviation service fee settlement system, data may originate from multiple sources, such as its own application system or other application systems. Filtering, sorting, and paginating this diverse data is currently the main challenge.

[0027] Figure 1 This is a flowchart illustrating the multi-source dataset query processing method in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0028] Step 101: Determine the dataset query parameters that conform to the user's habits based on the user's historical multi-source dataset query processing records; the dataset query parameters include filter parameters, sorting parameters, and pagination parameters;

[0029] Step 102: Based on the edge computing mode, using the filtering condition parameters and the preset Bloom filter of each data source node, select the specified data source node from multiple data source nodes.

[0030] Step 103: Read data from the specified data source node;

[0031] Step 104: Sort the read data using sorting parameters to obtain a sorted data set;

[0032] Step 105: Use pagination parameters to paginate the sorted data set to obtain the paginated data set.

[0033] The multi-source dataset query processing method in the embodiments of the present invention will be explained in detail below.

[0034] First, determine the dataset query parameters that conform to the user's habits based on the user's historical multi-source dataset query processing records.

[0035] For example, by using AI, machine learning, and programming technologies, data filtering parameters, pagination parameters, and sorting parameters can be automatically split according to user operating habits.

[0036] Figure 2 This is a specific example diagram of the multi-source dataset query processing method in an embodiment of the present invention, such as... Figure 2 As shown, determining dataset query parameters that conform to user habits based on the user's historical multi-source dataset query processing records can include:

[0037] Step 201: Collect and process user history data from multiple sources.

[0038] Step 202: Based on machine learning and dynamic weight adjustment, analyze the user's historical multi-source dataset query processing record data to obtain dataset query parameters that conform to user habits.

[0039] During implementation, an operating platform is provided to support users in accessing multi-source data. A dynamic data processing algorithm is constructed using dynamic weight adjustment and machine learning to collect user operating habits. Through continuous learning and optimization of the machine learning algorithm, user behavior, such as sorting and pagination habits, is analyzed and recorded. This reduces user operations by eliminating the need for repeated sorting and pagination settings when using the platform again. Dynamic weight adjustment, for example, involves recording each user's operation and then using a pre-defined algorithm to identify the query conditions, pagination parameters, and sorting conditions for data lists that the user is interested in or frequently views (i.e., viewed more than a specified number of times). These query parameters are then dynamically adjusted based on the user's subsequent logins.

[0040] In implementation, front-end parameters can be extracted through a parameter object collection "BaseQueryVO". "BaseQueryVO" can have three attributes: filter parameters, sort parameters, and pagination parameters. For example, filter parameters use Java generics to store filter objects for filtering operations. Sorting parameters use a List collection, which can accept sorting rules for multiple attributes to implement user data sorting operations. The sorting parameter definition rule is: "sorting direction symbol" plus "sorting attribute". The sorting direction symbol has "+" for ascending order and "-" for descending order; for example, "+userName" means sorting by "userName". Pagination parameters include two attributes: pagination index and page size, used for data pagination operations.

[0041] When end users use the platform, the data that they care about most, including dataset query parameters, should be displayed first to improve the user experience.

[0042] After analyzing and obtaining the user-preferred dataset query parameters, in one embodiment, before selecting the specified data source node from multiple data source nodes using the filtering condition parameters and the preset Bloom filter for each data source node in the edge computing mode, the method may further include:

[0043] Receive confirmation instructions from users regarding dataset query parameters that conform to their habits. After user confirmation, directly use the user-confirmed dataset query parameters for subsequent processing.

[0044] Alternatively, it can receive user instructions to modify dataset query parameters that conform to user habits, modify and update dataset query parameters that conform to user habits based on the modification instructions, and use the dataset query parameters that the user has modified and confirmed for subsequent processing.

[0045] In step 102, based on the edge computing mode, the specified data source node is selected from multiple data source nodes using filtering condition parameters and a preset Bloom filter for each data source node. In step 103, data is read from the specified data source node.

[0046] In this embodiment, an edge computing scenario is introduced to optimize the performance and accuracy of querying data from multiple data sources. This solution is based on joint filtering using Bloom filters. Multiple data source nodes pre-generate and share Bloom filters for their local data. Before querying data, the intersection of the filters is used to locate the node information where the data the user wants to view may exist. Then, a precise query is initiated based on the node information. This way, queries can be performed on only certain data sources each time, thus avoiding the need to perform queries on all data sources every time.

[0047] For example, based on the edge computing model, using filtering condition parameters and a preset Bloom filter for each data source node, a specified data source node can be selected from multiple data source nodes. This can include: sending the filtering condition parameters to the edge computing node so that the edge computing node can: use the filtering condition parameters and the preset Bloom filter for each data source node to select the specified data source node from multiple data source nodes and then return the selected data source node.

[0048] In one embodiment, instead of edge computing, data filtering can be implemented using the Java Stream API. This approach first uses an asynchronous loading strategy, employing asynchronous multithreading to simultaneously read data from multiple data source nodes. The asynchronously read data is then cached. After all threads have finished reading data, the entire dataset is packaged and processed. The collection's streaming methods are used to convert the entire dataset into multiple data streams. Then, the stream's filtering methods are used to filter these streams. The filtering method receives a function that determines whether a data element is filtered. Elements returning "true" remain and proceed to the next stream, while elements returning "false" are filtered out. This only requires comparing the filtering parameters with the data element's attribute value. If the data element's attribute value contains or equals the filtering parameters, it proceeds to the next stream; otherwise, it is filtered out. Finally, the stream's collection methods are used to place the stream elements into a container of a specified type, resulting in the filtered dataset.

[0049] In step 104, the read data is sorted using sorting parameters to obtain a sorted data set.

[0050] As mentioned earlier, by analyzing user habits through machine learning, this step achieves intelligent data sorting based on user sorting habits, so that users do not need to repeat the sorting operation when using the system later, thereby improving user experience and work efficiency and reducing the tedious operation of adjusting the sorting.

[0051] In addition to automatic sorting, users can also define a sorting rule. When determining the dataset query parameters in step 101, some common sorting rules can be provided to users, and user-defined sorting rules are also supported. Users can set the sorting to be based on historical sorting habits, the selected sorting rule, or the custom sorting rule.

[0052] In one embodiment, the aforementioned intelligent sorting and edge computing scenarios refer to the process of filtering query conditions to achieve sorting. The essence of sorting using sorting parameters is to transform data features into quantifiable "comparison keys," achieving ordered arrangement through single or combined rules. For example, when sorting read data using sorting parameters, refer to... Figure 3 , Figure 3 This is a flowchart illustrating the sorting process in an embodiment of the present invention. In this method:

[0053] 1) Input the sorting parameters and the dataset;

[0054] 2) Check if the sorting parameters or data set are empty. If they are empty, end the process; otherwise, proceed to the next step.

[0055] 3) Traverse the sorting parameters and generate a sorting comparison chain;

[0056] 4) Obtain the data element value and determine if it is empty. If it is empty, proceed to step 5); otherwise, proceed to step 6.

[0057] 5) Generate a comparator, which is:

[0058] 1. If both values ​​are empty, return 0;

[0059] 2. If the first value is empty and the second value is not empty, return -1;

[0060] 3. If the first value is not empty and the second value is empty, return 1;

[0061] 6) Generate a comparator, which is:

[0062] The data type of the data element is obtained using reflection, and different comparison methods are called according to the data type. Data types include character, integer, and date, etc.

[0063] 7) Place the comparator in the comparison chain, and use the `sort` method, passing in the sorting comparison chain, to sort the list (the `sort` method in Java is for sorting lists such as List<T>). <t>(Methods for sorting);

[0064] 8) End.

[0065] In step 105, the sorted data set is paginated using pagination parameters to obtain the paginated data set.

[0066] In one embodiment, the paging parameters include a paging index and the size of each page.

[0067] Using pagination parameters to paginate a sorted dataset, the resulting paginated dataset may include:

[0068] The sorted dataset is paginated using the pagination index and the size of each page to obtain the paginated dataset.

[0069] In implementation, data truncation is achieved using the Java Stream API. First, the collection's streaming methods are used to convert the collection into a data stream. Next, the offset of each data element is calculated based on the pagination index and the size of each page (referred to as page size) in the pagination parameters. If not specified, the default pagination index and page size can be used, and the stream's skip method is used to skip data elements at that offset. Then, the stream's truncation method is used to truncate the data elements corresponding to the page size specified in the pagination parameters. Next, the stream elements are placed into a container of a specified type using a collection, thus obtaining the data collection corresponding to the pagination index. Finally, the total number of pages is calculated based on the pagination parameters and the data collection, and returned in a uniform format.

[0070] Finally, the paginated data is displayed on the front end to the user.

[0071] This invention provides a flexible, efficient, and easy-to-use solution that can help developers achieve data filtering, sorting, and pagination effects similar to those of database systems.

[0072] This invention also provides a multi-source dataset query processing device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the multi-source dataset query processing method, the implementation of this device can refer to the implementation of the multi-source dataset query processing method; repeated details will not be elaborated further.

[0073] Figure 4 This is a schematic diagram of a multi-source dataset query processing device in an embodiment of the present invention, such as... Figure 4 As shown, the device 400 includes:

[0074] The parameter determination module 401 is used to determine the dataset query parameters that conform to the user's habits based on the user's historical multi-source dataset query processing records; the dataset query parameters include filtering condition parameters, sorting parameters, and pagination parameters;

[0075] The filtering processing module 402 is used to filter a specified data source node from multiple data source nodes based on the edge computing mode, using filtering condition parameters and a preset Bloom filter for each data source node; and to read data from the specified data source node.

[0076] The sorting processing module 403 is used to sort the read data using sorting parameters to obtain a sorted data set;

[0077] The pagination processing module 404 is used to perform pagination processing on the sorted data set using pagination parameters to obtain the paginated data set.

[0078] In one embodiment, the parameter determination module 401 is specifically used for:

[0079] Collect and process user history data from multiple sources by querying and processing records.

[0080] Based on machine learning and dynamic weight adjustment, the system analyzes and processes historical multi-source dataset query records to obtain dataset query parameters that conform to user habits.

[0081] In one embodiment, the device further includes:

[0082] The parameter user confirmation module is used to receive the user's confirmation instruction for the dataset query parameters that conform to the user's habits before the filtering processing module 402 filters the specified data source node from multiple data source nodes based on the edge computing mode, using the filtering condition parameters and the preset Bloom filter of each data source node.

[0083] Alternatively, it can receive user instructions to modify dataset query parameters that conform to user habits, and modify or update the dataset query parameters that conform to user habits based on the modification instructions.

[0084] In one embodiment, the filtering processing module 402 is specifically used for:

[0085] The filtering parameters are sent to the edge computing nodes so that the edge computing nodes can use the filtering parameters and the Bloom filter preset by each data source node to select the specified data source node from multiple data source nodes and return the selected data source node.

[0086] In one embodiment, the paging parameters include a paging index and the size of each page.

[0087] The pagination processing module 404 is specifically used for:

[0088] The sorted dataset is paginated using the pagination index and the size of each page to obtain the paginated dataset.

[0089] Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention, such as... Figure 5 As shown, this embodiment of the invention also provides a computer device 500, including a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, it implements the above-mentioned multi-source dataset query processing method.

[0090] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multi-source dataset query processing method.

[0091] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described multi-source dataset query processing method.

[0092] In this embodiment of the invention, the first step is to determine dataset query parameters that conform to user habits based on the user's historical multi-source dataset query processing records. These parameters include filtering parameters, sorting parameters, and pagination parameters. This eliminates the need for additional user operations, improving processing efficiency and user experience. Then, based on an edge computing model, the specified data source node is selected from multiple data source nodes using the filtering parameters and a pre-set Bloom filter for each data source node. This avoids obtaining unnecessary data from unknown sources, saving computational and storage resources. Finally, data is read from the specified data source node, sorted using the sorting parameters, and then paginated using the pagination parameters to obtain a paginated dataset. Compared to existing solutions, this method significantly improves the efficiency of multi-source dataset query processing and saves time.

[0093] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / t>

Claims

1. A method for querying and processing multi-source datasets, characterized in that, include: Determine dataset query parameters that conform to user habits based on the user's historical multi-source dataset query processing records; Data set query parameters include filter parameters, sorting parameters, and pagination parameters; Based on the edge computing model, the specified data source node is selected from multiple data source nodes by using the filtering condition parameters and the preset Bloom filter of each data source node. Read data from the specified data source node; The read data is sorted using sorting parameters to obtain a sorted data set; The sorted data set is paginated using pagination parameters to obtain the paginated data set.

2. The method as described in claim 1, characterized in that, Based on the user's historical multi-source dataset query processing records, determine the dataset query parameters that conform to the user's habits, including: Collect and process user history data from multiple sources by querying and processing records. Based on machine learning and dynamic weight adjustment, the system analyzes and processes historical multi-source dataset query records to obtain dataset query parameters that conform to user habits.

3. The method as described in claim 1, characterized in that, Before selecting a specified data source node from multiple data source nodes using edge computing-based filtering parameters and a pre-defined Bloom filter for each data source node, the process also includes: Receive confirmation instructions from users regarding query parameters for datasets that match their preferences; Alternatively, it can receive user instructions to modify dataset query parameters that conform to user habits, and modify or update the dataset query parameters that conform to user habits based on the modification instructions.

4. The method as described in claim 1, characterized in that, Based on edge computing, using filtering parameters and a pre-set Bloom filter for each data source node, a specified data source node is selected from multiple data source nodes, including: The filtering parameters are sent to the edge computing nodes so that the edge computing nodes can use the filtering parameters and the Bloom filter preset by each data source node to select the specified data source node from multiple data source nodes and return the selected data source node.

5. The method as described in claim 1, characterized in that, The pagination parameters include the pagination index and the size of each page. The sorted dataset is paginated using pagination parameters to obtain the paginated dataset, which includes: The sorted dataset is paginated using the pagination index and the size of each page to obtain the paginated dataset.

6. A multi-source dataset query processing device, characterized in that, include: The parameter determination module is used to determine the dataset query parameters that conform to the user's habits based on the user's historical multi-source dataset query processing records. Data set query parameters include filter parameters, sorting parameters, and pagination parameters; The filtering module is used to filter out a specified data source node from multiple data source nodes based on the edge computing mode, using filtering condition parameters and a preset Bloom filter for each data source node. Read data from the specified data source node; The sorting module is used to sort the read data using sorting parameters to obtain a sorted data set. The pagination module is used to perform pagination processing on the sorted data set using pagination parameters to obtain the paginated data set.

7. The apparatus as claimed in claim 6, characterized in that, The parameter determination module is specifically used for: Collect and process user history data from multiple sources by querying and processing records. Based on machine learning and dynamic weight adjustment, the system analyzes and processes historical multi-source dataset query records to obtain dataset query parameters that conform to user habits.

8. The apparatus as claimed in claim 6, characterized in that, Also includes: The parameter user confirmation module is used to receive the user's confirmation instruction on the dataset query parameters that conform to the user's habits before the filtering processing module filters the specified data source node from multiple data source nodes based on the edge computing mode, using the filtering condition parameters and the preset Bloom filter of each data source node. Alternatively, it can receive user instructions to modify dataset query parameters that conform to user habits, and modify or update the dataset query parameters that conform to user habits based on the modification instructions.

9. The apparatus as claimed in claim 6, characterized in that, The filtering module is specifically used for: The filtering parameters are sent to the edge computing nodes so that the edge computing nodes can use the filtering parameters and the Bloom filter preset by each data source node to select the specified data source node from multiple data source nodes and return the selected data source node.

10. The apparatus as claimed in claim 6, characterized in that, The pagination parameters include the pagination index and the size of each page. The pagination processing module is specifically used for: The sorted dataset is paginated using the pagination index and the size of each page to obtain the paginated dataset.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 5.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 5.

Images