Data query method and device, computer device and computer readable storage medium

Abstract

The application provides a data query method and device, computer equipment and a computer readable storage medium, comprising: receiving a data query request carrying application scenario path information; in response to the data query request, determining a target data source according to the application scenario path information, and constructing a data query condition corresponding to the target data source; generating a data query statement according to the data query condition; sending the data query statement to the target data source, so that the target data source feeds back a data query result based on the data query statement. By using the method, different data sources can be provided to obtain a data query result in response to different data application scenarios, thereby improving the query and retrieval efficiency of logistics data.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data query method, apparatus, computer device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of logistics technology, logistics companies face the pressure of processing tens of millions of data points daily. How to quickly and efficiently retrieve this data has become a primary challenge in the logistics field. Currently, based on users' requirements for fast system response and low latency, an increasing number of companies are choosing to develop professional logistics data retrieval systems to support scenarios involving rapid data querying and downloading.

[0003] However, limited by enterprises' technological capabilities and R&D costs, existing logistics data retrieval systems still have many shortcomings: 1. Traditional logistics data retrieval systems often use relational databases, which cannot meet the high-concurrency read / write requirements of massive amounts of data, nor can performance and load capacity be easily expanded by adding more hardware and service nodes. 2. Some logistics data retrieval systems using Elasticsearch have low query efficiency with billions of data points, and still consume a large amount of I / O even when querying only some fields, resulting in excessively long user wait times. 3. Some logistics data retrieval systems using HBase do not support complex query and aggregation operations, and cannot meet users' requirements for rich metrics.

[0004] Therefore, existing logistics data retrieval systems suffer from technical problems such as low query efficiency due to unreasonable retrieval strategies and database settings. Summary of the Invention

[0005] Therefore, it is necessary to provide a data query method, apparatus, computer equipment, and computer-readable storage medium to address the aforementioned technical problems, so as to improve the query and retrieval efficiency of massive logistics data.

[0006] Firstly, this application provides a data query method, including:

[0007] Receive data query requests carrying application scenario path information;

[0008] In response to the data query request, the target data source is determined based on the application scenario path information, and the data query conditions corresponding to the target data source are constructed.

[0009] Generate a data query statement based on the data query conditions;

[0010] The data query statement is sent to the target data source so that the target data source can provide a data query result based on the data query statement.

[0011] In some embodiments of this application, the step of determining a target data source based on the application scenario path information and constructing data query conditions corresponding to the target data source in response to the data query request includes: in response to the data query request, analyzing the application scenario path information; if the application scenario path information is first path information, determining a preset data search engine, Elasticsearch, as the target data source; if the application scenario path information is second path information, obtaining the total amount of data to be downloaded and determining the target data source based on the total amount of data to be downloaded; and constructing data query conditions based on the target data source.

[0012] In some embodiments of this application, the step of obtaining the total amount of data to be downloaded and determining the target data source based on the total amount of data to be downloaded if the application scenario path information is the second path information includes: if the application scenario path information is the second path information, obtaining the total amount of data to be downloaded through a preset data search engine Elasticsearch; if the total amount of data to be downloaded is less than or equal to a preset download volume threshold, determining the data search engine Elasticsearch as the target data source; if the total amount of data to be downloaded is greater than the download volume threshold, determining a preset distributed database HBase as the target data source.

[0013] In some embodiments of this application, the step of constructing data query conditions based on the target data source includes: if the target data source is a preset data search engine Elasticsearch, then obtaining and constructing data query conditions based on the request parameters in the data query request; if the target data source is a preset distributed database HBase, then obtaining and creating a download task based on the request parameters in the data query request, so as to construct data query conditions based on the request parameters in the download task when the task status of the download task is in an unstarted state.

[0014] In some embodiments of this application, the step of obtaining and constructing data query conditions based on the request parameters in the data query request if the target data source is a preset data search engine, Elasticsearch, includes: if the target data source is a preset data search engine, Elasticsearch, obtaining the request parameters in the data query request; validating the request parameters to obtain a valid validation result; and when the valid validation result is successful, performing format conversion processing on the request parameters according to the data storage format of the data search engine, Elasticsearch, to obtain the data query conditions.

[0015] In some embodiments of this application, the step of obtaining and creating a download task based on the request parameters in the data query request if the target data source is a preset distributed database HBase, and constructing data query conditions based on the request parameters in the download task when the task status of the download task is in an unstarted state, includes: if the target data source is a preset distributed database HBase, obtaining the request parameters in the data query request, and obtaining task name information, task status, and task creation time; creating a download task based on the task name information, task status, task creation time, and request parameters, and storing the download task in a local database for scanning and analysis at preset times; when the task status of the download task scanned in the local database is in an unstarted state, and there are idle threads in the preset download task thread pool, validating the request parameters to obtain a valid validation result; when the valid validation result is successful, performing format conversion processing on the request parameters according to the data storage format of the distributed database HBase to obtain the data query conditions.

[0016] In some embodiments of this application, the step of generating a data query statement based on the data query conditions includes: if the data query conditions are data query conditions corresponding to the distributed database HBase, then generating primary key index information based on the request parameters in the data query request; and generating a data query statement based on the primary key index information and the data query conditions.

[0017] In some embodiments of this application, after the step of sending the data query statement to the target data source, the method further includes: receiving the data query result fed back by the target data source, the target data source including a preset data search engine Elasticsearch and a distributed database HBase; if the data query result is the data query result fed back by the data search engine Elasticsearch, then processing the data query result according to the application scenario path information to obtain a processed data query result, the processed data query result being used to feed back to the terminal for display; if the data query result is the data query result fed back by the distributed database HBase, then performing format conversion processing on the data query result, storing the converted data query result in a preset character delimiter file and then compressing it to obtain a compressed data query result, the compressed data query result being used to feed back to the terminal for display.

[0018] In some embodiments of this application, the step of processing the data query result according to the application scenario path information to obtain a processed data query result if the data query result is a data query result returned by the data search engine Elasticsearch includes: if the data query result is a data query result returned by the data search engine Elasticsearch, then analyzing the application scenario path information; if the application scenario path information is first path information, then adding the data query result to a preset target set for format conversion processing to obtain a converted data query result, the converted data query result being used for display on the terminal; if the application scenario path information is second path information, then performing format conversion processing on the data query result, storing the converted data query result in a preset character delimited file, and then compressing it to obtain a compressed data query result, the compressed data query result being used for display on the terminal.

[0019] Secondly, this application provides a data query device, comprising:

[0020] The request receiving module is used to receive data query requests carrying application scenario path information;

[0021] The request-response module is used to respond to the data query request, determine the target data source based on the application scenario path information, and construct the data query conditions corresponding to the target data source.

[0022] The statement generation module is used to generate a data query statement based on the data query conditions.

[0023] The result acquisition module is used to send the data query statement to the target data source so that the target data source can return the data query result based on the data query statement.

[0024] Thirdly, this application also provides a computer device, comprising:

[0025] One or more processors;

[0026] The memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the data query method.

[0027] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the steps in the data query method.

[0028] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the first aspect described above.

[0029] The aforementioned data query method, apparatus, computer equipment, and computer-readable storage medium involve a server receiving and responding to data query requests carrying application scenario path information, determining the target data source, constructing data query conditions corresponding to the target data source to generate a data query statement, and finally sending the data query statement to the target data source for feedback on the required data query results. This allows for responses to different data application scenarios, providing different data sources to obtain data query results, thereby improving the efficiency of logistics data query and retrieval. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of a data query method in an embodiment of this application;

[0032] Figure 2 This is a flowchart illustrating the data query method in an embodiment of this application;

[0033] Figure 3 This is a schematic diagram illustrating the specific process of the data query method in the embodiments of this application;

[0034] Figure 4 This is a schematic diagram of the data query device in the embodiments of this application;

[0035] Figure 5 This is a schematic diagram of the structure of the computer device in the embodiments of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0038] In the description of this application, it should be understood that the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0039] In the embodiments of this application, the data query method provided in this application is executed in a computer device. The processing objects of each computer device exist in the form of data or information, such as time, which is essentially time information. It can be understood that if status, page number, payment method, etc. are mentioned in subsequent embodiments, they are all corresponding data that exist so that the computer device can process them. The specifics will not be elaborated here.

[0040] This application provides a data query method, apparatus, computer device, and computer-readable storage medium, which will be described in detail below.

[0041] See Figure 1 , Figure 1This is a schematic diagram illustrating a scenario for the data query method provided in this application, which can be applied to a data query system. The data query system includes a terminal 100 and a server 200. The terminal 100 can be a device that includes both receiving and transmitting hardware, i.e., a device with receiving and transmitting hardware capable of performing bidirectional communication over a bidirectional communication link. Such a device can include cellular or other communication devices, having a single-line display, a multi-line display, or a cellular or other communication device without a multi-line display. Specifically, the terminal 100 can be a desktop terminal or a mobile terminal, and can also be a mobile phone, tablet computer, laptop computer, etc. The server 200 can be a standalone server, or a server network or server cluster, including but not limited to computers, network hosts, single network servers, multiple network server sets, or cloud servers composed of multiple servers. The cloud server consists of a large number of computers or network servers based on cloud computing.

[0042] Those skilled in the art should understand that Figure 1 The application environment shown is merely one applicable scenario for the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include more than one. Figure 1 The number of computer devices shown is more or less, for example Figure 1 Only one server 200 is shown in the diagram. It is understood that the data query system may also include one or more other servers or one or more other terminals, which are not specifically limited here. In addition, the data query system may also include a storage device for storing data, such as logistics data, such as various data from the logistics platform, such as express tracking numbers, express status, forwarding status, etc.

[0043] Those skilled in the art should understand that Figure 1 The schematic diagram of the data query system shown is merely an example. The data query system and scenario described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. As those skilled in the art will know, with the evolution of data query systems and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0044] See Figure 2 This application provides a data query method, which is mainly applied to the above-mentioned... Figure 1 Taking server 200 as an example, the method includes steps S201 to S204, as follows:

[0045] S201, Receive a data query request carrying application scenario path information.

[0046] The application scenario path information can be path identifier information pointing to different data application scenarios. For example, it includes first path information and second path information. The first path information points to the data application scenario of "data query scenario", and the second path information points to the data application scenario of "data download scenario".

[0047] Among them, a data query request can be an instruction message used to request the data results needed by the user.

[0048] Specifically, before the server 200 performs a data query operation, it must first receive an instruction that can trigger the start of the query operation, such as receiving a data query request carrying application scenario path information. This data query request can be a request sent by the terminal 100 in real time or a request sent by the terminal 100 at regular intervals, and the application scenario path information can be path information determined based on user operation instructions.

[0049] For example, terminal 100 has an interactive interface displaying at least two virtual buttons, each representing a different application function, such as "data query" and "data download". After a user triggers one of the virtual buttons, they can submit their selected application function to terminal 100. Upon detecting the submitted application function, terminal 100 can determine the corresponding preset interface path and obtain the corresponding application scenario path information. Based on this application scenario path information, terminal 100 can generate a data query request and send it to server 200. Upon receiving the data query request, server 200 can respond and complete the subsequent analysis and query.

[0050] S202, in response to the data query request, determine the target data source based on the application scenario path information, and construct the data query conditions corresponding to the target data source.

[0051] The data sources can include the data search engine Elasticsearch and the distributed database HBase; Elasticsearch is a search server based on Lucene that provides a distributed, multi-user, full-text search engine; HBase (Hadoop Database) is a highly reliable, high-performance, column-oriented, scalable distributed storage system.

[0052] The target data source can refer to the data search engine Elasticsearch or the distributed database HBase, and is specifically determined by the application scenario path information generated by the terminal 100 in response to user operations.

[0053] Among them, the query condition is a rule used to identify the records to be included in the query results; the data query condition can be an index that reflects the query effect of the query condition.

[0054] Specifically, after receiving a data query request carrying application scenario path information, if the server 200 needs to perform security verification on the request (such as validity verification or identity verification), it must respond to the request only after obtaining a "valid" verification result. If no security verification is required, the server can directly analyze the application scenario path information to determine the target data source required for the current query data result, so as to construct the data query conditions supported by the target data source based on its data source type. The target data source determination steps and data query condition construction steps involved in this embodiment will be described in detail below.

[0055] In one embodiment, this step includes: responding to the data query request, analyzing the application scenario path information; if the application scenario path information is a first path information, determining a preset data search engine, Elasticsearch, as the target data source; if the application scenario path information is a second path information, obtaining the total amount of data to be downloaded, and determining the target data source based on the total amount of data to be downloaded; and constructing data query conditions based on the target data source.

[0056] The first path information can refer to the path information of the data application scenario as "data query scenario", which can be represented by the special string "a1" in a preset manner, but is not limited to using this string.

[0057] The second path information can refer to the path information of the data application scenario as "data download scenario", which can be represented by the special string "b1" in a preset manner, but is not limited to using this string.

[0058] The total amount of data to be downloaded can be the total number of data records to be downloaded as queried from Elasticsearch, such as 3, 20, 500, etc.

[0059] Specifically, after receiving a data query request carrying application scenario path information, server 200 can respond to the request and analyze the corresponding application scenario path information. If the analysis determines that the application scenario path information matches preset first path information, then the application scenario path information is identified as the first path information, and the preset data search engine Elasticsearch can be identified as the current target data source. If the analysis determines that the application scenario path information matches preset second path information, then the application scenario path information is identified as the second path information. Therefore, it is necessary to further obtain the total amount of data to be downloaded in order to determine the target data source based on the analysis results of this total amount of data to be downloaded. Finally, data query conditions are constructed using the target data source.

[0060] For example, if the server 200 has pre-stored the first path information as "a1" and the second path information as "b1", if the application scenario path information received by the server 200 is "a1", it means that the application scenario path information is the first path information. If the application scenario path information received by the server is "b1", it means that the application scenario path information is the second path information.

[0061] More specifically, after server 200 analyzes and determines that the currently obtained application scenario path information is the second path information, it needs to further obtain the parameter "total amount of data to be downloaded". This parameter is obtained based on the data search engine Elasticsearch and the request parameters carried in the data query request to construct initial query conditions. Based on these initial query conditions, an Elasticsearch query statement is generated and then sent to the data search engine Elasticsearch via an HTTP request to obtain the corresponding result, i.e., the "total amount of data to be downloaded". The request parameters carried in the data query request include, but are not limited to: express tracking number, monthly settlement card number, start date, end date, payment method, express status, forwarding status, target field, sorting field, sorting method, current page number, and quantity per page, etc. This type of information is common and frequently used in the logistics field. The initial query condition construction steps involved in this embodiment can be referred to the data query condition construction steps described below.

[0062] In one embodiment, the step of obtaining the total amount of data to be downloaded and determining the target data source based on the total amount of data to be downloaded if the application scenario path information is the second path information includes: if the application scenario path information is the second path information, obtaining the total amount of data to be downloaded through a preset data search engine Elasticsearch; if the total amount of data to be downloaded is less than or equal to a preset download volume threshold, determining the data search engine Elasticsearch as the target data source; if the total amount of data to be downloaded is greater than the download volume threshold, determining a preset distributed database HBase as the target data source.

[0063] The download threshold can be a critical value used to distinguish the download volume of target data sources. For example, the download threshold can be selected as 5, 10, 100, etc., mainly depending on the maximum download volume set for each data source. For example, if the maximum download volume for Elasticsearch data source is 50 and the maximum download volume for HBase data source is 70, then the current download threshold can be set to 50.

[0064] Specifically, after server 200 analyzes and determines that the currently obtained application scenario path information is the second path information, it can obtain the request parameters from the data query request and construct initial query conditions based on the request parameters. These initial query conditions are the query conditions supported by Elasticsearch. Then, based on the initial query conditions, an Elasticsearch query statement is generated and sent to the data search engine Elasticsearch via HTTP to obtain the total amount of data to be downloaded. The Elasticsearch query statement can be a string consisting of alphanumeric characters.

[0065] Furthermore, after server 200 obtains the total amount of data to be downloaded from the data search engine Elasticsearch, it can analyze the relationship between the total amount of data to be downloaded and the download threshold. If the analysis determines that the total amount of data to be downloaded is less than or equal to the download threshold, then the data search engine Elasticsearch can be identified as the current target data source. If the analysis determines that the total amount of data to be downloaded is greater than the download threshold, then the distributed database HBase can be identified as the current target data source.

[0066] For example, if the maximum download size for Elasticsearch is 50 and the maximum download size for HBase is 70, and the download threshold is set to 50, then when the total amount of data to be downloaded is less than or equal to 50, Elasticsearch can meet the download requirements, and thus Elasticsearch can be selected as the target data source; when the total amount of data to be downloaded is greater than 50, HBase can meet the download requirements, and thus HBase can be selected as the target data source.

[0067] In one embodiment, the step of constructing data query conditions based on the target data source includes: if the target data source is a preset data search engine Elasticsearch, then obtaining and constructing data query conditions based on the request parameters in the data query request; if the target data source is a preset distributed database HBase, then obtaining and creating a download task based on the request parameters in the data query request, so as to construct data query conditions based on the request parameters in the download task when the task status of the download task is in an unstarted state.

[0068] Specifically, the process of constructing data query conditions will also be different if different target data sources are selected, mainly because the data sources support different query methods.

[0069] For example, if the target data source is the data search engine Elasticsearch, server 200 can first obtain the request parameters from the data query request. These parameters, as described above, include some or all of the logistics data. Then, query conditions are constructed based on the request parameters. Furthermore, if the target data source is Elasticsearch, not only are query conditions constructed, but also aggregation conditions are needed for certain scenarios (such as "logistics flow" or "product distribution"). This involves performing specific aggregation operations to provide richer data metrics. The final data query conditions consist of query conditions and aggregation conditions, mainly including field query conditions, range query conditions, and data sorting conditions. The role of aggregation conditions is as follows: when analyzing "logistics flow," aggregation and sorting can be performed based on the receiving region to obtain information on the top 10 regions with the highest receiving volume.

[0070] For example, if the target data source is the distributed database HBase, server 200 first needs to obtain the request parameters from the data query request. These parameters, as described above, include some or all of the logistics data. Then, a download task is created based on the request parameters, and this task can be stored in a local MySQL database. Next, a download task thread pool and a scheduled scanning task are created. Server 200 periodically scans the database for download tasks that are not yet started. When an idle thread exists in the download task thread pool, a download task currently in the "not started" state is started. Based on the information contained in the download task itself (task name, task status, creation time, request parameters, etc.), data query conditions are constructed.

[0071] In one embodiment, the step of obtaining and constructing data query conditions based on the request parameters in the data query request if the target data source is a preset data search engine, Elasticsearch, includes: obtaining the request parameters in the data query request if the target data source is a preset data search engine, Elasticsearch; validating the request parameters to obtain a valid validation result; and when the valid validation result is successful, performing format conversion processing on the request parameters according to the data storage format of the data search engine, Elasticsearch, to obtain the data query conditions.

[0072] Specifically, after the server 200 analysis identifies the target data source, if the target data source is the data search engine Elasticsearch, then after obtaining the request parameters in the data query request, the request parameters need to be validated. This involves checking whether required parameters are empty, whether the date format is correct, whether the date range exceeds the limit, and whether enumerated parameters exist, etc., until all validation conditions are validated successfully. Then, according to the data storage format of the data search engine Elasticsearch, some request parameters can be formatted, such as converting dates to timestamps and Chinese parameters to their corresponding English parameters. Finally, data query conditions can be constructed based on the converted request parameters and other request parameters.

[0073] For example, some data query conditions can be directly constructed through parameters, such as the sending time range and monthly billing card number. Other data query conditions require further processing of the request parameters. For example, when the forwarding status is forwarded and returned, it needs to be processed into a one-dimensional array containing forwarding, return, and forwarded and returned.

[0074] In one embodiment, the step of obtaining and creating a download task based on the request parameters in the data query request if the target data source is a preset distributed database HBase, and constructing data query conditions based on the request parameters in the download task when the task status of the download task is in an unstarted state, includes: if the target data source is a preset distributed database HBase, obtaining the request parameters in the data query request, and obtaining task name information, task status, and task creation time; creating a download task based on the task name information, task status, task creation time, and request parameters, and storing the download task in a local database for scanning and analysis at preset times; when the task status of the download task scanned in the local database is in an unstarted state, and there are idle threads in the preset download task thread pool, validating the request parameters to obtain a valid validation result; when the valid validation result is successful, converting the request parameters according to the data storage format of the distributed database HBase to obtain the data query conditions.

[0075] The task name information can refer to the name of the HBase download task, which can be selected according to the preset name information.

[0076] The task status information can refer to the execution status of the HBase download task, including the status of not started and download completed.

[0077] The task creation time can refer to the creation time of the HBase download task, for example, 12:00 on January 1, 2021.

[0078] Specifically, if the target data source is the distributed database HBase, server 200 first needs to obtain the request parameters in the data query request, as well as the currently required task name information, task status, and task creation time, so as to create an HBase download task based on the task name information, task status, task creation time, and request parameters, and store the HBase download task in the local MySQL database for subsequent scanning and execution.

[0079] Furthermore, after server 200 creates an HBase download task, it can create a download task thread pool and a scheduled scanning task to periodically scan the database for HBase download tasks that are not yet started. When there is an idle thread in the download task thread pool, the current HBase download task is started, that is, data query conditions are constructed based on the information contained in the HBase download task itself. In addition, before constructing data query conditions, the format of the request parameters contained in the HBase download task needs to be validated. After the validation is valid, the format can be converted to finally obtain the data query conditions. The validation steps involved in this embodiment are described in the above embodiment and will not be repeated here.

[0080] S203, Generate a data query statement based on the data query conditions.

[0081] Specifically, different types of data query conditions correspond to different data query statements. Since the data query scheme proposed in this application includes two types of data query conditions: those supported by the data search engine Elasticsearch and those supported by the distributed database HBase, and since Elasticsearch data query conditions typically involve formatting the condition information to generate a data query statement, this application will not describe them in detail. However, HBase data query conditions involve primary key indexes, so this application will describe them in detail below.

[0082] In one embodiment, this step includes: if the data query condition is a data query condition corresponding to the distributed database HBase, then generating primary key index information based on the request parameters in the data query request; and generating a data query statement based on the primary key index information and the data query condition.

[0083] Specifically, a primary key index, also known as a clustered index, stores the entire row of data in its leaf nodes; while a non-primary key index, called a secondary index, stores the primary key value in its leaf nodes. Generally, a query based on the primary key only requires searching the B+ tree for the ID, but a query using a non-primary key index requires first searching the k-index tree to find the corresponding primary key, and then searching the ID index tree again—a process known as a table lookup. Therefore, queries using non-primary key indexes require scanning an additional index tree, resulting in lower search efficiency. Thus, this application proposes an HBase primary key index suitable for logistics data, enabling efficient retrieval of massive amounts of data and improving data query efficiency.

[0084] More specifically, after server 200 obtains the data query conditions, if the data query conditions are the same as those for the distributed database HBase, it can generate an HBase primary key index RowKey based on information such as the monthly settlement card number and mailing time in the request parameters. Then, based on the primary key index RowKey and the data query conditions, it generates a data query statement to query the distributed database HBase and obtain the query results.

[0085] S204, send the data query statement to the target data source so that the target data source can provide a data query result based on the data query statement.

[0086] Specifically, if the data query statement is an Elasticsearch data query statement, it needs to be sent to the data search engine Elasticsearch via an HTTP request so that Elasticsearch can analyze the query statement and then return the data query results.

[0087] More specifically, if the data query statement is an HBase data query statement, then the HBase data query conditions and HBase primary key index are used as the basis to perform data query on the distributed database HBase to obtain the data query results.

[0088] Furthermore, after obtaining the data query results, the server 200 needs to process the data query results according to the data display requirements of the terminal 100 before feeding them back to the terminal 100 for display. The data query result processing steps involved in this embodiment will be described in detail below.

[0089] In one embodiment, this step is followed by: receiving data query results from the target data source, the target data source including a preset data search engine Elasticsearch and a distributed database HBase; if the data query result is a data query result from the data search engine Elasticsearch, then processing the data query result according to the application scenario path information to obtain a processed data query result, the processed data query result being used to display on the terminal; if the data query result is a data query result from the distributed database HBase, then performing format conversion processing on the data query result, storing the converted data query result in a preset character delimiter file, and then compressing it to obtain a compressed data query result, the compressed data query result being used to display on the terminal.

[0090] Specifically, after receiving the data query results from the target data source, the server 200 needs to perform format conversion processing on the data query results based on the feedback from different target data sources, converting them into a data format that the terminal 100 can successfully display.

[0091] For example, if the data query result is returned by the data search engine Elasticsearch, it can be further analyzed to determine which application scenario path information the data query result corresponds to. If it corresponds to the second path information, the data query result needs to be processed, the data stored in a CSV file, and returned to terminal 100 for the user to download. If it corresponds to the first path information, the data query result needs to be parsed, the required data added to the corresponding collection, and returned to terminal 100 in the form of JSON data for display.

[0092] For example, if the data query result is returned by the distributed database HBase, the data query result needs to be format converted, and then the converted data query result is stored in a CSV file, compressed into a ZIP file and stored locally. The task status is updated to download complete in the local MySQL database, and terminal 100 provides a download entry. When the user clicks, the ZIP file is returned to terminal 100.

[0093] In one embodiment, the step of processing the data query result according to the application scenario path information to obtain a processed data query result if the data query result is a data query result returned by the data search engine Elasticsearch includes: if the data query result is a data query result returned by the data search engine Elasticsearch, then analyzing the application scenario path information; if the application scenario path information is first path information, then adding the data query result to a preset target set for format conversion processing to obtain a converted data query result, the converted data query result being used for display on the terminal; if the application scenario path information is second path information, then performing format conversion processing on the data query result, storing the converted data query result in a preset character delimited file, and then compressing it to obtain a compressed data query result, the compressed data query result being used for display on the terminal.

[0094] Specifically, based on the previous embodiment, this embodiment details how to adopt different processing methods for different path information. The purpose of the processing is to enable the terminal 100 to display successfully. The difference is that the target set mentioned above can be a group number. Adding the result to the target set is equivalent to converting the data into text format, while storing the data in a CSV file is equivalent to converting the data into table format.

[0095] The data query method described in the above embodiments involves the server receiving and responding to data query requests carrying application scenario path information, determining the target data source, constructing data query conditions corresponding to the target data source to generate a data query statement, and finally sending the data query statement to the target data source for feedback on the actual required data query results. This allows for responses to different data application scenarios, providing different data sources to obtain data query results, and improving the efficiency of logistics data query and retrieval.

[0096] To enable those skilled in the art to fully understand the data query scheme proposed in this application, this application also provides an application scenario in which the above-described data query method is applied. Specifically, the application of the data query method in this application scenario will be combined with... Figure 3 The following explanation is provided:

[0097] like Figure 3 As shown, the data query scheme proposed in this application includes the following steps:

[0098] Step 301: Determine the retrieval method based on the data application scenario; among which, the data application scenarios mainly include: ① data query scenario, ② small amount of data download scenario and ③ large amount of data download scenario.

[0099] Step 302: For ① data query scenario and ② small amount of data download scenario, construct query conditions based on the request parameters input by the terminal, and retrieve data through Elasticsearch.

[0100] Step 303: For scenario ③, massive data download, data retrieval is performed through HBase based on query conditions and HBase primary key index.

[0101] Specifically, step S301 includes:

[0102] Step 3011: The server receives a request from the terminal. If the data application scenario is "data query", the data is retrieved directly through Elasticsearch.

[0103] Step 3012: If the data application scenario is "data download", then query the total number of data records to be downloaded via Elasticsearch.

[0104] Step 3013: Compare the total number of data entries with the preset maximum number of Elasticsearch downloads; when the total number of data entries is less than or equal to the maximum number of Elasticsearch downloads, download through Elasticsearch; when the total number of data entries is greater than the maximum number of Elasticsearch downloads, download through HBase.

[0105] Specifically, step S302 includes:

[0106] Step 3021: Construct Elasticsearch query conditions based on the parameters carried in the request received from the terminal by the server. These request parameters include some or all of the following data: tracking number, monthly billing card number, start date, end date, payment method, shipment status, forwarding status, target field, sorting field, sorting method, current page number, and number of items per page.

[0107] Step 3022: Building upon step 3021, for certain scenarios, aggregation conditions are constructed, and specific aggregation operations are performed to provide richer data metrics. For example, when analyzing logistics flow, aggregation and sorting are performed based on the receiving region to obtain the top 10 regions in terms of receiving volume.

[0108] Step 3023: Based on the above query and aggregation conditions, generate an Elasticsearch query statement, send it to the Elasticsearch search engine via an HTTP request, obtain the corresponding query results, and process them.

[0109] Specifically, step S303 includes:

[0110] Step 3031: Create a download task based on the parameters carried in the request received from the terminal by the server. The download task mainly includes information such as task name, task status, creation time, and request parameters, which are stored in a MySQL database.

[0111] Step 3032: When the server starts, a download task thread pool and a scheduled scanning task are created. The download tasks in the database that are not yet started are scanned periodically. When there are idle threads in the download task thread pool, the current download task is started and the data query results are obtained from HBase.

[0112] Step 3033: Process the data query results, store the data in a CSV file, compress it into a ZIP file, store it on the server, update the task status to download complete in the MySQL database, provide a download entry point on the terminal, and return the ZIP file to the terminal when the user clicks it.

[0113] The data query method provided in the above embodiments proposes a data retrieval strategy that switches between different data sources for different application scenarios. Specifically, it allows Elasticsearch and HBase to perform a portion of the download task, which can make full use of system resources and improve system performance. This not only overcomes the problem of rapid retrieval of massive logistics data, but also breaks through the performance bottleneck of traditional logistics data retrieval systems, achieving the data retrieval performance requirements for billions of data points. This effectively improves data retrieval efficiency and enhances the query efficiency of logistics data.

[0114] To better implement the data query method in the embodiments of this application, based on the data query method, the embodiments of this application also provide a data query device, such as... Figure 4 As shown, the data query device 400 includes:

[0115] The request receiving module 410 is used to receive data query requests carrying application scenario path information.

[0116] The request response module 420 is used to respond to the data query request, determine the target data source according to the application scenario path information, and construct the data query conditions corresponding to the target data source.

[0117] The statement generation module 430 is used to generate a data query statement based on the data query conditions;

[0118] The result acquisition module 440 is used to send the data query statement to the target data source so that the target data source can return the data query result based on the data query statement.

[0119] In some embodiments of this application, the request response module 420 is further configured to respond to the data query request by analyzing the application scenario path information; if the application scenario path information is a first path information, then a preset data search engine, Elasticsearch, is determined as the target data source; if the application scenario path information is a second path information, then the total amount of data to be downloaded is obtained, and the target data source is determined based on the total amount of data to be downloaded; and data query conditions are constructed based on the target data source.

[0120] In some embodiments of this application, the request response module 420 is further configured to: if the application scenario path information is second path information, obtain the total amount of data to be downloaded through a preset data search engine Elasticsearch; if the total amount of data to be downloaded is less than or equal to a preset download threshold, determine the data search engine Elasticsearch as the target data source; if the total amount of data to be downloaded is greater than the download threshold, determine a preset distributed database HBase as the target data source.

[0121] In some embodiments of this application, the request response module 420 is further configured to, if the target data source is a preset data search engine Elasticsearch, obtain and construct data query conditions based on the request parameters in the data query request; if the target data source is a preset distributed database HBase, obtain and create a download task based on the request parameters in the data query request, so as to construct data query conditions based on the request parameters in the download task when the task status of the download task is not started.

[0122] In some embodiments of this application, the request response module 420 is further configured to, if the target data source is a preset data search engine Elasticsearch, obtain the request parameters in the data query request; perform validity verification on the request parameters to obtain a validity verification result; and when the validity verification result is successful, perform format conversion processing on the request parameters according to the data storage format of the data search engine Elasticsearch to obtain the data query conditions.

[0123] In some embodiments of this application, the request response module 420 is further configured to, if the target data source is a preset distributed database HBase, obtain the request parameters in the data query request, and obtain the task name information, task status, and task creation time; create a download task based on the task name information, task status, task creation time, and request parameters, and store the download task in a local database for scanning and analysis at preset times; when the task status of the download task scanned in the local database is not started, and there are idle threads in the preset download task thread pool, perform validity verification on the request parameters to obtain a validity verification result; when the validity verification result is successful, perform format conversion processing on the request parameters according to the data storage format of the distributed database HBase to obtain the data query conditions.

[0124] In some embodiments of this application, the statement generation module 430 is further configured to, if the data query condition is a data query condition corresponding to the distributed database HBase, generate primary key index information according to the request parameters in the data query request; and generate a data query statement according to the primary key index information and the data query condition.

[0125] In some embodiments of this application, the data query device 400 further includes a result processing module for receiving data query results fed back by the target data source, the target data source including a preset data search engine Elasticsearch and a distributed database HBase; if the data query result is a data query result fed back by the data search engine Elasticsearch, then the data query result is processed according to the application scenario path information to obtain a processed data query result, which is used to be displayed on the terminal; if the data query result is a data query result fed back by the distributed database HBase, then the data query result is format converted, and the converted data query result is stored in a preset character delimiter file and then compressed to obtain a compressed data query result, which is used to be displayed on the terminal.

[0126] In some embodiments of this application, the result processing module is further configured to: if the data query result is a data query result returned by the data search engine Elasticsearch, then analyze the application scenario path information; if the application scenario path information is first path information, then add the data query result to a preset target set for format conversion processing to obtain a converted data query result, which is used to be displayed on the terminal; if the application scenario path information is second path information, then perform format conversion processing on the data query result, store the converted data query result in a preset character delimiter file, and then compress it to obtain a compressed data query result, which is used to be displayed on the terminal.

[0127] In the above embodiments, the server receives and responds to data query requests carrying application scenario path information, determines the target data source, constructs data query conditions corresponding to the target data source to generate a data query statement, and finally sends the data query statement to the target data source so that the target data source can provide the actual data query results it needs. Thus, it can respond to different data application scenarios and provide different data sources to obtain data query results, thereby improving the efficiency of logistics data query and retrieval.

[0128] For specific limitations regarding the data query device, please refer to the limitations on the data query method above, which will not be repeated here. Each module in the aforementioned data query device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0129] In some embodiments of this application, the data query device 400 can be implemented as a computer program, and the computer program can be implemented in, for example... Figure 5 The computer device shown operates on this device. The computer device's memory can store the various program modules that make up the data query device 400, for example, Figure 4 The diagram shows a request receiving module 410, a request response module 420, a statement generation module 430, and a result acquisition module 440. The computer program comprised of these modules causes the processor to execute the steps of the data query methods in the various embodiments of this application described in this specification.

[0130] For example, Figure 5 The computer device shown can be used as follows Figure 4 The request receiving module 410 in the data query device 400 shown executes step S201. The computer device can execute step S202 through the request response module 420. The computer device can execute step S203 through the statement generation module 430. The computer device can execute step S204 through the result acquisition module 440. The computer device includes a processor, memory, and network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with external computer devices via a network connection. When the computer program is executed by the processor, it implements a data query method.

[0131] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0132] In some embodiments of this application, a computer device is provided, including one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processors using the steps of the data query method described above. The steps of the data query method here may be steps from the data query methods of the various embodiments described above.

[0133] In some embodiments of this application, a computer-readable storage medium is provided, storing a computer program that is loaded by a processor, causing the processor to execute the steps of the data query method described above. The steps of the data query method here can be the steps in the data query methods of the various embodiments described above.

[0134] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0135] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The data query method, apparatus, computer device, and computer-readable storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data query method, characterized in that, include: Receive data query requests carrying application scenario path information; In response to the data query request, the target data source is determined based on the application scenario path information, and the data query conditions corresponding to the target data source are constructed. Generate a data query statement based on the data query conditions; Send the data query statement to the target data source so that the target data source can return the data query result based on the data query statement; The step of responding to the data query request by determining the target data source based on the application scenario path information and constructing the data query conditions corresponding to the target data source includes: In response to the data query request, the application scenario path information is analyzed; If the application scenario path information is the first path information, then the preset data search engine Elasticsearch is determined as the target data source; If the application scenario path information is the second path information, then obtain the total amount of data to be downloaded, and determine the target data source based on the total amount of data to be downloaded; Based on the target data source, construct the data query conditions; The step of obtaining the total amount of data to be downloaded and determining the target data source based on the total amount of data to be downloaded if the application scenario path information is the second path information includes: If the application scenario path information is the second path information, then the total amount of data to be downloaded is obtained through the preset data search engine Elasticsearch; If the total amount of data to be downloaded is less than or equal to a preset download threshold, then the data search engine Elasticsearch is selected as the target data source. If the total amount of data to be downloaded is greater than the download threshold, then a preset distributed database, HBase, is selected as the target data source.

2. The data query method as described in claim 1, characterized in that, The step of constructing data query conditions based on the target data source includes: If the target data source is a preset data search engine Elasticsearch, then obtain and construct data query conditions based on the request parameters in the data query request; If the target data source is a preset distributed database HBase, then a download task is created based on the request parameters in the data query request, so that when the task status of the download task is not started, data query conditions are constructed based on the request parameters in the download task.

3. The data query method as described in claim 2, characterized in that, If the target data source is a preset data search engine, Elasticsearch, the step of obtaining and constructing data query conditions based on the request parameters in the data query request includes: If the target data source is a preset data search engine Elasticsearch, then obtain the request parameters in the data query request; The validity of the request parameters is validated to obtain a valid validation result; When the valid verification result is successful, the request parameters are converted according to the data storage format of the data search engine Elasticsearch to obtain the data query conditions.

4. The data query method as described in claim 2, characterized in that, If the target data source is a preset distributed database HBase, then the step of obtaining and creating a download task based on the request parameters in the data query request, and constructing data query conditions based on the request parameters in the download task when the task status of the download task is not started, includes: If the target data source is a preset distributed database HBase, then the request parameters in the data query request are obtained, and the task name information, task status and task creation time are obtained. Based on the task name information, task status, task creation time, and request parameters, a download task is created and stored in a local database for scanning and analysis at preset intervals. When the download task status detected in the local database is not started, and there are idle threads in the preset download task thread pool, the validity of the request parameters is validated to obtain a valid validation result. When the valid verification result is successful, the request parameters are converted according to the data storage format of the distributed database HBase to obtain the data query conditions.

5. The data query method as described in claim 1, characterized in that, The step of generating a data query statement based on the data query conditions includes: If the data query condition is the same as the data query condition for the distributed database HBase, then primary key index information is generated based on the request parameters in the data query request. A data query statement is generated based on the primary key index information and the data query conditions.

6. The data query method as described in claim 1, characterized in that, After the step of sending the data query statement to the target data source, the method further includes: Receive data query results from the target data source, which includes a preset data search engine Elasticsearch and a distributed database HBase; If the data query result is the data query result returned by the data search engine Elasticsearch, then the data query result is processed according to the application scenario path information to obtain the processed data query result, which is used to be displayed on the terminal. If the data query result is the data query result returned by the distributed database HBase, then the data query result is format converted, and the converted data query result is stored in a preset character delimiter file and then compressed to obtain a compressed data query result. The compressed data query result is used to be displayed on the terminal.

7. The data query method as described in claim 6, characterized in that, The step of processing the data query result according to the application scenario path information to obtain the processed data query result if the data query result is the data query result returned by the data search engine Elasticsearch includes: If the data query result is the data query result returned by the data search engine Elasticsearch, then analyze the application scenario path information; If the application scenario path information is the first path information, the data query result is added to the preset target set for format conversion processing to obtain the converted data query result, which is used to feed back to the terminal for display. If the application scenario path information is the second path information, the data query result is format converted, and the converted data query result is stored in a preset character delimiter file and then compressed to obtain the compressed data query result. The compressed data query result is used to feed back to the terminal for display.

8. A data query device, characterized in that, include: The request receiving module is used to receive data query requests carrying application scenario path information; The request-response module is used to respond to the data query request, determine the target data source based on the application scenario path information, and construct the data query conditions corresponding to the target data source. The statement generation module is used to generate a data query statement based on the data query conditions; The result acquisition module is used to send the data query statement to the target data source, so that the target data source can return the data query result based on the data query statement; The request-response module is used to respond to the data query request, determine the target data source based on the application scenario path information, and construct the data query conditions corresponding to the target data source, including: In response to the data query request, the application scenario path information is analyzed; If the application scenario path information is the first path information, then the preset data search engine Elasticsearch is determined as the target data source; If the application scenario path information is the second path information, then obtain the total amount of data to be downloaded, and determine the target data source based on the total amount of data to be downloaded; Based on the target data source, construct the data query conditions; Specifically, the request response module is used to obtain the total amount of data to be downloaded through a preset data search engine, Elasticsearch, if the application scenario path information is the second path information. If the total amount of data to be downloaded is less than or equal to a preset download threshold, then the data search engine Elasticsearch is selected as the target data source. If the total amount of data to be downloaded is greater than the download threshold, then a preset distributed database, HBase, is selected as the target data source.

9. A computer device, characterized in that, The computer device includes: One or more processors; The memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the data query method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps of the data query method according to any one of claims 1 to 7.

Images