Data query method and device, equipment, storage medium and computer program product

Abstract

The invention discloses a data query method and device, equipment, a storage medium and a computer program product, and relates to the technical field of data processing.The method comprises the steps that when the parameter number of query parameters of a user query task is larger than a preset parameter number threshold value, parameter splicing is conducted on the query parameters, and the spliced query parameters are obtained; generating a target hash value corresponding to the user query task based on the spliced query parameters through a preset hash algorithm; and according to the target hash value, judging whether a target query task which is the same as the user query task exists in the historical query record, and if so, obtaining and returning a historical query result of the target query task based on the historical query record. By means of the technical scheme, the technical problem that in the prior art, when a user conducts data query repeatedly, the query pressure of a big data query engine can be increased, and consequently the data query efficiency is reduced is solved, and then the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data query method, device, equipment, storage medium and computer program product. Background Art

[0002] When interacting with data on a web page, if the underlying data is huge, especially when the data scale is above PB (Petabyte), a query often takes minutes to get a response, and this slow query affects the user experience. At present, although the use of various big data components for data query can effectively improve the efficiency of data query, when a large number of users query massive data, the QPS (Queries-per-second) of the big data query engine is usually less than 100. If users repeatedly query massive data, it will increase the query pressure of the big data query engine, resulting in reduced data query efficiency, thus affecting the user experience. Summary of the invention

[0003] The main purpose of this application is to provide a data query method, device, equipment, storage medium and computer program product, aiming to solve the technical problem in the prior art that when users repeatedly perform data queries, the query pressure of the big data query engine will increase, resulting in reduced data query efficiency.

[0004] To achieve the above purpose, the present application proposes a data query method, which includes: When receiving a user query task, determining a query parameter of the user query task; When the number of parameters of the query parameters is greater than a preset parameter number threshold, performing parameter splicing on the query parameters to obtain spliced ​​query parameters; Generate a target hash value corresponding to the user query task based on the concatenated query parameters by using a preset hash algorithm; It is determined whether there is a target query task identical to the user query task in the historical query record according to the target hash value; if so, a historical query result of the target query task is obtained and returned based on the historical query record.

[0005] In one embodiment, after the step of determining whether there is a target query task identical to the user query task in the historical query record according to the target hash value, the method further includes: If not, determining the data query structure of the user query task based on the query parameters; Perform data query on the user query task based on the data query structure and the target hash value, and obtain and return the data query result of the target query task.

[0006] In one embodiment, the step of performing data query on the user query task based on the data query structure and the target hash value, and obtaining and returning the data query result of the target query task includes: Performing task slicing on the user query task based on the data query structure to obtain a plurality of slicing query tasks; Performing data query on the shard query task based on the target hash value and a preset data query method to obtain a shard data query result; Loading the shard data query results into the cache database based on asynchronous loading technology; Return the query result of the shard data in the cache database in real time; The cache database is polled and queried until the data query result of the user query task is obtained and returned.

[0007] In one embodiment, the step of performing a polling query on the cache database until obtaining and returning the data query result of the user query task includes: Performing polling query on the cache database to obtain polling query data; Real-time detection of whether the data volume of the polling query data reaches a preset query data volume, or real-time detection of whether the current query duration reaches a preset query duration threshold; If so, the polling query data is determined as the data query result of the user query task, and the polling query data is returned.

[0008] In one embodiment, after the step of performing polling query on the cache database until the data query result of the user query task is obtained and returned, the step further includes: Upon receiving a service analysis request, determining a service auxiliary field according to the service analysis request; Adding the business auxiliary field to the cache database; Return to the step of performing polling query on the cache database until the data query result of the user query task is obtained and returned.

[0009] In one embodiment, the step of obtaining and returning the historical query result of the target query task based on the historical query record if it exists includes: If so, determining the current query state of the target hash value; If the current query state is a query completion state, the historical query result of the target query task is acquired and returned based on the historical query record.

[0010] In addition, to achieve the above purpose, the present application also proposes a data query device, the device comprising: A parameter determination module, used to determine the query parameters of the user query task when receiving the user query task; A parameter splicing module, used for performing parameter splicing on the query parameters to obtain spliced ​​query parameters when the number of parameters of the query parameters is greater than a preset parameter number threshold; A hash value determination module, used to generate a target hash value corresponding to the user query task based on the concatenated query parameters by using a preset hash algorithm; The data query module is used to determine whether there is a target query task identical to the user query task in the historical query record according to the target hash value, and if so, obtain and return the historical query result of the target query task based on the historical query record.

[0011] In addition, to achieve the above objectives, the present application also proposes a data query device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data query method described above.

[0012] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data query method described above are implemented.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data query method described above are implemented.

[0014] The present application provides a data query method, which discloses that when a user query task is received, query parameters of the user query task are determined; when the number of parameters of the query parameters is greater than a preset parameter number threshold, the query parameters are spliced ​​to obtain the spliced ​​query parameters; a target hash value corresponding to the user query task is generated based on the spliced ​​query parameters by a preset hash algorithm; according to the target hash value, it is judged whether there is a target query task identical to the user query task in the historical query record, and if so, the historical query result of the target query task is obtained and returned based on the historical query record; compared with the prior art, when a large number of users repeatedly perform massive data queries, the pressure on the big data query engine is increased, resulting in low data query efficiency. Since the present invention can directly obtain the corresponding historical query result from the historical query record when there is a query task identical to the user query task in the historical query record, the technical problem in the prior art that when users repeatedly perform data queries, the query pressure on the big data query engine is increased, resulting in reduced data query efficiency is solved, thereby improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 A flowchart of the first embodiment of the data query method of this application is provided; Figure 2 A flowchart of the second embodiment of the data query method of this application is provided; Figure 3 A flowchart of the third embodiment of the data query method of this application is provided; Figure 4 This is the overall flow chart of the data query method for this application; Figure 5 This is a schematic diagram of the module structure of the data query device according to an embodiment of the present application; Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the data query method in the embodiment of the present application.

[0018] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0020] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0021] The main solution of the embodiment of the present application is: when a user query task is received, a target hash value corresponding to the user query task is generated; based on the target hash value, it is determined whether there is a target query task identical to the user query task in the historical query record; if so, the historical query result of the target query task is obtained and returned based on the historical query record.

[0022] Although the use of various big data components for data query in the prior art can effectively improve data query efficiency, when a large number of users perform massive data queries, the query rate per second of the big data query engine is low. If users repeatedly query massive data, it will increase the pressure on the big data query engine, resulting in reduced query efficiency, thereby affecting the user experience.

[0023] The present application provides a solution that can directly obtain the corresponding historical query results from the historical query records when there is a query task identical to the user's query task in the historical query records, thereby solving the technical problem in the prior art that when users repeatedly perform data queries, the query pressure on the big data query engine will increase, resulting in reduced data query efficiency, thereby improving the user experience.

[0024] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data query device, etc. The following takes a data query device as an example (hereinafter referred to as the device) to illustrate this embodiment and the following embodiments.

[0025] Based on this, the present application embodiment provides a data query method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the data query method of the present application.

[0026] In this embodiment, the data query method includes steps S10 to S40: Step S10: upon receiving a user query task, determining a query parameter of the user query task.

[0027] It should be understood that the above user query task can be a data query task currently submitted by the user to the device. In this embodiment, the query task submitted by the user can carry a specific query statement (such as an SQL statement), parameters or keywords, so that the big data query engine in the device can perform data query based on these query statements or keywords.

[0028] It should be understood that the above query parameters may be parameters for performing data query carried in the query task submitted by the user, such as name, time, etc., and this embodiment does not limit this.

[0029] Step S20: When the number of parameters of the query parameters is greater than a preset parameter number threshold, the query parameters are concatenated to obtain concatenated query parameters.

[0030] It can be understood that the above number of parameters is the number of parameters carried in the query task submitted by the user.

[0031] In this embodiment, the preset parameter quantity threshold may be set to 1, that is, if there are multiple parameters in the user query task, these query parameters may be concatenated to obtain concatenated post-query parameters.

[0032] Step S30: Generate a target hash value corresponding to the user query task based on the concatenated query parameters using a preset hash algorithm.

[0033] It should be noted that the above-mentioned preset hash algorithm can be an algorithm for transforming query parameters of arbitrary length into an output of fixed length through a hash function, for example, the Average Hash Algorithm (AHA), the Secure Hash Algorithm (SHA family), etc. In this embodiment, the device can use the secure hash algorithm SHA-256 to calculate the 64-bit hash value corresponding to the query parameters of the user query task to convert the query parameters into a fixed hash value to obtain the above-mentioned target hash value.

[0034] It is understandable that the target hash value may be a parameter for uniquely identifying a user query task. In this embodiment, the device may generate a unique target hash value for the query task according to the parameters carried in the query task submitted by the user, so as to distinguish the query task submitted by the current user from other query tasks.

[0035] It should be understood that after concatenating all query parameters of the user query task, the device can calculate the 64-bit hash value corresponding to the concatenated query parameters through a hash algorithm to obtain a target hash value. In addition, if the user query task has only one query parameter, the 64-bit hash value corresponding to the query parameter can be directly calculated through a hash algorithm to obtain the target hash value.

[0036] Step S40: judging whether there is a target query task identical to the user query task in the historical query record according to the target hash value; if so, obtaining and returning the historical query result of the target query task based on the historical query record.

[0037] It should be noted that the above historical query record may be a record of data query situations in the device over a period of time in the past. In this embodiment, the content recorded in the historical query record may include but is not limited to the hash value of the query task, the query time, the query user, the query result, etc. Among them, the above target query task may be a query task that exists in the historical query record and is the same as the query task submitted by the current user.

[0038] In this embodiment, after receiving the query task submitted by the user, the device can generate a target hash value corresponding to the query task based on the query parameters corresponding to the query task, and then search in the historical query records based on the target hash value of the user query task to determine whether there is a historical query task in the historical query record with a hash value that is the same as the target hash value. If so, the historical query task can be determined as the target query task in the historical query record that is the same as the user query task.

[0039] It is understandable that the above historical query results can be the query results of the target query tasks stored in the historical query records. In actual applications, if there is a target query task that is the same as the task currently submitted by the user in the historical query records, the historical query results corresponding to the target query task can be directly extracted from the historical query records, and the historical query results can be used as the query results of the query task currently submitted by the user, so that the big data search engine in the device can directly obtain the corresponding query results when the target query task that is the same as the query task currently submitted by the user is hit in the historical query records, so as to reduce the number of queries, thereby effectively reducing the data query pressure of the big data query engine and improving the data query efficiency.

[0040] Specifically, the step S30 includes: if it exists, determining the current query state of the target hash value; if the current query state is a query completion state, acquiring and returning the historical query result of the target query task based on the historical query record.

[0041] It should be understood that the above current query state can be a feature used to characterize the current query progress of the query task. The current query state in this embodiment can include but is not limited to the query completion state, the querying state, and the query failure state. In this embodiment, the state of the hash value can be used to characterize the query state of the query task corresponding to it. For example, if the state of the hash value is the query completion state, it means that the query task corresponding to the hash value has been queried; if the state of the hash value is the querying state, it means that the query task corresponding to the hash value is currently being queried; if the state of the hash value is the query failure state, it means that the query task corresponding to the hash value has failed.

[0042] In actual applications, when there is a target query task that is the same as the query task currently submitted by the user in the historical query record, since the hash value of the target query task is the same as the target hash value of the query task currently submitted by the user, the device can determine the current query status of the target query task based on the query status of the target hash value. For example, if the target hash value is currently in the query completion state, it means that the target query task has been queried. At this time, the historical query results of the target query task can be directly obtained from the historical query record and returned as the query results of the user query task. In addition, if the target hash value is currently in the querying state, it means that the target query task has not yet been queried. At this time, the target query task can be continuously polled and the query results of the target query task that has been queried can be returned in real time until all the query results of the target query task are returned; if the target hash value is currently in the query failure state, the device can return a prompt message of data query failure, so that the user can re-query the data when receiving the prompt message of data query failure.

[0043] The present embodiment provides a data query method, which discloses that when a user query task is received, query parameters of the user query task are determined; when the number of parameters of the query parameters is greater than a preset parameter number threshold, the query parameters are spliced ​​to obtain the spliced ​​query parameters; a target hash value corresponding to the user query task is generated based on the spliced ​​query parameters by a preset hash algorithm; according to the target hash value, it is determined whether there is a target query task identical to the user query task in the historical query record, and if so, the historical query result of the target query task is obtained and returned based on the historical query record; compared with the prior art, when a large number of users repeatedly perform massive data queries, the pressure on the big data query engine is increased, resulting in low data query efficiency. Since the present embodiment can directly obtain the corresponding historical query results from the historical query record when there is a query task identical to the user query task in the historical query record, the technical problem in the prior art that when users repeatedly perform data queries, the query pressure on the big data query engine is increased, resulting in reduced data query efficiency is solved, thereby improving the user experience.

[0044] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , Figure 2 A flowchart diagram of the second embodiment of the data query method of this application is provided.

[0045] In this embodiment, after the step of determining whether there is a target query task identical to the user query task in the historical query record according to the target hash value, the following step is further included: Step S401: If not present, determining the data query structure of the user query task based on the query parameters.

[0046] It should be noted that the above data query structure can be a data structure used to query data that meets user-specific conditions. In this embodiment, the data query structure corresponding to the task can be defined in the form of query parameters + aggregation parameters + analysis parameters, wherein the query parameters can be the parameters required when using functions or methods to perform data query operations, and these parameters are used to determine the specific content and conditions of the query. In this embodiment, it can be divided into single-parameter query and multi-parameter query according to the number of parameters. For example, in SQL query, the WHERE clause can be used to specify the query conditions, and the values ​​in these conditions are query parameters. Specifically, if the employee information with ID 10 needs to be queried, "10" is the query parameter. Aggregation parameters can be parameters used to summarize and calculate query results, and are usually used together with aggregate functions, wherein aggregate functions are a type of functions used in the database to calculate multiple rows and return a single result, such as COUNT, SUM, AVG, MIN, MAX, etc. These functions can summarize, count and calculate data, and are often used to extract summary information about data sets. In this embodiment, the device can specify the target column or expression to be calculated, as well as the calculation method (such as sum, count, average, etc.) through aggregation parameters. For example, when the SUM function is used to calculate the sum of a field, the field name is the aggregation parameter. Analysis parameters can be specific content and conditions for data analysis. These parameters may include: data range, analysis indicators, filtering conditions, etc. In this embodiment, analysis parameters can be used to help users more accurately understand the characteristics and trends of data, so as to make more informed decisions. For example, in Python data analysis, the pandas library can be used to read data, and the data can be filtered, sorted, grouped, and other operations can be performed by specifying analysis parameters such as column names and filtering conditions.

[0047] Step S402: performing data query on the user query task based on the data query structure and the target hash value, and obtaining and returning the data query result of the target query task.

[0048] In this embodiment, the device can design all queries as a unified data query structure, define query parameters, aggregation parameters and analysis parameters, and at the same time, define the response data body as a unified structure, covering all required fields involved in the business, so that when performing data queries on user query tasks later, the data query structure corresponding to the query task can be generated according to the query parameters, query conditions, query methods (such as Search function) and other information of the user query task, and data query can be performed based on the data query structure. Once the data query result of the target query task is obtained, it will be returned immediately.

[0049] In actual applications, if there is no target query task identical to the query task currently submitted by the user in the historical query records, the device can first record the status of the user query task as the querying status, and generate a pre-established unified data query structure based on the query parameters, query conditions, and query methods of the user query task. The big data engine can then perform data query on the user query task based on the data query structure, and ultimately obtain the data query results of the user query task.

[0050] In this embodiment, it is disclosed that if there is no target query task identical to the user query task in the historical query records, the data query structure of the user query task is determined based on the query parameters; a data query is performed on the user query task based on the data query structure and the target hash value, and the data query result of the target query task is obtained and returned. Since in this embodiment, when there is no target query task identical to the user query task in the historical query records, a data query can be performed based on the data query structure and the target hash value corresponding to the user query task. Through this design of normalization of query and analysis entries, the complexity of business queries can be reduced and the data query process can be simplified.

[0051] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 3 , Figure 3 A flowchart diagram of the third embodiment of the data query method of this application is provided.

[0052] In this embodiment, step S402 includes steps S402a to S402e: Step S402a: based on the data query structure, the user query task is segmented to obtain a plurality of segmented query tasks.

[0053] It should be noted that the above-mentioned sharded query task may be a plurality of smaller query tasks obtained by splitting the user query task according to a set rule, wherein these sharded query tasks may be stored on different database nodes (or servers).

[0054] In this embodiment, the device can split the user query task according to the data query structure. For example, if the query parameter part in the data query structure defines that user information corresponding to users with user IDs of 1 to 100 needs to be queried, then the device can slice the query task of "querying user information corresponding to users with user IDs of 1 to 100" to obtain 10 shard query tasks, which are respectively for querying user information corresponding to users with user IDs of 1 to 10, 10 to 20, 20 to 30, 30 to 40, 40 to 50, 50 to 60, 60 to 70, 70 to 80, 80 to 90 and 90 to 100.

[0055] Step S402b: Perform data query on the shard query task based on the target hash value and a preset data query method to obtain a shard data query result.

[0056] It should be noted that the above-mentioned preset data query method can be a pre-set method for performing data query, including: query method Query, statistical analysis method Statistic, deduplication query method Distinct and search method Search, wherein Query in data query can be an information request submitted by a user to a database or search engine. In actual application, a user can obtain the required data or information by entering a specific query statement or keyword. For example, in a database, Query may be an SQL statement for retrieving, inserting, updating or deleting data. Statistic in data query and processing can be a process of statistically analyzing data, such as descriptive statistics, #inferential statistics, regression analysis, etc. Distinct is used to return unique and different values ​​in SQL query, wherein when a user needs to perform deduplication processing on certain fields in the query results, the Distinct keyword can be used. For example, when querying a column in a table, if the user only wants to obtain non-duplicate values, the Distinct keyword can be used in the SQL statement. Search can include a preliminary query request submitted by a user, and further screening and filtering based on the preliminary query results, which is usually referred to as secondary search or result filtering. For example, in a search engine, a user may first enter a broad keyword to search, and then use filters or further enter more specific keywords on the search results page to narrow the search scope. In addition, Search also involves the function of fuzzy retrieval, which allows users to search without being sure of the specific information. Fuzzy retrieval is usually implemented by supporting wildcards, partial matching, or similarity matching. For example, in a search engine, a user can use wildcards such as asterisks or question marks to search for strings containing specific characters or patterns. In addition, some search engines also provide fuzzy retrieval based on similarity matching, allowing users to search for content that is semantically close or related to the input keywords.

[0057] In this embodiment, the device can call the Search method to perform data query on the shard query task through the target hash value corresponding to the user query task, and obtain the shard data query results corresponding to each shard query task.

[0058] It is understandable that after obtaining the query results of all shard query tasks corresponding to the user query task, the device can integrate these shard data query results, and finally obtain and return the data query results corresponding to the user query task.

[0059] Step S402c: Load the shard data query result into the cache database based on asynchronous loading technology.

[0060] It should be noted that the above-mentioned asynchronous loading technology can be a technology for loading data in an asynchronous manner. In practical applications, the asynchronous loading technology allows updating of a specific part of a web page by exchanging local data with the server in the background without reloading the entire web page. In this embodiment, when querying a user query task, an asynchronous waiting method for results can be adopted based on the asynchronous loading technology, that is, once the target hash value of the user query task is generated, the query result is returned immediately without waiting for all the data of the user query task to be queried before returning.

[0061] It can be understood that the cache database may be a database for storing query results corresponding to query tasks of users.

[0062] Step S402d: Return the query result of the shard data in the cache database in real time.

[0063] In this embodiment, during the asynchronous processing of data query, the big data query engine can perform shard query on the user query task. Once the query results are returned, the returned query results can be stored in the cache database. At the same time, the page can load the query results in the cache database in real time to realize the real-time return of all shard data query results in the cache database.

[0064] Step S402e: performing polling query on the cache database until the data query result of the user query task is obtained and returned.

[0065] It should be noted that when querying massive data, since massive data is generally distributed and stored on multiple machines, it will be distributed to various servers for processing when searching, and the query results will be returned in batches. In this case, if you wait for the query results of all servers to be returned before responding to the business, and some servers query quickly and some servers query slowly, the waiting time for the user to query will be very long. In addition, if the data reading fails, the user's current query page will be stuck, thus affecting the user experience. Therefore, in this implementation, data query can be performed based on abnormal loading technology, and the sharded data query results of the user query task can be asynchronously returned, and the returned query results can be immediately stored in the cache database. The user query page can gradually load the currently returned data query results into the page by continuously polling the query database to avoid the phenomenon of page jamming, thereby improving the user experience.

[0066] Further, the step S402e includes: performing polling query on the cache database to obtain polling query data; detecting in real time whether the data volume of the polling query data reaches the preset query data volume, or detecting in real time whether the current query duration reaches the preset query duration threshold; if so, determining the polling query data as the data query result of the user query task, and returning the polling query data.

[0067] It should be understood that polling the cache database may be a process of periodically querying the cache database to obtain the latest data in the cache database, and the polling query data may be a new query result obtained after each query on the cache database.

[0068] It can be understood that the above preset query data volume may be the maximum data volume required for the query task currently submitted by the user.

[0069] In actual applications, if the preset query data volume required for the user query task is 5,000, the device can obtain data from the cache database through polling queries when performing data queries on the user query task, and detect in real time whether the amount of data currently obtained has reached 5,000. If so, all polling query data currently obtained from the cache database can be used as the data query results of the user query task and returned.

[0070] In this embodiment, if the amount of data query is large, it can be detected in real time whether the current query duration reaches the preset query duration threshold; if so, the polling query data is determined as the data query result of the user query task, and the polling query data is returned.

[0071] It should be understood that the above-mentioned preset query duration threshold can be the maximum duration for the big data query engine to query the query task. In actual applications, if the amount of data is large but the user query task does not require so much data, and continuous querying will increase the pressure on the query engine, therefore, a query duration threshold can be set in advance, and when the current query duration reaches the preset query duration threshold, the query of the user query task is stopped, and all the currently acquired query data is determined as the query result of the user query task and returned.

[0072] Furthermore, after step S402e, it also includes: when receiving a business analysis request, determining the business auxiliary field according to the business analysis request; adding the business auxiliary field to the cache database; returning to the step of polling the cache database until obtaining and returning the data query result of the user query task.

[0073] It should be noted that the above-mentioned business auxiliary field may be a field used to assist users in performing business analysis.

[0074] In actual applications, the number and format of fields in the cache database are usually fixed. If a user needs to perform a multi-dimensional query or analysis on a certain business (such as querying the geographical location of the IP, the location of the mobile phone number, etc.), and because there are usually no redundant fields stored in the underlying data structure, the user's query needs cannot be met at this time. Therefore, this embodiment can determine the user's data analysis needs based on the business analysis request submitted by the user, and determine the corresponding business auxiliary fields (such as IP address, latitude and longitude, etc.) based on the data analysis needs. Then, after the big data query engine returns the query results, the device can add these business auxiliary fields to the cache database, so that the data corresponding to the auxiliary fields in the query results can be added to the corresponding fields later, so as to support various multi-dimensional analysis, fuzzy retrieval and field sorting, etc.

[0075] In the specific implementation, refer to Figure 4 , Figure 4 This is the overall flow chart of the data query method of this application. Figure 4 As shown, when the user needs to perform a data query, the corresponding query task can be submitted in the device's big data search engine. After receiving the query task currently submitted by the user, the device can generate a unique hash value for the query task based on the query parameters of the current query task, and query whether the user has previously submitted the same target query task from the historical query record based on the hash value. If so, the query result corresponding to the target query task is directly obtained from the historical query record based on the hash value of the current query task. In addition, if there is no query task identical to the user's query task in the historical query record, the state of the hash value of the user's query task can be recorded as the querying state, and the Search method can be called through the hash value to perform data query on the user's query task. At this time, the underlying server can perform a shard query on the user's query task, obtain the shard task query results corresponding to each shard query task, and asynchronously load these shard task query results into the cache database. At this time, the page can continuously poll the query cache database until all query results of the user's query task are returned.

[0076] In this embodiment, it is disclosed to perform task sharding on a user query task based on a data query structure to obtain a number of shard query tasks; perform data query on the shard query task based on a preset data query method through a target hash value to obtain a shard data query result; load the shard data query result into a cache database based on an asynchronous loading technology; perform polling query on the cache database until the data query result of the user query task is obtained and returned, thereby realizing shard query of the user query task. At the same time, this embodiment can perform asynchronous processing on the shard query task and poll and load the query result, so that the query page can respond to data quickly to avoid page jams that affect user experience.

[0077] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data query method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0078] This application also provides a data query device, please refer to Figure 5 , the data query device comprises: A parameter determination module 10, configured to determine the query parameters of the user query task when receiving the user query task; A parameter concatenation module 20, configured to concatenate the query parameters to obtain concatenated query parameters when the number of parameters of the query parameters is greater than a preset parameter number threshold; A hash value determination module 30, configured to generate a target hash value corresponding to the user query task based on the concatenated query parameters by using a preset hash algorithm; The data query module 40 is used to determine whether there is a target query task identical to the user query task in the historical query record according to the target hash value, and if so, obtain and return the historical query result of the target query task based on the historical query record.

[0079] The data query device provided by the present application adopts the data query method in the above embodiment, which can solve the technical problem in the prior art that when users repeatedly perform data queries, the query pressure of the big data query engine will increase, resulting in reduced data query efficiency. Compared with the prior art, the beneficial effects of the data query device provided by the present application are the same as the beneficial effects of the data query method provided by the above embodiment, and the other technical features in the data query device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0080] The present application provides a data query device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data query method in the above-mentioned embodiment 1.

[0081] Reference below Figure 6, which shows a schematic diagram of the structure of a data query device suitable for implementing an embodiment of the present application. The data query device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The data query device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0082] like Figure 6 As shown, the data query device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data query device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data query device to communicate with other devices wirelessly or by wire to exchange data. Although the data query device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0083] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0084] The data query device provided by the present application adopts the data query method in the above embodiment to solve the technical problem of data query. Compared with the prior art, the beneficial effects of the data query device provided by the present application are the same as the beneficial effects of the data query method provided by the above embodiment, and the other technical features in the data query device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0085] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0086] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0087] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data query method in the above-mentioned embodiment.

[0088] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0089] The computer-readable storage medium may be included in the data query device; or may exist independently without being assembled into the data query device.

[0090] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the data query device, the data query device: generates a target hash value corresponding to the user query task when receiving a user query task; determines whether there is a target query task identical to the user query task in the historical query record based on the target hash value; if so, obtains and returns the historical query result of the target query task based on the historical query record.

[0091] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0092] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0093] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0094] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data query method, and can solve the technical problem in the prior art that when users repeatedly perform data queries, the query pressure of the big data query engine will increase, resulting in reduced data query efficiency. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data query method provided in the above-mentioned embodiment, and will not be repeated here.

[0095] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data query method when executed by a processor.

[0096] The computer program product provided by the present application can solve the technical problem in the prior art that when users repeatedly perform data queries, the query pressure of the big data query engine will increase, resulting in reduced data query efficiency. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the data query method provided by the above embodiment, and will not be repeated here.

[0097] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data query method, characterized in that: The method includes: When receiving a user query task, determining a query parameter of the user query task; When the number of parameters of the query parameters is greater than a preset parameter number threshold, performing parameter splicing on the query parameters to obtain spliced ​​query parameters; Generate a target hash value corresponding to the user query task based on the concatenated query parameters by using a preset hash algorithm; It is determined whether there is a target query task identical to the user query task in the historical query record according to the target hash value; if so, a historical query result of the target query task is obtained and returned based on the historical query record.

2. The method according to claim 1, characterized in that After the step of determining whether there is a target query task identical to the user query task in the historical query record according to the target hash value, the method further includes: If not, determining the data query structure of the user query task based on the query parameters; Perform data query on the user query task based on the data query structure and the target hash value, and obtain and return the data query result of the target query task.

3. The method according to any one of claims 1 to 2, characterized in that The step of performing data query on the user query task based on the data query structure and the target hash value, and obtaining and returning the data query result of the target query task includes: Based on the data query structure, the user query task is segmented to obtain a plurality of segmented query tasks; Performing data query on the shard query task based on the target hash value and a preset data query method to obtain a shard data query result; Loading the shard data query results into the cache database based on asynchronous loading technology; Return the query result of the shard data in the cache database in real time; The cache database is polled and queried until the data query result of the user query task is obtained and returned.

4. The method according to claim 3, characterized in that The step of performing polling query on the cache database until obtaining and returning the data query result of the user query task includes: Performing polling query on the cache database to obtain polling query data; Real-time detection of whether the data volume of the polling query data reaches a preset query data volume, or real-time detection of whether the current query duration reaches a preset query duration threshold; If so, the polling query data is determined as the data query result of the user query task, and the polling query data is returned.

5. The method according to claim 3, characterized in that After the step of performing polling query on the cache database until the data query result of the user query task is obtained and returned, the method further includes: Upon receiving a service analysis request, determining a service auxiliary field according to the service analysis request; Adding the business auxiliary field to the cache database; Return to the step of performing polling query on the cache database until the data query result of the user query task is obtained and returned.

6. The method according to claim 1, characterized in that If so, the step of obtaining and returning the historical query result of the target query task based on the historical query record comprises: If so, determining the current query state of the target hash value; If the current query state is a query completion state, the historical query result of the target query task is acquired and returned based on the historical query record.

7. A data query device, characterized in that: The device comprises: A parameter determination module, used to determine the query parameters of the user query task when receiving the user query task; A parameter splicing module, used for performing parameter splicing on the query parameters to obtain spliced ​​query parameters when the number of parameters of the query parameters is greater than a preset parameter number threshold; A hash value determination module, used to generate a target hash value corresponding to the user query task based on the concatenated query parameters by using a preset hash algorithm; The data query module is used to determine whether there is a target query task identical to the user query task in the historical query record according to the target hash value, and if so, obtain and return the historical query result of the target query task based on the historical query record.

8. A data query device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data query method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data query method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data query method according to any one of claims 1 to 6 are implemented.

Images