Data query method and device, electronic equipment and storage medium

Abstract

The invention relates to a data query method and device, electronic equipment and a storage medium. The data query method comprises the steps that a query statement corresponding to a query request is acquired, and query elements of the query statement are determined; judging whether the query element exists in a query cache or not, wherein the query cache is used for storing historical query elements and historical query results; in response to the query element existing in the query cache, obtaining a query result corresponding to the query element in the query cache; and in response to the situation that no or part of the query elements exist in the query cache, performing query based on the data sources corresponding to the query elements, and obtaining a query result. According to the method and the device, the query result is determined to be obtained or the query result is obtained after query, so that the query performance is improved, and the query pressure on the cache data is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a data query method, device, electronic device and storage medium. Background Art

[0002] When it is necessary to associate data or expand dimensions, related technologies usually require Structured Query Language (SQL) or other coding methods to process data. Association processing is required at each location where the dimension is expanded. Low performance requires high manpower and development costs. In the data calculation stage, changes in calculation indicators and dimensions usually require re-issuing calculation tasks and manual intervention. In the data query stage, there is a problem of low cache hit rate when the query as a whole is used as a cache object. How to reduce labor costs and improve the efficiency of data processing and data query still needs to be solved. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a data query method, device, electronic device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a data query method is provided, including obtaining a query statement corresponding to a query request, and determining a query element of the query statement; judging whether the query element exists in a query cache, the query cache being used to store historical query elements and historical query results; in response to the query element existing in the query cache, obtaining a query result corresponding to the query element in the query cache; in response to the query element not existing or partially existing in the query cache, querying based on a data source corresponding to the query element, and obtaining a query result.

[0005] In one embodiment, the query element includes a first query element; obtaining the query result corresponding to the query element in the query cache includes: obtaining the first query element in the query element and using the first query element as a query identifier; based on the first query element, obtaining the query result corresponding to the first query element in the query cache.

[0006] In one implementation, querying based on the data source corresponding to the query element and obtaining the query result includes: obtaining a metadata table corresponding to the data source, the metadata table including a data field, a calculation type, and an update time field, wherein the data field is a field corresponding to the data source, and the calculation type is a calculation type preset for the data field; based on the data field, the calculation type, and the update time field, obtaining updated data to obtain the query result.

[0007] In one embodiment, the query element includes a second query element, and the query result in the query cache is obtained in the following manner: obtaining the query result of the first query element, the first query element including one or more elements; storing each query result in the query cache according to the format of the second query element, the first query element, and the query result of the first query element.

[0008] In one embodiment, obtaining the metadata table corresponding to the data source includes: obtaining the data source in the query statement and multiple data fields corresponding to the data source; determining that a dimension extension field corresponding to the data field does not exist in the hash model structure, mapping the data field to the hash model structure to obtain the dimension extension field, and the hash model structure is used to store the extension field corresponding to the data field; setting a calculation type for the dimension extension field, and generating a metadata table including the calculation type.

[0009] In one embodiment, mapping the data field to the hash model structure to obtain the dimension extension field includes: determining an extension rule corresponding to the data source; traversing multiple data fields in the data source, and expanding according to the data fields matching the extension rule to obtain the dimension extension field.

[0010] In one implementation, the method further includes: hot updating the extension rules.

[0011] According to a second aspect of an embodiment of the present disclosure, a data query device is provided, including a determination unit, configured to obtain a query statement corresponding to a query request and determine a query element of the query statement; a processing unit, configured to determine whether the query element exists in a query cache, the query cache being used to store historical query elements and historical query results; an acquisition unit, configured to, in response to the query element existing in the query cache, acquire a query result corresponding to the query element in the query cache; in response to the query element not existing or partially existing in the query cache, perform a query based on a data source corresponding to the query element and acquire a query result.

[0012] In one embodiment, the query elements include a first query element; the acquisition unit acquires the query result corresponding to the query element in the query cache in the following manner: acquiring the first query element in the query elements and using the first query element as a query identifier; based on the first query element, acquiring the query result corresponding to the first query element in the query cache.

[0013] In one embodiment, the acquisition unit performs a query based on the data source corresponding to the query element and obtains the query result in the following manner: obtain a metadata table corresponding to the data source, the metadata table including a data field, a calculation type, and an update time field, wherein the data field is a field corresponding to the data source, and the calculation type is a calculation type preset for the data field; based on the data field, the calculation type, and the update time field, obtain updated data to obtain the query result.

[0014] In one embodiment, the query element includes a second query element, and the query result in the query cache is obtained in the following manner: obtaining the query result of the first query element, the first query element including one or more elements; storing each query result in the query cache according to the format of the second query element, the first query element, and the query result of the first query element.

[0015] In one embodiment, the acquisition unit acquires the metadata table corresponding to the data source in the following manner: acquiring the data source in the query statement and multiple data fields corresponding to the data source; determining that a dimension extension field corresponding to the data field does not exist in the hash model structure, mapping the data field to the hash model structure to obtain the dimension extension field, and the hash model structure is used to store the extension field corresponding to the data field; setting a calculation type for the dimension extension field, and generating a metadata table including the calculation type.

[0016] In one embodiment, the processing unit maps the data field to the hash model structure to obtain the dimension extension field in the following manner: determining the extension rule corresponding to the data source; traversing multiple data fields in the data source, and expanding according to the data fields matching the extension rule to obtain the dimension extension field.

[0017] In one implementation, the processing unit is further configured to: perform hot update on the extension rule.

[0018] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the data query method described in the first aspect or any one of the implementations of the first aspect.

[0019] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the data query method described in the first aspect or any one of the embodiments of the first aspect.

[0020] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: obtaining a query statement corresponding to a query request, determining a query element corresponding to the query statement, determining whether a query element exists in a query cache, and determining whether to obtain a query result or obtain a query result after a query based on whether the query element exists, thereby improving query performance and reducing query pressure on cached data.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] Figure 1 The figure is a flow chart of a data query method according to an exemplary embodiment.

[0024] Figure 2 The figure is a flowchart of a method for obtaining query results according to an exemplary embodiment.

[0025] Figure 3 The figure is a flowchart of a method for obtaining query results according to an exemplary embodiment.

[0026] Figure 4 The figure is a flowchart of a method for determining a query result according to an exemplary embodiment.

[0027] Figure 5 The figure is a flowchart of a method for obtaining a metadata table according to an exemplary embodiment.

[0028] Figure 6 The present invention is a flowchart of a method for obtaining a dimension extension field according to an exemplary embodiment.

[0029] Figure 7 The figure is a flow chart of a data query method according to an exemplary embodiment.

[0030] Figure 8 It is a schematic diagram showing a rule path query interaction design according to an exemplary embodiment.

[0031] Fig. 9 It is a schematic diagram of a data query method according to an exemplary embodiment.

[0032] Fig.10 The invention is a block diagram of a data query device according to an exemplary embodiment.

[0033] Fig.11It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0035] Apache Flink and Apache Druid big data components are commonly used in data processing and data query. Apache Flink is a stream processing framework that supports low-latency and high-throughput data processing. Apache Druid is a high-performance analytical database designed for fast query and real-time analysis of large-scale data.

[0036] In the related technology, in the data processing stage, the real-time dimension table is associated through SQL or join operations, and the association processing needs to be performed in each place where the dimension is expanded, which has high development costs and low performance. In the data calculation stage, for the changes in indicators and dimensions in the data calculation stage, the indicators are usually calculated through computing engines such as Flink, which requires manual adjustment of the calculation logic, restart of the service to re-publish the task, and manual maintenance. In the data query stage, the query as a whole is usually used as a cache object, resulting in a low cache hit rate, and then the query pressure problem of cached data occurs.

[0037] In view of this, the present disclosure proposes a data query method that can achieve one-time acquisition of cache results, improve cache hits and high-performance cache data acquisition. And when performing data calculations, a method for automatically processing calculation logic changes is provided to reduce manual maintenance costs. In the data processing stage, dimension expansion can be automatically completed through simple configuration, and changes can take effect in real time.

[0038] Figure 1 FIG. 1 is a flow chart of a data query method according to an exemplary embodiment. Figure 1 As shown, the method includes steps S11 to S13b.

[0039] In step S11, a query statement corresponding to the query request is obtained, and query elements of the query statement are determined.

[0040] In the embodiment of the present disclosure, in response to detecting a query request, a query statement corresponding to the query request is obtained, wherein the query statement includes a query element to be queried. Exemplarily, the query element may be other elements such as a time series (queryType), a data source (dataSource), a granularity (granularity), a time range (intervals), or an aggregation operation list (aggregations). The aggregation operation list may include a type (type), a field name (FieldName), and a name (name), etc.

[0041] In step S12, it is determined whether there is a query element in the query cache.

[0042] In the disclosed embodiment, the query cache is used to store historical query elements and historical query results, for example, the query element determined based on the query request when the query request was obtained last time, and the query result obtained after querying the query element.

[0043] In step S13a, in response to the query element existing in the query cache, a query result corresponding to the query element in the query cache is obtained.

[0044] In the embodiment of the present disclosure, the existence of query elements in the query cache can be understood as, after obtaining the query request, using the individual query elements in the query request as the query key to determine whether the key in this query request exists in the query cache. If so, it is a cache hit, and the query result corresponding to the key in the query cache can be obtained.

[0045] In step S13b, in response to the query cache not containing or partially containing the query element, a query is performed based on the data source corresponding to the query element, and a query result is obtained.

[0046] In the disclosed embodiment, the query element includes at least two elements. Therefore, for the currently acquired query request, there are the following two situations: the query element corresponding to the current query request does not exist in the query cache, and the query element corresponding to some of the current query request exists in the query cache.

[0047] In the embodiment of the present disclosure, for the above two situations, if the query element corresponding to the current query request does not exist in the query cache, each query element can be queried based on the query element corresponding to the current query request to obtain the query result. If some query elements corresponding to the current query request exist in the query cache, the query elements that exist can hit the cache in the query cache, and the elements other than the existing query elements continue to be queried, so as to obtain the query result through the query.

[0048] In the disclosed embodiment, when performing data query, the query result of the hit cache can be directly obtained, which reduces the query pressure of the cache data and improves the query performance.

[0049] In the embodiment of the present disclosure, in response to the existence of a query element in the query cache, the query result may be obtained in the following manner. Figure 2 FIG. 1 is a flow chart showing a method for obtaining query results according to an exemplary embodiment. Figure 2 As shown, the method includes steps S21 to S22.

[0050] In step S21, a first query element in the query elements is obtained, and the first query element is used as a query identifier.

[0051] In the disclosed embodiment, the first query element may be an element related to time, or may be defined as other elements. Exemplarily, the first query element is described as an element related to time. In response to detecting a query request, and the query request contains a time span element, the time range included in the time span is obtained. For example, intervals: "2018-08-14T00:00 / 2018-08-16T23:59:59". It can be understood as querying data between 0:00 on August 14, 2018 and 23:59:59 on August 16, 2018. The time span can be split into multiple finer-grained time ranges, and each of the split time ranges is used as a query identifier and queried separately, that is, the time span is split into 2018-8-14, 2018-8-15 and 2018-8-16 according to the day granularity, and a separate query is performed based on the split time range to obtain the query results under each granularity, so that each query result is aggregated and the final query result is returned.

[0052] In step S22, based on the first query element, a query result corresponding to the first query element in the query cache is obtained.

[0053] In the disclosed embodiment, in response to detecting a query request, the first query element in the query request is determined. It is determined whether all first query elements of the current query request hit the query cache. If they can hit the cache, the query result that hits can be directly returned to obtain the query result corresponding to the first query element.

[0054] In the disclosed embodiment, based on the query result corresponding to the first query element in the query cache, the query result can be directly obtained after hitting the query cache, which improves the data query rate and reduces the query pressure problem of the data cache.

[0055] In the embodiment of the present disclosure, in response to the query cache not containing the query element or containing part of the query element, the query may be performed in the following manner. Figure 3 FIG. 1 is a flow chart showing a method for obtaining query results according to an exemplary embodiment. Figure 3 As shown, the method includes steps S31 to S32.

[0056] In step S31, a metadata table corresponding to the data source is obtained, wherein the metadata table includes data fields, calculation types, and update time fields.

[0057] In the embodiment of the present disclosure, in response to obtaining a query request, a query element is determined. The query element includes a first query element and a second query element. The first query element may be a time span element, and the second query element may be an element other than the time span element. If the query element does not exist in the query cache or some query elements exist, the query element that does not hit the cache may continue to be queried to obtain the query result.

[0058] In the disclosed embodiment, for a query element that does not hit the cache, the data source of the query request corresponding to the query element is obtained. It can be understood that the data source corresponds to a metadata table, and calculations can be performed based on the data fields and calculation types in the metadata table to obtain updated data and obtain query results.

[0059] In the disclosed embodiment, the query request includes a query statement, and the query statement includes query elements, such as "queryType", "dataSource", "intervals", etc. In response to the query cache corresponding to intervals, the cache is hit and the query result can be directly obtained. In response to the query cache not existing or partially existing intervals, the metadata table under dataSource is obtained based on dataSource, and more detailed data is obtained to obtain the query result.

[0060] In the embodiment of the present disclosure, the metadata table is explained in conjunction with Table 1.

[0061] Table 1

[0062]

[0063] As shown in Table 1, the relevant metadata core messages about dataSource are shown. Set the calculation type of the field for each field of dataSource. As shown in Table 2. dataSource can include a basic universal unique identifier (basic.uuid), basic.ip, and basic identifier (basic.uid), set the calculation type for the above three fields respectively, and design thetaSketch calculation type for the above three fields respectively, set the time attribute for dataSource to event_time, and set the last update time.

[0064] Table 2

[0065] basic.uuid thetaSketch basic.ip thetaSketch basic.uid thetaSketch timeField event_time lastChangeTime 2019-01-01 01:01:01

[0066] In the disclosed embodiment, the calculation type can be set for the field in advance, and the calculation type can also be understood as a calculation indicator. The indicators to be calculated can be configured for each level of path, such as calculating the page views (pv) and unique visitors (uv) of the hierarchical path data, and marking the calculation method of each indicator field for the data indicators to be calculated under different dimensional combinations.

[0067] In the embodiment of the present disclosure, the dimension combination can select the analysis field of the data and the field added by the dimension expansion. Among them, the analysis field is composed of multiple fields in sequence, and each field is used as a grouping dimension of the data, and can also drill down to the next level dimension for drill-down analysis. In other words, the fields of different dimensions are combined and connected in sequence.

[0068] In step S32, based on the data field, calculation type and update time field, the updated data is obtained to obtain the query result.

[0069] In the disclosed embodiment, each time a data ingestion task is created, the metadata table is used to determine whether any changes have occurred through the lastChangeTime and the previous lastChangeTime fields. If there are changes, the most recent metadata is obtained from the data fields and calculation types in the metadata table, and an ingestion task is created based on the latest metadata.

[0070] In the disclosed embodiment, Druid can dynamically add new dimensions for dimension combinations. For example, when a new field first appears in the input data, Druid will automatically add it as a new dimension. At the same time, in order to ensure that useless fields are used as dimensions for calculation, fields will be filtered so that useless fields will not be transferred to Druid.

[0071] In the disclosed embodiment, for the query elements that need to be calculated, it is possible to combine the two perspectives of calculation indicators and dynamic dimensions. For the calculation indicators, the updated data can be directly obtained from the metadata table, avoiding the need to restart the Druid task and configure new data source indicators. For dynamic dimensions, they can be added dynamically, avoiding the need to restart the calculation task and manually provide the latest dimension information, realizing a method for automatically changing the indicator dimension, and realizing dynamic indicator loading and dynamic dimension loading. And a high-performance method for obtaining cached data is realized.

[0072] In the disclosed embodiment, the query results in the query cache may be obtained in the following manner. Figure 4 FIG. 1 is a flow chart showing a method for determining a query result according to an exemplary embodiment. Figure 4 As shown, the method includes steps S41 to S42.

[0073] In step S41, the query result of the first query element is obtained.

[0074] In the disclosed embodiment, if the time span is from 2018-08-14 to 2018-08-16, it is split into 2018-08-14, 2018-08-15 and 2018-08-16 at a daily granularity and the query request is reassembled, each split time range is queried separately, and finally each query result is summarized and the final result is returned.

[0075] In step S42, each query result is stored in the query cache according to the format of the second query element, the first query element and the query result of the first query element.

[0076] In the disclosed embodiment, a second query element other than the first query element in the query element is obtained, and the first query element can be an intervals field, that is, the time span intervals field is removed to generate a time-independent query body, wherein the time-independent query body can be understood as the second query element. The time-independent query body is used as the key (key) of the remote dictionary service (redis) hash data structure, each split time range is used as the hash data structure field (Field), and the query result of each time range is used as the value (value). In response to obtaining the query request, the query element is determined. The time-independent query body of the user query request is obtained, and the time-independent query body is used as the redis key, and the query is queried through the redis acquisition command. If the key does not exist, the cache is not hit. If the key exists, MGET obtains the cache data of all unit time spans, and then determines whether all unit time spans of the current request hit the cache, and the hit result will be directly returned.

[0077] For example, as shown in Table 3, 2018-08-14, 2018-08-15, and 2018-08-16 may be the following query results.

[0078] Table 3

[0079] Query time Query results 2018-08-14 12 2018-08-15 13 2018-08-16 14

[0080] If the time span of the query request at this time is: 2018-08-13T00:00 / 2018-08-16T23:59:59. 2018-08-13 does not exist in the query cache, and the cache related to 2018-08-13 cannot be hit, so it is necessary to query the data related to 2018-08-13.

[0081] In the disclosed embodiment, the cached result is obtained through the second query element. The unit time span is split according to the query request, and the split unit time spans are 2018-08-13, 2018-08-14, 2018-08-15 and 2018-08-16. Among them, the time spans of 2018-08-14, 2018-08-15 and 2018-08-16 can directly obtain the results from the cache. The cache of the unit time span of 2018-08-13 does not exist, and an actual query is required.

[0082] In the disclosed embodiment, the problem of low cache hit rate when the query as a whole is used as a cache object is reduced, a method for obtaining query results at one time is implemented, and query pressure on cache data is resolved.

[0083] In the disclosed embodiment, a data source corresponds to a metadata table, which includes data fields. Data fields in different dimensions need to be unified into the same dimension, and the dimensions of the data fields can be expanded in the following manner.

[0084] Figure 5 FIG. 1 is a flowchart of a method for obtaining a metadata table according to an exemplary embodiment. Figure 5 As shown, the method includes steps S51 to S53.

[0085] In step S51, a data source in a query statement and a plurality of data fields corresponding to the data source are obtained.

[0086] In the disclosed embodiment, obtaining a query request can also be understood as obtaining an event message. Event messages need to be registered in the data center, and only registered messages can be analyzed and applied. Unregistered messages will be discarded, i.e., useless data. Among them, each event message will identify the basic information of the source of the event, the name of the product to which it belongs, or the specific behavior. For example, the source of the identified data, for example, can be the embedded data on the web page, or the embedded events on the back end, and the specific behavior triggers, for example, can be clicks, exposures, etc. And the time message is composed of multiple data fields. Combined with Table 4 and Table 5, the event source information registration and the data fields of the event are explained respectively.

[0087] Table 4

[0088]

[0089] Table 5

[0090]

[0091] In the embodiment of the present disclosure, in combination with Table 4 and Table 5, after obtaining the event message, the example shown in Table 6 can be generated.

[0092] Table 6

[0093]

[0094] In step S52, it is determined that there is no dimension extension field corresponding to the data field in the hash model structure, and the data field is mapped to the hash model structure to obtain the dimension extension field.

[0095] In the disclosed embodiment, since data needs to be analyzed from different angles, each analysis angle is a dimension. Therefore, in multi-dimensional analysis, the data needs to be expanded in dimension, such as if a time field is a day, it can be expanded to a week, month, and year.

[0096] In the disclosed embodiment, the dimension extension field is described in conjunction with Table 7.

[0097] Table 7

[0098] Comments(field name) Example Destination ID (mdd_id) 10065 Destination Chinese name (mdd_name) Destination 1 City destination ID (city_mdd_id) 21536 City destination name (city_mdd_name) City 1 Province destination ID (province_mdd_id) 21536 Province destination name (province_mdd_name) Province 1 State Destination ID (state_mdd_id) 12692 State Destination Name (state_mdd_name) Country 1

[0099] In the embodiment of the present disclosure, as shown in Table 7, Table 7 is a dimension table related to the destination. Table 7 is synchronized to the redis database as a hot cache using the hash model structure in redis. The hash model structure is used to store the extended fields corresponding to the data fields. The hash data structure in redis is used, the dimension table name is used as the key of redis, the field name of the dimension table is used as the field of the redis hash data, and the value of the corresponding dimension table data field is used as the value of the redis hash.

[0100] For example, based on Table 7, the following data needs to be synchronized to redis: the redis key is "destination: Chinese name of destination (mdd:mdd_name)", the redis field is 10065, and the hash value is destination 1. The redis key is "destination: Chinese name of city destination (mdd:city_mdd_name)", the redis field is 10065, and the hash value is city 1. The redis key is "destination: Chinese name of province destination (mdd:province_mdd_name)", the redis field is 10065, and the hash value is province 1. The redis key is "destination: Chinese name of country destination (mdd:state_mdd_name)", the redis field is 10065, and the hash value is country 1.

[0101] In the disclosed embodiment, in response to obtaining an event message, and the data field in the event message is not associated with the dimension table, the data field is synchronized to redis according to the above hash model structure to obtain a dimension extension field.

[0102] In step S53, a calculation type is set for the dimension extension field, and a metadata table including the calculation type is generated.

[0103] In the embodiment of the present disclosure, based on the dimension extension field in redis, the calculation type is set in advance for the dimension extension field, and after obtaining the calculation type of different dimension extension fields, a metadata table including the calculation type can be generated.

[0104] In the disclosed embodiment, dimension expansion is accomplished through simple configuration, and a metadata table including calculation types can be generated, thereby improving the efficiency of data processing and data calculation.

[0105] In the disclosed embodiment, data fields are mapped to a hash model structure through extension rules to obtain dimension extension fields. Figure 6 FIG. 1 is a flowchart of a method for obtaining a dimension extension field according to an exemplary embodiment. Figure 6 As shown, the method includes steps S61 to S62.

[0106] In step S61, the expansion rule corresponding to the data source is determined.

[0107] In the disclosed embodiment, the extended rules are pre-configured, and the real-time engine reads the rule data of the message middleware in real time. If there are new rules or changes, the latest rules can be obtained in real time. At the same time, the real-time engine also reads the message data, and the message data is matched with the rule data and the data processing of the expanded dimensions after matching is performed.

[0108] In the disclosed embodiment, the extension rule includes a data source, an application name to which the data belongs, a time name to which the event belongs, a dimension table name to be associated, and a new dimension field name to be extended. The data source in the extension rule is matched with the data source in the message event to determine the extension rule that matches it.

[0109] In step S62, multiple data fields in the data source are traversed, and data fields matching the expansion rule are expanded to obtain dimension expansion fields.

[0110] In the disclosed embodiment, each data field of the data is traversed. If the field name can match an item in the rule list of the extended rule, each virtual field information of the virtual field (virtualField) list of the item is traversed, and the virtualField in the virtual field is added to the data as the newest field, and the value of the virtualField is queried from the hash structure model of the redis key-value database. The virtual table (virtualTable): virtualField is used as the key, and the matched data field value is used as the field of the hash structure model.

[0111] For example, the corresponding extension rule is matched for the umddid field, and the data dimension field is expanded by traversing the virtual field column of the rule. Add the "mdd_name" field, the field name is virtualTable:virtualField, that is, mdd:mdd_name, the field value is queried from the redis hash through hget, the key is mdd:mdd_name, and the field is 10065. Add the "city_mdd_name" field, the field name is virtualTable:virtualField, that is, mdd:city_mdd_name, the field value is queried from the redis hash through hget, the key is mdd:city_mdd_name, and the field is 10065. Add the "province_mdd_name" field, the field name is virtualTable:virtualField, that is, mdd:province_mdd_name, the field value is queried from the redis hash through hget, the key is mdd:province_mdd_name, and the field is 10065. Add the "state_mdd_name" field. The field name is virtualTable:virtualField, that is, mdd:state_mdd_name. The field value is queried from the redis hash through hget. The key is mdd:state_mdd_name and the field value is 10065.

[0112] In the disclosed embodiment, in the data processing stage, real-time data dimension association is achieved by matching the fields with corresponding extension rules, and key mapping of automated dimension fields is achieved through simple configuration.

[0113] In the disclosed embodiment, the extension rule remains in a hot update state.

[0114] In the disclosed embodiment, the extended rules are kept in a hot update state so that the latest extended rules can be acquired in real time when the rules are added or changed, and updated to the extended rules, thereby improving the efficiency of data processing.

[0115] The present disclosure embodiments are combined with Figure 7 Describe the data query method. Figure 7 FIG. 1 is a flow chart of a data query method according to an exemplary embodiment. Figure 7 As shown, it includes step 1 event message registration. Event messages need to be registered in the data center, and only registered messages can be analyzed and applied. Step 2, event message reporting, extended dimension data processing. Among them, for extended dimension data processing, it includes dimension table loading hot storage, field mapping rules and dimension extension data processing. Among them, loading hot storage uses the hash data structure in redis, takes the dimension table name as the key of redis, the field name of the dimension table as the field of the redis hash data, and the value of the corresponding dimension table data field as the value of the redis hash. In the field mapping rule, if the data field needs dimension expansion, it is necessary to generate dimension expansion rules for the data field to be expanded. That is, the field to be expanded, the application name to which the data belongs, the event name to which the event belongs, the dimension table name that needs to be keyed, and the new dimension field name that needs to be expanded. Once the rules are configured, the dimension rules will be sent to the message middleware.

[0116] In the disclosed embodiment, step three is to configure the calculation rules. This includes analyzing the path rules and configuring the calculation indicators. Among them, the analysis path rules can select the analysis fields of the data and the fields added by the dimension expansion. The analysis path is composed of multiple fields in sequence. Each field serves as a grouping dimension for the data, and can also drill down to the next level of dimension for drill-down analysis. The calculation indicators can be configured to configure the indicators that need to be calculated for each level of the path, such as performing pv operations or uv operations on the data of the hierarchical path, etc. Under the above different dimensional combinations, the calculation method of each indicator field is marked for the data indicators that need to be calculated. Step four, generate a data indicator metadata table. Step five, rule path query interaction design. Figure 8 FIG. 1 is a schematic diagram showing a rule path query interaction design according to an exemplary embodiment. Figure 8 As shown, data drilling analysis can be performed based on the query front-end interactive interface of the rule path.

[0117] In the disclosed embodiment, step six, data query, query request time span split parallel query. According to the time span intervals, split the time span into multiple finer-grained time ranges, and query each split time range separately. For example, for the time range from 2018-08-14 to 2018-08-16, split the intervals into 2018-08-14, 2018-08-15, 2018-08-16 by day and reassemble the query request, query each split time range separately, and finally summarize each query result and return the final result. Figure 7 The cache data in, including the query results, are written into the cache design, and all cached data results that are hit in one query are designed. The query body (Request) can remove the time span intervals field to generate a time-independent query body. The time-independent query body is used as the key of the redis hash data structure, each split time range is used as the hash data structure Field field, and the query result of each time range is used as the value. If a user's query request is queried once, the time-independent query body of the user's query request is first obtained, and the time-independent query body is used as the redis key. The query is queried through the redis command. If the key does not exist, the cache is not hit. If the key exists, MGET obtains the cache data of all unit time spans. Then determine whether all unit time spans of the current request hit the cache, and the hit results will be returned directly. If it does not hit, the actual query is performed in druid. Druid and druid have the same meaning and can be replaced with each other. For example Fig. 9 , Fig. 9 FIG. 1 is a schematic diagram of a data query method according to an exemplary embodiment. Fig. 9 As shown, the query time span of the query request of query example 3-1 is changed to "2018-08-13T00:00 / 2018-08-16T23:59:59". Trigger the query again. First, remove the time span intervals field from the query body (Request), and obtain the cached result from the cache by using the query body key that is not related to time. Split the unit time span according to the query request. The split unit time spans are 2018-08-13, 2018-08-14, 2018-08-15, and 2018-08-16. The time spans of 2018-08-14, 2018-08-15, and 2018-08-16 can directly obtain the results from the cache. The cache of the unit time span of 2018-08-13 does not exist, and an actual query is required.

[0118] In the disclosed embodiment, in the data processing stage, the standardized data field information and data source information of the event message data reported by the tracking point are obtained through the event registration mechanism. In the data processing stage, it is necessary to associate the data or expand the dimension. In the related technology, the real-time data dimension association usually requires SQL or other coding methods to process the data, and it is necessary to perform association processing in each place where the dimension is expanded. The performance is low and the manpower and development costs are high. Based on the above data processing method, a unified dimension association is realized, and the key mapping of the automated dimension field is completed through simple configuration. In the data calculation stage, a method is implemented in which a data calculation process can perform dynamic indicator and dimension changes, which reduces the manual maintenance cost and improves efficiency. In the data query stage, a cache data acquisition method that improves cache hit rate and high performance is implemented, which improves query performance and reduces and solves the query pressure problem on cache data. In terms of visualization applications, an interactive method of rule query is implemented to improve the efficiency and ease of use of data query analysis.

[0119] Based on the same concept, the embodiment of the present disclosure also provides a data query device 100.

[0120] It is understandable that the data query device 100 provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0121] Fig.10 is a block diagram of a data query device 100 according to an exemplary embodiment. Fig.10 The device includes a determining unit 101, a processing unit 102 and an acquiring unit 103.

[0122] The determining unit 101 is used to obtain a query statement corresponding to a query request and determine a query element of the query statement.

[0123] The processing unit 102 is used to determine whether there is a query element in the query cache, and the query cache is used to store historical query elements and historical query results.

[0124] The acquisition unit 103 is used to, in response to the existence of the query element in the query cache, acquire the query result corresponding to the query element in the query cache; in response to the absence or partial existence of the query element in the query cache, query based on the data source corresponding to the query element and acquire the query result.

[0125] In one implementation, the query elements include a first query element; the acquisition unit 103 acquires the query result corresponding to the query element in the query cache in the following manner: acquires the first query element in the query elements, and uses the first query element as a query identifier; based on the first query element, acquires the query result corresponding to the first query element in the query cache.

[0126] In one implementation, the acquisition unit 103 performs a query based on a data source corresponding to a query element and obtains a query result in the following manner: obtain a metadata table corresponding to the data source, the metadata table including a data field, a calculation type, and an update time field, wherein the data field is a field corresponding to the data source, and the calculation type is a calculation type preset for the data field; based on the data field, the calculation type, and the update time field, obtain updated data to obtain a query result.

[0127] In one embodiment, the query element includes a second query element, and the query results in the query cache are obtained in the following manner: the query results of the first query element are obtained, and the first query element includes one or more elements; each query result is stored in the query cache according to the format of the second query element, the first query element, and the query result of the first query element.

[0128] In one implementation, the acquisition unit 103 acquires the metadata table corresponding to the data source in the following manner: acquire the data source in the query statement and multiple data fields corresponding to the data source; determine that a dimension extension field corresponding to the data field does not exist in the hash model structure, map the data field to the hash model structure, and obtain the dimension extension field, the hash model structure is used to store the extension field corresponding to the data field; set the calculation type for the dimension extension field, and generate a metadata table including the calculation type.

[0129] In one embodiment, the processing unit 102 maps the data field to the hash model structure to obtain the dimension extension field by: determining the extension rule corresponding to the data source; traversing multiple data fields in the data source, and expanding according to the data fields matching the extension rule to obtain the dimension extension field.

[0130] In one implementation, the processing unit 102 is further configured to: perform hot update on the extension rule.

[0131] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0132] like Fig.11 As shown, Fig.11 : is a schematic diagram of an electronic device provided by an embodiment of the present invention. One embodiment of the present invention provides an electronic device 200. Among them, the electronic device 200 includes a memory 201, a processor 202, and an input / output (I / O) interface 203. Among them, the memory 201 is used to store instructions. The processor 202 is used to call the instructions stored in the memory 201 to execute the file merging method of the embodiment of the present invention. Among them, the processor 202 is connected to the memory 201 and the I / O interface 203 respectively, for example, it can be connected through a bus system and / or other forms of connection mechanisms (not shown). The memory 201 can be used to store programs and data, including the program of the file merging method involved in the embodiment of the present invention, and the processor 202 executes various functional applications and data processing of the electronic device 200 by running the program stored in the memory 201.

[0133] In the embodiment of the present invention, the processor 202 can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processing, DSP), a field programmable gate array (Field-Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). The processor 202 can be a central processing unit (Central Processing Unit, CPU) or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0134] The memory 201 in the embodiment of the present invention may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), flash memory, hard disk drive (HDD) or solid-state drive (SSD).

[0135] In the embodiment of the present invention, the I / O interface 203 may be used to receive input instructions (such as digital or character information, and key signal input related to user settings and function control of the electronic device 200), and may also output various information to the outside (such as images or sounds). In the embodiment of the present invention, the I / O interface 203 may include one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel.

[0136] It is to be understood that, although the operations are described in a specific order in the drawings in the embodiments of the present invention, it should not be understood as requiring to perform these operations in the specific order shown or in a serial order, or requiring to perform all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0137] The methods and devices involved in the embodiments of the present invention can be implemented using standard programming techniques, using rule-based logic or other logic to implement various method steps. It should also be noted that the words "device" and "module" used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0138] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations or procedures.

[0139] The foregoing description of the implementation of the present invention has been given for purposes of illustration and description. The foregoing description is not intended to be exhaustive nor to limit the invention to the exact form disclosed, and various variations and modifications are possible in accordance with the above teachings or may be obtained from the practice of the invention. These embodiments are selected and described in order to illustrate the principles of the invention and its practical application, so that those skilled in the art can utilize the invention in various embodiments and with various modifications suitable for the particular use contemplated.

Claims

1. A data query method, characterized in that: include: Obtaining a query statement corresponding to the query request, and determining a query element of the query statement; Determine whether the query element exists in a query cache, wherein the query cache is used to store historical query elements and historical query results; In response to the query element existing in the query cache, obtaining a query result corresponding to the query element in the query cache; In response to the query element not existing or partially existing in the query cache, a query is performed based on a data source corresponding to the query element, and a query result is obtained.

2. The method according to claim 1, characterized in that The query elements include a first query element; The obtaining the query result corresponding to the query element in the query cache includes: Obtaining a first query element from the query elements, and using the first query element as a query identifier; Based on the first query element, a query result corresponding to the first query element in the query cache is obtained.

3. The method according to claim 1, characterized in that: The querying based on the data source corresponding to the query element and obtaining the query result includes: Obtaining a metadata table corresponding to the data source, wherein the metadata table includes a data field, a calculation type, and an update time field, wherein the data field is a field corresponding to the data source, and the calculation type is a calculation type preset for the data field; Based on the data field, the calculation type and the update time field, the updated data is obtained to obtain the query result.

4. The method according to any one of claims 1 to 3, characterized in that The query element includes a second query element, and the query result in the query cache is obtained in the following manner: Obtaining a query result of a first query element, where the first query element includes one or more than two elements; According to the format of the second query element, the first query element, and the query result of the first query element, each query result is stored in the query cache.

5. The method according to claim 3, characterized in that: The obtaining of the metadata table corresponding to the data source includes: Obtain a data source in a query statement and multiple data fields corresponding to the data source; Determine that a dimension extension field corresponding to the data field does not exist in the hash model structure, map the data field to the hash model structure to obtain the dimension extension field, and the hash model structure is used to store the extension field corresponding to the data field; A calculation type is set for the dimension extension field, and a metadata table including the calculation type is generated.

6. The method according to claim 5, characterized in that Mapping the data field to the hash model structure to obtain the dimension extension field includes: Determine an expansion rule corresponding to the data source; The plurality of data fields in the data source are traversed, and data fields matching the expansion rule are expanded to obtain the dimension expansion field.

7. The method according to claim 1, characterized in that The method further comprises: Perform hot update on extension rules.

8. A data query device, characterized in that: include: A determination unit, configured to obtain a query statement corresponding to the query request and determine a query element of the query statement; A processing unit, configured to determine whether the query element exists in a query cache, wherein the query cache is configured to store historical query elements and historical query results; an acquiring unit, configured to acquire, in response to the query element existing in the query cache, a query result corresponding to the query element in the query cache; In response to the query element not existing or partially existing in the query cache, a query is performed based on a data source corresponding to the query element, and a query result is obtained.

9. The device according to claim 8, characterized in that The query elements include a first query element; The acquisition unit acquires the query result corresponding to the query element in the query cache in the following manner: Obtaining a first query element from the query elements, and using the first query element as a query identifier; Based on the first query element, a query result corresponding to the first query element in the query cache is obtained.

10. The device according to claim 8, characterized in that The acquisition unit performs a query based on the data source corresponding to the query element and acquires the query result in the following manner: Obtaining a metadata table corresponding to the data source, wherein the metadata table includes a data field, a calculation type, and an update time field, wherein the data field is a field corresponding to the data source, and the calculation type is a calculation type preset for the data field; Based on the data field, the calculation type and the update time field, the updated data is obtained to obtain the query result.

11. The device according to any one of claims 8 to 10, characterized in that The query element includes a second query element, and the query result in the query cache is obtained in the following manner: Obtaining a query result of a first query element, where the first query element includes one or more than two elements; According to the format of the second query element, the first query element, and the query result of the first query element, each query result is stored in the query cache.

12. The device according to claim 10, characterized in that The acquisition unit acquires the metadata table corresponding to the data source in the following manner: Obtain a data source in a query statement and multiple data fields corresponding to the data source; Determine that a dimension extension field corresponding to the data field does not exist in the hash model structure, map the data field to the hash model structure to obtain the dimension extension field, and the hash model structure is used to store the extension field corresponding to the data field; A calculation type is set for the dimension extension field, and a metadata table including the calculation type is generated.

13. The device according to claim 12, characterized in that The processing unit maps the data field to the hash model structure in the following manner to obtain the dimension extension field: Determine an expansion rule corresponding to the data source; The plurality of data fields in the data source are traversed, and data fields matching the expansion rule are expanded to obtain the dimension expansion field.

14. The device according to claim 8, characterized in that The processing unit is also used for: Perform hot update on extension rules.

15. An electronic device, characterized in that: include: processor: a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the data query method according to any one of claims 1 to 7.

16. A storage medium, characterized in that: The storage medium stores instructions. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the data query method according to any one of claims 1 to 7.

Images