Data query method, device and equipment, readable storage medium and program product

Abstract

The invention relates to a data query method and device, equipment, a readable storage medium and a program product. The method comprises the following steps: analyzing a data query request initiated by a demander, wherein an initial query statement in the data query request is an initial query object pointed in the initial query statement; wherein the object type of the query object comprises a data table, a data field and a database; obtaining metadata corresponding to the query object; wherein the metadata comprises field description information of a data field, table structure information, table index information and table identification information of a data table, and connection information of a database; and querying target data corresponding to the query object in the local data model by adopting the metadata corresponding to the query object, and feeding back the target data. By adopting the method, the data processing result can be quickly obtained.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data query method, device, equipment, readable storage medium and program product. Background Art

[0002] In modern enterprise data management and analysis, reporting systems and data query tools are essential components. With the rapid growth of data volume and the diversification of demand-side needs, traditional reporting systems face huge challenges. With the development of technology, intelligent reporting systems have gradually evolved, and some key trends and technologies have emerged. Although these tools and technologies are highly demand-side friendly in self-service analysis and visualization, they still have shortcomings in some aspects, such as low efficiency in data processing. Summary of the invention

[0003] Based on this, it is necessary to provide a data query method, device, equipment, readable storage medium and program product that can quickly obtain data processing results in response to the above technical problems.

[0004] In a first aspect, the present application provides a data query method, the method comprising:

[0005] Parse the data query request initiated by the demander to obtain the query object pointed to in the initial query statement in the data query request; wherein the object types of the query object include data tables, data fields and databases; obtain metadata corresponding to the query object; wherein the metadata includes field description information of the data field, table structure information, table index information and table identification information of the data table, and database connection information; use the metadata corresponding to the query object to query the target data corresponding to the query object in the local data model, and feed back the target data.

[0006] In one of the embodiments, metadata corresponding to a query object is used to query target data corresponding to the query object in a local data model, and the target data is fed back, including: determining a local object corresponding to the query object in a local data model based on the metadata; updating an initial query statement based on the local object to obtain a local query statement corresponding to the local data model; querying target data corresponding to the local object in a local data model based on the local query statement, and feeding back the target data.

[0007] In one embodiment, the field description information includes a data type; based on the local object, the initial query statement is updated to obtain a local query statement in the local data model, including: replacing the query object in the initial query statement with the local object to obtain a replaced initial query statement; based on the data type, the format of the replaced initial query statement is converted to obtain a local query statement in the local data model.

[0008] In one of the embodiments, the data query request also includes a data processing strategy; querying the target data corresponding to the local object in the local data model according to the local query statement includes: adopting the data processing strategy, querying the target data corresponding to the local object in the local data model according to the local query statement.

[0009] In one of the embodiments, the data processing strategy includes permission verification rules and caching rules; the data processing strategy is used to query the target data corresponding to the local object in the local data model according to the local query statement, including: according to the local query statement, querying the target data corresponding to the local object in the local data model, and using the permission verification rules to verify whether the demander has data usage rights for the target data; if so, caching the queried target data according to the caching rules.

[0010] In one of the embodiments, according to the local query statement, the target data corresponding to the local object is queried in the local data model, including: if the query object is a target database, controlling the local data model to establish a connection with the target database according to the connection information of the target database; according to the local query statement, querying the target data corresponding to the local object in the target database through the local data model.

[0011] In a second aspect, the present application further provides a data query device, the device comprising:

[0012] A parsing module is used to parse the data query request initiated by the demander to obtain the query object pointed to in the initial query statement in the data query request; wherein the object types of the query object include data tables, data fields and databases; an acquisition module is used to obtain metadata corresponding to the query object; wherein the metadata includes field description information of the data field, table structure information, table index information and table identification information of the data table, and database connection information; a feedback module is used to use the metadata corresponding to the query object to query the target data corresponding to the query object in the local data model and feedback the target data.

[0013] In a third aspect, the present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program: parsing a data query request initiated by a demander to obtain a query object pointed to in an initial query statement in the data query request; wherein the object type of the query object includes a data table, a data field, and a database; obtaining metadata corresponding to the query object; wherein the metadata includes field description information of the data field, table structure information, table index information and table identification information of the data table, and connection information of the database; using the metadata corresponding to the query object, querying the target data corresponding to the query object in the local data model, and feeding back the target data.

[0014] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the following steps when executed by a processor: parsing a data query request initiated by a demander to obtain a query object pointed to in an initial query statement in the data query request; wherein the object type of the query object includes a data table, a data field, and a database; obtaining metadata corresponding to the query object; wherein the metadata includes field description information of the data field, table structure information, table index information and table identification information of the data table, and connection information of the database; using the metadata corresponding to the query object, querying the target data corresponding to the query object in the local data model, and feeding back the target data.

[0015] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the following steps when executed by a processor: parsing a data query request initiated by a demander to obtain a query object pointed to in an initial query statement in the data query request; wherein the object type of the query object includes a data table, a data field, and a database; obtaining metadata corresponding to the query object; wherein the metadata includes field description information of the data field, table structure information, table index information and table identification information of the data table, and connection information of the database; using the metadata corresponding to the query object, querying the target data corresponding to the query object in the local data model, and feeding back the target data.

[0016] The above-mentioned data query method, device, equipment, readable storage medium and program product parse the data query request initiated by the demander to obtain the query object pointed to in the initial query statement in the data query request; then, by obtaining the metadata corresponding to the query object, a direction is provided for obtaining the data in the data query request; further, the metadata corresponding to the query object can be used to query the target data corresponding to the query object in the local data model, and the target data is fed back to the demander. This solution lays the foundation for determining the general direction of the data query request by first determining the query object, and then provides ideas for accurately determining the details of the data query request by obtaining metadata, that is, by introducing metadata, a basis is provided for accurately determining the information of the query object; finally, by introducing the local data model, a way is provided for finally determining the target data corresponding to the query object, and finally accurate and rapid data query is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] FIG1 is an application diagram of a data query method in one embodiment;

[0019] FIG2 is a schematic diagram of a flow chart of a data query method in one embodiment;

[0020] FIG3 is a schematic diagram of a process of feeding back target data in one embodiment;

[0021] FIG4 is a schematic diagram of a flow chart of a data query method in another embodiment;

[0022] FIG5 is a structural block diagram of a data query device in one embodiment;

[0023] FIG6 is a structural block diagram of a data query device in another embodiment;

[0024] FIG. 7 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] In an exemplary embodiment, as shown in FIG1 , the present application provides an intelligent reporting system, which includes a front-end function subsystem, a data service subsystem, and a back-end management subsystem;

[0027] Among them, the front-end functional subsystem includes business report organization module, dashboard analysis module, SQL editor module, data modeling module, graphical query editor module, indicator and partition module; the data service subsystem includes service interface API (Application Programming Interface) module, query processing engine module, task scheduling module and plug-in management module; the back-end management subsystem includes task and scheduling module, performance management module, log monitoring module, data access module, permission management module and audit tracking module.

[0028] Exemplarily, the service API in the service interface API module in the data service subsystem includes RESTful services related to cards, dashboards, collections, permissions, metadata and demanders; the query processing engine module converts the query instructions of the demander into data results suitable for front-end display and returns them to the front end through the query processing engine through various front-end middleware, compilation middleware (converting structured query representation into SQL statements executed by native databases), execution middleware (such as application column-level permission check middleware, check download permission middleware, query permission check middleware and query result cache return middleware) and post-processing middleware (such as recording and returning metadata middleware, limiting the number of query return rows middleware, returning time zone information middleware, formatting return results middleware and updating visualization settings middleware), so as to support the display and rendering of reports, dashboards and data models.

[0029] Among them, the query processing engine can control and return result data according to the system default permissions and cache configuration results, and can also be flexibly extended through middleware plug-ins to cover the default behavior of the system. It can also call external interfaces or read local extension modules to perform flexible query processing, task scheduling modules and plug-in management modules; the task scheduling module can be based on Quarz (open source job scheduling framework) to periodically synchronize the scan data of data source metadata and field values ​​according to the schedule, and periodically initiate ETL (Extract-Transform-Load) jobs to cache and persist data models to support related query construction, processing and performance improvement; the plug-in management module can implement plug-in interface definition, registration and deregistration, life cycle management and other functions, so that the system can implement various query processing middleware through plug-ins, and meet flexible query processing requirements according to system expansion upgrades or administrator configurations. The application column-level permission check middleware in the execution middleware obtains the column-level permission information in the cache. If there is no information in the cache, the column information that the demander can access is obtained from the table permission group access policy table and the permission group member table to check whether the demander has access rights to all fields in the query. If the client attempts to query a field that they do not have access to, a warning log will be recorded and an exception will be thrown. Depending on whether the context parameter contains a row-level permission level identifier, all field information is extracted from the source query, and the IDs of these fields are checked to see if they are in the set of fields that the client has access to. If the queried field contains a field that the client does not have access to, an insufficient permission exception will be thrown and the query will be terminated.

[0030] The check download permission middleware in the execution middleware can check whether the demander has the permission to download and limit the number of items downloaded. If the demander does not have the permission, an exception will be thrown; the query permission check middleware in the execution middleware can obtain the demander's collection permission and data permission for the queried card, table, and database from the permission group table, permission table, and permission group member table, and comprehensively judge whether the query is processed normally to check the current demander's permission for the specified query. First, determine the query type and apply different permission check logic. If the query involves a specific card, dashboard filter, or audit database, the corresponding permission check function will be called according to the specific situation. If the permission is not satisfied, an exception will be thrown to prevent the query from being executed; the query result cache return middleware in the execution middleware searches for results from the cache according to the hash value of the query through the cache configuration table, query cache table, query execution table, and query table. If it is found and not expired, the cached result is returned. If the cache does not hit, the query is executed and the result is stored in the cache when the conditions are met.

[0031] The recording and returning metadata middleware in the post-processing middleware extracts the column attributes and metadata from the metadata of the query request by analyzing the returned columns. The metadata of the returned columns is appended to the result_metadata in the returned result to record the metadata related to the query result columns after the query is executed. This process usually includes analyzing the attributes of the columns, generating the corresponding metadata information, and attaching it to the final query results so that the demander or system can use these metadata to further process or display the query results; the limiting query return row number middleware in the post-processing middleware can limit the returned result entries. When the number of rows in the query result is limited, the :rows_truncated field is added to the result, and the truncated number of rows is added to the result according to the maximum number of rows returned by permission control, the system default maximum number of rows returned, and the entries that limit the returned results. The return time zone information middleware in the post-processing middleware sets the :results_timezone of the returned results to the time zone information of the report, so as to return the time zone information of the report results according to the system time zone, the requested report time zone, and the database time zone. The format return result middleware in the post-processing middleware formats the returned query result value and converts it into the format that the front end wants to display. The update visualization settings middleware in the post-processing middleware updates :viz-settings to obtain and process the visualization settings of the results. For example, the display format of the value (percent sign, decimal places, suffix, etc.), the configuration style of the chart, such as the rotation degree of the bar chart axis label, the conditional format of the table, etc.

[0032] In order to implement the functions of the above data service subsystem, in an exemplary embodiment, as shown in FIG. 2 , this embodiment provides a data query method, which is applied to the query processing engine module in the above data service subsystem, and the method includes:

[0033] S201, parsing the data query request initiated by the demander to obtain the query object pointed to by the initial query statement in the data query request.

[0034] Among them, the data query request can be a request for the query object information initiated by the demander, and the request can be a structured query request or a native query request (such as a native SQL statement); the object types of the query object include data tables, data fields and databases. Exemplarily, the front-end middleware in the query processing engine module can verify and parse the data query request initiated by the demander, and can also update and expand the content of the query request in the data query request according to the metadata and card data stored in the system, and can save metadata indexes such as tables, data fields and databases to the query processing memory, so as to finally obtain the query object pointed to in the initial query statement in the data query request.

[0035] S202: Obtain metadata corresponding to the query object.

[0036] The metadata includes field description information of the data field, table structure information, table index information and table identification information of the data table, and database connection information. Optionally, the query middleware can be used to unify the query request representation of the data query request and the query object information, and the query middleware performs query processing from each query processing memory in a chain manner until the metadata corresponding to the query object is found, and the metadata is processed uniformly by the query middleware to realize the unified query result display. The query middleware can unify the data query request representation and the query result representation.

[0037] S203, using the metadata corresponding to the query object, querying the target data corresponding to the query object in the local data model, and feeding back the target data.

[0038] Among them, the local data model can be used to store the mapping relationship between various types of report information and the corresponding names of various types of reports; the target data can represent the data that the demander wants to obtain through the data query request. Exemplarily, the metadata corresponding to the query object can be input into the local data model, and the metadata can be converted, analyzed and judged by the compilation middleware and execution middleware in the local data model. If the metadata analyzed and processed by the compilation middleware and the execution middleware meets the conditions for obtaining the target data, the corresponding target data can be retrieved from the local data model, and the target data can be formatted by the post-middleware, such as providing metadata of the result column, providing column information, updating visualization settings, etc., and converting the query results (i.e., target data) into a format suitable for front-end consumption and rendering chart results, and finally feeding back the target data to the demander in an appropriate manner.

[0039] The above data query method parses the data query request initiated by the demander to obtain the query object pointed to in the initial query statement in the data query request; then, by obtaining the metadata corresponding to the query object, it provides a direction for obtaining the data in the data query request; further, the metadata corresponding to the query object can be used to query the target data corresponding to the query object in the local data model, and the target data is fed back to the demander. This solution lays the foundation for determining the general direction of the data query request by first determining the query object, and then provides ideas for accurately determining the details of the data query request by obtaining metadata, that is, by introducing metadata, it provides a basis for accurately determining the information of the query object; finally, by introducing the local data model, it provides a way to finally determine the target data corresponding to the query object, and finally achieves accurate and rapid data query.

[0040] Based on the above embodiment, the embodiment of the present application explains the above embodiment S202 in detail. Specifically, the embodiment of the present application involves the process of obtaining metadata corresponding to the query object, which can be input into the query middleware according to the query object and the data query request initiated by the demander; the role of each query middleware and other information can be referred to as shown in Table 1.

[0041] Table 1 Front-end middleware

[0042]

[0043]

[0044] Furthermore, when the data query request is a structured query (i.e., not a native query-native SQL statement), the query middleware can make a unified query representation for the data query request (such as a unified setting of the interface), which can specifically include indicators such as database identification, query type, structured query, native query, cache strategy, parameters, settings, constraints, middleware options and information. The definition of each indicator in the unified query structure is shown in Table 2.

[0045] Table 2 Unified query structure

[0046]

[0047] The structured query indicators in Table 2 may specifically include sub-indicators such as source query, source table, aggregation, grouping, expression, field, filter condition, restriction, sorting, paging information, connection operation and query metadata. The definitions of each sub-indicator under the structured query indicator are shown in Table 3.

[0048] Table 3 Structured query structure

[0049]

[0050] Among them, the connection operation indicators in Table 3 can be determined in the data query request, and are used to determine which data sources the target data can be obtained from, and how to connect data sources and other operations; specifically, through the specified data source (table or query), and under the given conditions, it can be connected with the current query, allowing the demander to combine data through operations similar to JOIN in SQL (Structured Query Language), which can specifically include sub-indicators such as source query, source table, connection condition, connection strategy, field, alias, foreign key field identifier and query metadata. The definition of each sub-indicator under the connection operation indicator is shown in Table 4.

[0051] Table 4 Connection operation structure

[0052]

[0053] The aggregation indicators in Table 3 can be used to calculate statistical information such as the sum, average, and count of data, including multiple specific aggregation types, and support expressions, filter conditions, etc. Specifically, they can include sub-indicators such as source basic aggregation, special aggregation, and aggregation options. The definitions of each sub-indicator under the aggregation indicator are shown in Table 5.

[0054] Table 5 Aggregation structure

[0055]

[0056] The native query indicators in Table 2 may specifically include sub-indicators such as query subject, template tag, template tag type, query fragment, source query and parameter. The definitions of each sub-indicator under the native query indicator are shown in Table 6.

[0057] Table 6 Native query structure

[0058]

[0059] The expression indicators in Table 3 may specifically include sub-indicators such as string expression, numerical expression, Boolean expression, date and time expression, aggregation, function, parameter, and function tag. The definition of each sub-indicator under the expression indicator is shown in Table 7.

[0060] Table 7 Expression structure

[0061]

[0062] The filter condition indicators in Table 3 may specifically include sub-indicators such as logical operators, comparison operators, interval operators, string filters, time filters, other filters, and case expressions. The definitions of the sub-indicators under the filter condition indicators are shown in Table 8.

[0063] Table 8 Filter condition structure

[0064]

[0065] The field indicators in Table 3 may specifically include sub-indicators such as field identifiers and field options. The definitions of each sub-indicator under the field indicator are shown in Table 9.

[0066] Table 9 Filter condition structure

[0067]

[0068] Furthermore, the query middleware can present the query result page in a unified manner (e.g., the interface can be set in a unified manner). Optionally, the query result returned by the query processing can be standardized by defining a unified query result structure, which can specifically include indicators such as data, cache status, database identifier, query request, start time, query context, number of rows, average execution time, query status, and running time. The definition of each indicator in the unified query result structure is shown in Table 10.

[0069] Table 10 Unified query result structure

[0070]

[0071] The data indicators in Table 10 may specifically include sub-indicators such as row data, column information, download permissions, native query, sandbox status, result time zone, and result metadata. The definitions of each sub-indicator under the data indicator are shown in Table 11.

[0072] Table 11 Data structure

[0073]

[0074] The column information indicators in Table 11 may specifically include sub-indicators such as column description, semantic type, table identifier, field name, display name, basic type, effective type, field reference, and fingerprint. The definitions of each sub-indicator under the column information indicator are shown in Table 12.

[0075] Table 12 Column information structure

[0076]

[0077] In the embodiment of the present application, unified settings are performed on the interfaces of various query middlewares to achieve unified representation of query requests and query results, thereby achieving standardized representation.

[0078] Based on the above embodiment, the present application embodiment explains the above embodiment S203 in detail. Specifically, the process of feeding back target data in the present application embodiment, as shown in FIG3, specifically includes the following steps:

[0079] S301, determining a local object corresponding to a query object in a local data model according to metadata.

[0080] The local object may represent the name of each object in the local data model. Optionally, a mapping table storing the mapping relationship between the query object name and the local object name is stored in the local data model, and the local object corresponding to the query object may be determined based on comparing the name of the query object with the name in the mapping table.

[0081] S302: Update the initial query statement according to the local object to obtain a local query statement corresponding to the local data model.

[0082] Exemplarily, the query object in the initial query statement can be replaced with a local object to update the initial query statement and obtain a local query statement corresponding to the local data model. It should be noted that the metadata corresponding to the query object includes field description information of the data field; wherein the field description information includes the data type.

[0083] One implementable method is to replace the query object in the initial query statement with a local object to obtain a replaced initial query statement; based on the data type, the replaced initial query statement is formatted to obtain a local query statement in the local data model.

[0084] The data type may include but is not limited to a numeric type, a time type, a binary type, a string type, other types, and a NULL type, etc. Exemplarily, the query object in the initial query statement may be converted into an object name in the local data model, that is, replaced with a local object name, to obtain a replaced initial query statement; further, the data type based on the key information in the initial query statement may be converted into a data type that can be processed in the local data model, that is, the format of the replaced initial query statement is converted to obtain a local query statement in the local data model.

[0085] Another possible implementation method is to directly input the initial query statement and local objects into the statement conversion model, and the statement conversion model converts the initial query statement based on the processable data types of the local data model to obtain a local query statement suitable for processing by the local data model.

[0086] S303: According to the local query statement, the target data corresponding to the local object is searched in the local data model, and the target data is fed back.

[0087] Exemplarily, the local data model extracts the local query statement, and based on the extracted information, retrieves the stored data corresponding to the local data model, that is, the target data, and finally feeds back the target data to the demander.

[0088] Optionally, the data query request also includes a data processing strategy.

[0089] One possible implementation method is to use a data processing strategy to query the target data corresponding to the local object in the local data model according to the local query statement. The data processing strategy may include but is not limited to permission verification rules and cache rules. Exemplarily, the data processing strategy corresponding to each data stored in the local data model can be pre-set, such as the need to limit the permissions of each demander to protect the privacy of the data; further, if the corresponding data processing strategy is met, the target data corresponding to the local object can be queried in the local data model according to the local query statement.

[0090] Optionally, according to the local query statement, the target data corresponding to the local object is queried in the local data model, and the permission verification rule is used to verify whether the demander has the data use permission for the target data; if so, the queried target data is cached according to the cache rule. The permission may include but is not limited to the permission of data use and data management; the cache may be determined according to at least one of the adaptive cache strategy, the duration cache strategy, the schedule cache strategy and the default cache strategy.

[0091] Optionally, the adaptive cache strategy can be configured in the configuration table. For example, {"multiplier":1000,"min_duration_ms":2000}, the demand side specifies a multiplier and a minimum query duration to determine the cache storage and retrieval strategy. The duration of the cache during the query is determined by the average query time multiplied by a fixed multiplier, specifically including: determining the maximum cache lifetime - by multiplying the average query execution time (avg-execution-ms) in the cache strategy with the multiplier (multiplier) to obtain max-age-ms, that is, the maximum cache lifetime; calculating the time point of cache expiration - using the cache update time plus the maximum cache lifetime to indicate the time point of cache expiration; comparing the current time and the cache expiration time point - if the current time is earlier than the cache expiration time point, the cache result is returned, otherwise no cache hit is found. If the cache does not hit, it will determine whether to store it in the cache after executing the query. If the statistical query time is greater than the minimum query time set by the demand side, the result will be cached, otherwise it will not be cached.

[0092] The duration cache strategy configuration is as follows: {"duration" : 2, "unit" : "hours"}. The demander directly specifies the duration and unit of the cache. At this time, the cache expiration time is calculated using the cache update time + duration. If the current time is earlier than the cache expiration time, the cache result is returned, otherwise the cache misses. If the cache misses, the cache record is directly updated after the query is executed.

[0093] Schedule cache policy configuration, such as: {"schedule":"0 0 * ? * 2"} The demander specifies a schedule, such as a certain hour every day, and every Monday to update the cache. The system generates a cron expression based on the specified result, and periodically updates the expiration time of the cache configuration according to the configured schedule. When the query is executed, the system directly compares the current time with the expiration time of the cache configuration table, and directly returns the cache result if it has not expired.

[0094] In the default cache strategy, if the customer does not specify a specific strategy when specifying the cache strategy for cards, dashboards, and databases, selecting default means that each cache level will inherit the cache strategy of the parent level. The order of inheritance is: dashboard > card > database > global strategy.

[0095] It should be noted that different plug-ins can be used for different data sources. For example, in cache processing, the query result cache return middleware will check the middleware configuration in the query storage, find out whether the data source object has an extensible query result cache middleware, and if so, obtain the relevant middleware configuration, and use the extended middleware for cache processing. Through the extension of the middleware, you can customize the cache expiration time calculation logic, cache acquisition logic, and cache update logic. For example, in a multi-tenant management scenario, a tenant will connect to multiple databases and have their own special cache processing requirements for these databases. For example, increase the maximum cache value and save the cache in your own database instead of the query system's application database.

[0096] For example, the authority of the demander can be verified. If the demander has data usage authority for the target data, the target data corresponding to the local object can be queried in the local data model according to the local query statement, and the queried target data can be cached according to the caching rules.

[0097] Another possible implementation method is to input the data query request and the local query statement into the permission judgment model, and the permission judgment model will judge the permission of the requester. If the judgment is passed, the target data obtained by the local data model query can be fed back to the requester.

[0098] In the embodiment of the present application, by introducing local objects, a foundation is laid for converting the initial query statement into a local query statement that can be processed by the local data model, which provides a prerequisite for accurately obtaining the target data.

[0099] Based on the above embodiment, the embodiment of the present application explains the above embodiment S303 in detail. Specifically, the process of querying target data in the embodiment of the present application includes the following steps: if the query object is a target database, then according to the connection information of the target database, control the local data model to establish a connection with the target database; according to the local query statement, query the target data corresponding to the local object in the target database through the local data model.

[0100] Exemplarily, in combination with Table 3, if the query object is a target database, the connection information of the target database can be determined based on the information in Table 3; further, based on the connection information, the local data model can be connected to the target database, and based on the local query statement, the required data, that is, the target data corresponding to the local object, can be queried in the target database.

[0101] In the embodiment of the present application, by introducing the target database, the connection between the local and external databases is achieved, the available ways of obtaining data are expanded, and a way is provided for accurately and quickly querying the corresponding data.

[0102] On the basis of the above-mentioned embodiments, the embodiments of the present application also introduce extended middleware, and explain the extended middleware in detail. Specifically, the extended middleware can extend the custom indicators and filters referenced in the query, and fill the actual expressions and filter conditions into the query request. The specific processing process includes:

[0103] First, traverse the metrics and custom filters included in the data query request. Get the relevant definitions from the database / query processor cache. For example, the following query associates the order table with the demand side table, and counts the total number of high-value orders per day for the demand side created this year. Among them, high-value orders is a custom filter segment 2, and the total number of daily orders is a custom metric 1. The request before processing is as follows: {:query {:source-table 5, :joins [{:alias "People - User", :condition [:= [:field 43 {:base-type :type / Integer}][:field 46 {:base-type :type / BigInteger, :join-alias "People - User"}]], :source-table 3}], :aggregation [[:metric 1]], :filter [:and [:segment 2] [:time-interval [:field 56 {:base-type :type / DateTime, :join-alias "People -User"}] -12 :month]]}, :async? true}.

[0104] Furthermore, you can query the indicator table and obtain the indicator definition of metric 1, as shown in Table 13.

[0105] Table 13 Index table

[0106]

[0107] Furthermore, you can query the custom filter table to obtain the indicator definition of segment 2, as shown in Table 14.

[0108] Table 14 Filter table

[0109]

[0110] Then, you can replace the referenced indicators and filters. For example, the result after replacement is as follows:

[0111] { :query {:source-table 5, :joins [{:alias "People - User", :condition [:= [:field 43 {:base-type :type / Integer}] [:field 46 {:base-type :type / BigInteger, :join-alias "People - User"}]], :source-table 3}], :aggregation [[:aggregation-options [:count] {:display-name "Total orders today"}]], :filter [:and [:> [:field 42 nil] 50] [:time-interval [:field 56 {:base-type :type / DateTime, :join-alias "People - User"}] -12 :month] [:time-interval [:field 41 nil] :current :day {:include-current true}]]}, :async? true}.

[0112] In the embodiments of the present application, by introducing extended middleware, a tool is provided for the convenience and adaptability of the data query method.

[0113] Based on the above embodiments, the embodiments of the present application explain the application sandbox middleware in detail. Specifically, the application sandbox middleware filters the rows and columns of the underlying table to be queried according to the permission definition. The filtering conditions can refer to parameters related to the attributes of the demander. For example, when querying the order table, the permission control table stores the sandbox control of the data to only display the orders placed by the demander. Application sandbox middleware processing process:

[0114] First, query the permission group member table to record the relationship between the demander and the permission group. Find that the demander belongs to demander group 7, as shown in Table 15.

[0115] Table 15 Permission group member table

[0116]

[0117] Further, the table permission group access policy table can be queried to define the sandbox control policy corresponding to the demand side group, including the parameter mapping of the sandbox and the card identifier associated with the sandbox definition. It is found that the card identifier of the sandbox definition is 76, and the corresponding parameter is user_id mapping parameter is sandbox, as shown in Table 16.

[0118] Table 16 Access policy table

[0119]

[0120] Further, the card table can be queried to obtain the SQL query definition of card 76. Here, data access to the entire table is controlled by filtering user_id, as shown in Table 17.

[0121] Table 17 Card table

[0122]

[0123]

[0124] Furthermore, the demander table may be queried, in which the attribute information of the demander is recorded, and the attribute value of user_id is obtained to be 1, as shown in Table 18.

[0125] Table 18 Card table

[0126]

[0127] Furthermore, you can query the sandbox definition and demander information based on the table identifier, and then update source-query and source-metadata. For example, the original query is a related query between the order table and the product table:

[0128] :source-table 5,

[0129] :joins [{:alias "Products",

[0130] :condition [:= [:field 40 {:base-type :type / Integer}] [:field 62 {:base-type :type / BigInteger, :join-alias "Products"}]],

[0131] :aggregation [[:count]],

[0132] :breakout [[:field 58 {:base-type :type / Text, :join-alias "Products"}]]

[0133] },

[0134] The order table (source-table 5) is sandboxed and filtered to obtain only the product information of the demander's own ID. The query after replacement is as follows:

[0135] :query {

[0136] :source-query {:params ["1"],

[0137] :native "select ID , USER_ID , total, CREATED_AT,QUANTITY , PRODUCT_ID from orders\nwhere USER_ID in (?)"},

[0138] :joins [{:alias "Products",

[0139] :condition [:= [:field 40 {:base-type :type / Integer}] [:field 62 {:base-type :type / BigInteger, :join-alias "Products"}]],

[0140] :source-table 8,

[0141] :sandbox.query-processor.middleware.row-level-restrictions / gtap? true}],

[0142] :aggregation [[:count]],

[0143] :breakout [[:field 58 {:base-type :type / Text, :join-alias "Products"}]],

[0144] :source-metadata": []

[0145] }.

[0146] Furthermore, you can add native query permissions for the underlying data to prevent the demander from not being able to perform subsequent query processing without native query permissions. For example: perms {:gtaps #{" / db / 1 / native / "}}.

[0147] In the embodiment of the present application, by introducing the application sandbox middleware, a tool is provided for filtering relevant information, laying a foundation for accurate query.

[0148] Based on the above embodiment, this embodiment provides an optional example of a data query method. As shown in FIG4 , the specific implementation process is as follows:

[0149] S41, parsing the data query request initiated by the demander, the initial query statement in the data query request and the query object pointed to by the initial query statement.

[0150] Among them, the object types of the query object include data table, data field and database.

[0151] S402: Obtain metadata corresponding to the query object.

[0152] The metadata includes field description information of the data field, table structure information, table index information and table identification information of the data table, and database connection information. Optionally, the field description information includes the data type.

[0153] S403: Determine the local object corresponding to the query object in the local data model according to the metadata.

[0154] S404: Replace the query object in the initial query statement with the local object to obtain a replaced initial query statement.

[0155] S405: Based on the data type, the replaced initial query statement is formatted to obtain a local query statement in the local data model.

[0156] S406: According to the local query statement, the target data corresponding to the local object is searched in the local data model, and the target data is fed back.

[0157] Optionally, the data query request may also include a data processing strategy. Accordingly, the data processing strategy may be used to query the target data corresponding to the local object in the local data model according to the local query statement. Further, the data processing strategy includes permission verification rules and cache rules. Accordingly, the target data corresponding to the local object may be queried in the local data model according to the local query statement. The permission verification rules are used to verify whether the requester has data usage rights for the target data. If so, the queried target data is cached according to the cache rules. It should be noted that if the query object is a target database, the local data model is controlled to establish a connection with the target database according to the connection information of the target database. According to the local query statement, the target data corresponding to the local object is queried in the target database through the local data model.

[0158] The specific process of the above S401-S406 can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0159] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0160] Based on the same inventive concept, the embodiment of the present application also provides a data query device for implementing the data query method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data query device embodiments provided below can refer to the limitations on the data query method above, and will not be repeated here.

[0161] In an exemplary embodiment, as shown in FIG5 , a data query device is provided, comprising: a parsing module 10, an acquisition module 20 and a feedback module 30, wherein:

[0162] The parsing module 10 is used to parse the data query request initiated by the demander, the initial query statement in the data query request and the query object pointed to in the initial query statement; the object types of the query object include data table, data field and database.

[0163] The acquisition module 20 is used to acquire metadata corresponding to the query object; the metadata includes field description information of the data field, table structure information, table index information and table identification information of the data table, and database connection information.

[0164] The feedback module 30 is used to use the metadata corresponding to the query object to query the target data corresponding to the query object in the local data model and feed back the target data.

[0165] In one embodiment, as shown in FIG6 , the feedback module 30 specifically further includes:

[0166] The object determination unit 31 is used to determine the local object corresponding to the query object in the local data model according to the metadata.

[0167] The statement determination unit 32 is used to update the initial query statement according to the local object to obtain a local query statement corresponding to the local data model.

[0168] The data feedback unit 33 is used to query the target data corresponding to the local object in the local data model according to the local query statement, and to provide feedback on the target data.

[0169] In one embodiment, the statement determination unit 32 is further specifically used to: replace the query object in the initial query statement with a local object to obtain a replaced initial query statement; and convert the format of the replaced initial query statement based on the data type to obtain a local query statement in the local data model.

[0170] In one embodiment, the data feedback unit 33 is specifically used to: adopt a data processing strategy and query the target data corresponding to the local object in the local data model according to the local query statement.

[0171] In one embodiment, the data feedback unit 33 is also specifically used to: query the target data corresponding to the local object in the local data model according to the local query statement, and use the permission verification rules to verify whether the demander has data usage rights for the target data; if so, cache the queried target data according to the caching rules.

[0172] In one embodiment, the data feedback unit 33 is further specifically used to: if the query object is a target database, control the local data model to establish a connection with the target database according to the connection information of the target database; and query the target data corresponding to the local object in the target database through the local data model according to the local query statement.

[0173] Each module in the above data query device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0174] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG7. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store query object related data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data query method is implemented.

[0175] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0176] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0177] In one embodiment, a computer-readable storage medium is provided on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0178] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0179] It should be noted that the information (including but not limited to demander information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0180] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0181] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0182] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data query method, characterized in that: The method comprises: Parsing the data query request initiated by the demander, the query object pointed to in the initial query statement in the data query request; wherein the object type of the query object includes a data table, a data field and a database; Acquire metadata corresponding to the query object; wherein the metadata includes field description information of the data field, table structure information, table index information and table identification information of the data table, and connection information of the database; The metadata corresponding to the query object is used to query the target data corresponding to the query object in the local data model, and the target data is fed back.

2. The method according to claim 1, characterized in that The using metadata corresponding to the query object to query target data corresponding to the query object in the local data model and feeding back the target data includes: Determining, in a local data model, a local object corresponding to the query object according to the metadata; According to the local object, the initial query statement is updated to obtain a local query statement corresponding to the local data model; According to the local query statement, the target data corresponding to the local object is queried in the local data model, and the target data is fed back.

3. The method according to claim 2, characterized in that The field description information includes a data type; and the initial query statement is updated according to the local object to obtain a local query statement in the local data model, including: Replacing the query object in the initial query statement with the local object to obtain a replaced initial query statement; Based on the data type, the replaced initial query statement is formatted to obtain a local query statement in the local data model.

4. The method according to claim 2, characterized in that: The data query request also includes a data processing strategy; and querying the target data corresponding to the local object in the local data model according to the local query statement includes: The data processing strategy is adopted to query the target data corresponding to the local object in the local data model according to the local query statement.

5. The method according to claim 4, characterized in that The data processing strategy includes permission verification rules and cache rules; the adopting of the data processing strategy to query the target data corresponding to the local object in the local data model according to the local query statement includes: According to the local query statement, query the target data corresponding to the local object in the local data model, and use the permission verification rule to verify whether the demander has the data use permission for the target data; If so, the queried target data is cached according to the cache rules.

6. The method according to claim 2, characterized in that The querying, in the local data model according to the local query statement, of target data corresponding to the local object includes: If the query object is a target database, controlling the local data model to establish a connection with the target database according to the connection information of the target database; According to the local query statement, the target data corresponding to the local object is queried in the target database through the local data model.

7. A data query device, characterized in that: The device comprises: A parsing module, used to parse the data query request initiated by the demander, the initial query statement in the data query request and the query object pointed to by the initial query statement; the object type of the query object includes a data table, a data field and a database; An acquisition module, used to acquire metadata corresponding to the query object; the metadata includes field description information of the data field, table structure information, table index information and table identification information of the data table, and connection information of the database; The feedback module is used to use the metadata corresponding to the query object to query the target data corresponding to the query object in the local data model, and feed back the target data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images