A data ad hoc query method based on data snapshot and related equipment

Abstract

The present application is applicable to the field of data processing technology, and provides a data ad hoc query method based on data snapshot and related equipment, the method comprising: determining a target data table in a database of different data types based on user selection information; determining dimensions and measurement data of ad hoc queries according to business demand data, and generating query statements according to the dimensions and measurement data; extracting target fields corresponding to the query statements in the target data table through streaming query technology according to the query statements to obtain mirror data; establishing an association relationship between the mirror data tables through a star data model, performing data query on the mirror data according to the association relationship and the query statements to obtain query results; solidifying the query results to obtain data snapshots, and generating a single target query table according to the data snapshots.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a data ad hoc query method based on data snapshot and related equipment. Background Art

[0002] Ad hoc query means that users can generate query requests in real time according to business needs to obtain information from the database without defining query conditions in advance. Ad hoc query is very flexible, allowing users to quickly obtain data according to their needs without having to design and understand query statements in advance.

[0003] In the prior art, due to factors such as different business system construction time points and different developers, there is no unified planning for business data, and there are situations where business databases are scattered, or even databases of different types. In this case, it is impossible to directly use a unified syntax to perform ad hoc queries on the scattered business databases. Constructing query instances based on each individual business database will obviously lead to data missing and inaccurate query results. Summary of the invention

[0004] The embodiment of the present application provides a data snapshot-based ad hoc query method and related equipment, which can perform ad hoc queries on scattered business library tables, which is impossible to do directly with a unified syntax in the prior art. Constructing query instances based on each individual business library table will obviously lead to data missing and inaccurate query results.

[0005] In a first aspect, an embodiment of the present application provides a data ad hoc query method based on a data snapshot, comprising:

[0006] Determine a target data table based on selection information of a target user for a preset graphical user interface;

[0007] Determine the dimensions and measurement data of ad hoc queries based on business demand data, and generate query statements based on the dimensions and measurement data;

[0008] According to the query statement, the target field corresponding to the query statement in the target data table is extracted by using a streaming query technology to obtain mirror data;

[0009] Establishing an association relationship of the mirror data through a star data model, performing a data query on the mirror data according to the association relationship and the query statement, and obtaining a query result;

[0010] The query results are solidified to obtain a data snapshot, and a single target query table is generated according to the data snapshot.

[0011] Optionally, before the step of determining the target data table based on the selection information of the target user for the preset graphical user interface, the method further includes:

[0012] Acquire the content data to be queried for the ad hoc query service and the authority information of the target user;

[0013] Determine the data query scope of the target user based on the permission information and preset query rules;

[0014] and selecting a target database storing the content data to be queried from databases within the data query range according to the content data to be queried, and obtaining identity identification information of the target database;

[0015] Taking a data table in the target database containing the content data to be queried as a data table to be selected, and obtaining name information of the data table to be selected;

[0016] Based on the identity recognition information and the name information, the preset graphical user interface is established through visualization technology, and the preset graphical user interface is displayed to the target user.

[0017] Optionally, the step of determining the dimensions and measurement data of the ad hoc query according to the business demand data includes:

[0018] Acquire historical demand data of historical ad hoc query services and historical dimension and measurement data corresponding to the historical demand data;

[0019] Training a preset prediction model based on the historical demand data and the historical dimensions and measurement data to obtain a target prediction model, wherein the preset prediction model is a model built based on a deep learning network;

[0020] Substituting the demand data of the ad hoc query business into the target prediction model to obtain a first prediction dimension and measurement data corresponding to the demand data;

[0021] The user's first request information is obtained, and the first prediction dimension and metric data are updated based on the user's first request information to obtain the dimension and metric data of the ad hoc query.

[0022] Optionally, the step of determining the dimensions and measurement data of the ad hoc query according to the business demand data further includes:

[0023] selecting historical demand data of the same type as the demand data as target historical demand data;

[0024] Using the historical dimension and measurement data corresponding to the target historical demand data as the second prediction dimension and measurement data;

[0025] Obtain the user's second request information, and update the second prediction dimension and metric data based on the user's second request information to obtain the dimension and metric data of the ad hoc query

[0026] Optionally, the step of extracting the target field corresponding to the query statement in the target data table by using a streaming query technology according to the query statement to obtain the mirror data includes:

[0027] When different target data tables are stored in target databases of different data types, obtaining the data type of the target database;

[0028] Determining a preset field extraction method corresponding to the data type in the preset field extraction rule as a target extraction method for the target field;

[0029] According to the query statement, the target field of the target data table is extracted in the target extraction method by using a streaming query technology to obtain mirror data.

[0030] Optionally, the method further includes:

[0031] Generating initial interpretation data for the query statement according to the dimension and measurement data and the association relationship;

[0032] The user's modification information for the initial interpretation data is obtained, and the initial interpretation data is updated according to the modification information to obtain target interpretation data.

[0033] Optionally, the method further includes:

[0034] Acquire update request information of the target user, and acquire an update data table corresponding to the update request information;

[0035] According to the query statement, extracting the to-be-updated fields corresponding to the query statement in the update data table by using a streaming query technology to obtain update mirror data;

[0036] Establishing an update association relationship between the mirror data and the updated mirror data through the star data model, performing data query on the mirror data and the updated mirror data according to the update association relationship and the query statement, and obtaining an update query result;

[0037] The update query result is solidified to obtain an update data snapshot, and a single update query table is generated according to the update data snapshot.

[0038] In a second aspect, the present application provides a data ad hoc query device based on a data snapshot, the data ad hoc query device based on a data snapshot comprising:

[0039] An information receiving module, used for determining a target data table based on selection information of a target user for a preset graphical user interface;

[0040] A first calculation module is used to determine the dimensions and measurement data of the ad hoc query according to the business demand data, and generate a query statement according to the dimensions and measurement data;

[0041] An extraction module, configured to extract, according to the query statement, a target field in the target data table corresponding to the query statement by using a streaming query technology to obtain mirror data;

[0042] An association module, used to establish an association relationship of the mirror data through a star data model, perform a data query on the mirror data according to the association relationship and the query statement, and obtain a query result;

[0043] The snapshot module is used to solidify the query results to obtain a data snapshot, and generate a single target query table according to the data snapshot.

[0044] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for ad hoc data query based on data snapshot as described in any one of the first aspects above is implemented.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data ad hoc query method based on data snapshot as described in any one of the above-mentioned first aspects is implemented.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the data ad hoc query method based on data snapshot as described in any one of the above-mentioned first aspects.

[0047] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0048] Compared with the prior art, the embodiments of the present application have the following beneficial effects: determining the target data table based on the selection information of the target user for the preset graphical user interface; determining the dimensions and metric data of the ad hoc query according to the business demand data, and generating a query statement according to the dimensions and metric data; generating a query statement according to the dimensions and metric data and the association relationship; extracting the target field corresponding to the query statement in the target data table through the streaming query technology according to the query statement to obtain the mirror data; extracting data from the target data table using the streaming query technology, returning the iterator after the query is successful, and continuously and quickly copying the target field to the temporary table of the business database to form a data mirror. Compared with the traditional paging result set return method, the streaming query is more stable and can reduce the memory usage load. Establishing the association relationship of the mirror data through the star data model, performing data query on the mirror data according to the association relationship and the query statement to obtain the query result; solidifying the query result to obtain a data snapshot, and generating a single target query table according to the data snapshot. The data image is solidified. Therefore, the type of the original business database (the database where the target data table is located) is not important during the ad hoc query process, which supports cross-database operations and adapts to more business scenarios. The snapshot data builds a single target query table (single wide table) for the business, which improves the efficiency of ad hoc queries and / or ad hoc analysis, and can support millisecond-level responses for millions of data. The data snapshot solidifies the business database data and table structure, which is not affected by subsequent business changes, reducing the cost of analysis and operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 is a flow chart of a data ad hoc query method based on data snapshot provided in an embodiment of the present application;

[0051] Figure 2 is a flow chart of another data ad hoc query method based on data snapshot provided in an embodiment of the present application;

[0052] Figure 3 It is a structural schematic diagram of a data ad hoc query device based on data snapshot provided in an embodiment of the present application;

[0053] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0054] Figure 5 It is a structural schematic diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0056] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0057] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0058] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]" depending on the context.

[0059] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0060] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0061] like Figure 1 As shown, the embodiment of the present application provides a data ad hoc query method based on data snapshot, including:

[0062] S101, determining a target data table based on selection information of a target user for a preset graphical user interface;

[0063] Exemplarily, the process of determining the target data table based on the target user's selection information for a preset graphical user interface may be: the target user performs a click operation (single-click / double-click) on the data table displayed on the preset graphical user interface to select the target data table from multiple data tables displayed on the preset graphical user interface; the process of determining the target data table based on the target user's selection information for the preset graphical user interface may also be: the target user inputs identification information of the target data table (for example, the name of the target data table), and selects from multiple data tables displayed on the preset graphical user interface based on the identification information of the target data table to determine the target data table.

[0064] Exemplarily, the multiple data tables displayed by the preset graphical user interface may be derived from databases of different data types, for example, MySQL and Doris databases of two different data types.

[0065] S102, determining the dimensions and measurement data of the ad hoc query according to the business demand data, and generating a query statement according to the dimensions and measurement data;

[0066] For example, under the support of the data snapshot principle, the ad hoc tool will not pay attention to the impact of the database type on the statement interpretation, and automatically generate query statements and interpretations based on the dimensions, metrics and the association relationships selected by the user. For example, SQL statements are generated based on the query dimensions, metrics and relationships and displayed on the user interface, which helps users with a certain IT foundation to quickly determine whether the constructed ad hoc query is accurate. Specifically, the course types in table A and the course scores in table B are analyzed, and the course IDs of the two tables are associated. The automatically generated interpretation is: SELECT A. Course type, B. Course score FROM A, B WHERE A. Course ID = B. Course ID.

[0067] For example, you can select key dimensions and metric fields according to business needs. For example, when building an ad hoc query, according to the table currently selected by the user and the selected data of the most recent historical ad hoc query, quickly configure the dimensions, metrics and unanalyzed data of the field, and fill in the corresponding field rename, reduce unnecessary business wide table dimension reading, and improve the efficiency of computing resource utilization. You can also intelligently select the target field. For example, if the user's current selected table has not been used to build a historical ad hoc query, a default recommendation is required. The system will intelligently distinguish according to the field type and data dimension. For the "numeric" field type, we will recommend this field as a metric; for the "time" type field, we recommend not to analyze, such fields often have no analytical value; for the "character" type field, intelligent recommendations will be made based on the results after deduplication. If the amount of data after deduplication is <= 100, it is defaulted to be able to be analyzed as a data dimension, such as various business types, levels, and regions; if > 100, it is considered that private information with no distinguishing value, such as name, ID number, phone number, address, etc., can quickly identify and select data valuable for analysis.

[0068] In a possible implementation, the target user may rename the selected field (target field) to improve the readability and understandability of the selected field (target field).

[0069] S103, according to the query statement, extracting the target field corresponding to the query statement in the target data table by using a streaming query technology to obtain mirror data;

[0070] In a possible implementation, the step of extracting the target field corresponding to the query statement in the target data table by using a streaming query technology according to the query statement to obtain the mirror data includes:

[0071] When different target data tables are stored in target databases of different data types, obtaining the data type of the target database;

[0072] Determining a preset field extraction method corresponding to the data type in the preset field extraction rule as a target extraction method for the target field;

[0073] According to the query statement, the target field of the target data table is extracted in the target extraction method by using a streaming query technology to obtain mirror data.

[0074] Exemplarily, through the mechanism of data snapshot, using the data extraction and conversion tool (target extraction method) corresponding to the target database, all target data tables (regardless of the type of target database they come from) are uniformly converted (extracted) from the target fields of the target data tables to obtain mirror data, and the mirror data is uniformly stored in a temporary table of the business library, thereby solving the problem that ad hoc queries cannot associate business data across database types.

[0075] S104, establishing an association relationship of the mirror data through a star data model, and performing a data query on the mirror data according to the association relationship and the query statement to obtain a query result;

[0076] Exemplarily, the star data model is a multidimensional data relationship consisting of a fact table (FactTable) and a set of dimension tables (DimensionTable). Each dimension table has a dimension as a primary key, and the primary keys of all these dimensions are combined into the primary key of the fact table.

[0077] S105: Solidify the query results to obtain a data snapshot, and generate a single target query table according to the data snapshot.

[0078] For example, the ad hoc analysis / query performance of a single business wide table (target query table) can support millisecond-level responses for millions of data, thereby improving the response speed of ad hoc analysis / queries.

[0079] The target data table is determined based on the selection information of the target user for the preset graphical user interface; the dimension and measurement data of the ad hoc query are determined according to the demand data of the ad hoc query business, the target field of the target data table is determined based on the dimension and measurement data, and the association relationship of the target data table is established through the star data model; the query statement is generated according to the dimension and measurement data and the association relationship; according to the query statement, the target field of the target data table is extracted through the streaming query technology to obtain the mirror data; the target data table is extracted by the streaming query technology, and the iterator is returned after the query is successful, and the target field is continuously and quickly copied to the temporary table of the business database to form a data mirror. Compared with the traditional paging result set return method, the streaming query is more stable and can reduce the memory usage load. The mirror data is solidified to obtain a data snapshot, and a single target query table is generated according to the data snapshot. The data image is solidified. Therefore, the type of the original business database (the database where the target data table is located) is not important during the ad hoc query process, which supports cross-database operations and adapts to more business scenarios. The snapshot data builds a single target query table (single wide table) for the business, which improves the efficiency of ad hoc queries and / or ad hoc analysis, and can support millisecond-level responses for millions of data. The data snapshot solidifies the business database data and table structure, which is not affected by subsequent business changes, reducing the cost of analysis and operation.

[0080] In a possible implementation manner, before the step of determining the target data table based on the selection information of the target user for the preset graphical user interface, the method further includes:

[0081] Acquire the content data to be queried for the ad hoc query service and the authority information of the target user;

[0082] Determine the data query scope of the target user based on the permission information and preset query rules;

[0083] and selecting a target database storing the content data to be queried from databases within the data query range according to the content data to be queried, and obtaining identity identification information of the target database;

[0084] Taking a data table in the target database containing the content data to be queried as a data table to be selected, and obtaining name information of the data table to be selected;

[0085] Based on the identity recognition information and the name information, the preset graphical user interface is established through visualization technology, and the preset graphical user interface is displayed to the target user.

[0086] Exemplarily, by determining the data that a target user can call based on the target user's permission information, data security is ensured, and operations on data in the database by unauthorized personnel are prevented, thereby reducing risks.

[0087] Exemplarily, based on the content data to be queried, a target database storing the content data to be queried is screened out from databases within the data query range, and identity identification information of the target database is obtained; a data table in the target database containing the content data to be queried is used as a data table to be selected, and the name information of the data table to be selected is obtained, so that the target user can intuitively observe the source of the data while ensuring that no data is missed.

[0088] In a possible implementation, the step of determining the dimensions and metric data of the ad hoc query according to the business demand data includes:

[0089] Acquire historical demand data of historical ad hoc query services and historical dimension and measurement data corresponding to the historical demand data;

[0090] Training a preset prediction model based on the historical demand data and the historical dimensions and measurement data to obtain a target prediction model, wherein the preset prediction model is a model built based on a deep learning network;

[0091] Substituting the demand data of the ad hoc query business into the target prediction model to obtain a first prediction dimension and measurement data corresponding to the demand data;

[0092] The user's first request information is obtained, and the first prediction dimension and metric data are updated based on the user's first request information to obtain the dimension and metric data of the ad hoc query.

[0093] Exemplarily, the preset prediction model is trained by the historical demand data and the historical dimensions and measurement data, so that the preset prediction model built based on the deep learning network learns the correlation between the historical demand data and the historical dimensions and measurement data, that is, the preset prediction model that learns the correlation between the historical demand data and the historical dimensions and measurement data is the target prediction model, and then, the prediction of dimension and measurement data is realized through the target prediction model, which reduces unnecessary business wide table dimension reading, improves the utilization efficiency of computing resources, and also eliminates the process of target users setting dimension and measurement data one by one, thereby improving the efficiency of ad hoc queries.

[0094] Exemplarily, the user's first request information is obtained, and the first prediction dimension and metric data are updated based on the user's first request information. While improving the efficiency of ad hoc queries, the differences in ad hoc queries under different businesses are also taken into account. The first prediction dimension and metric data output by the target prediction model are updated according to the user's first request information, so that the obtained dimension and metric data are more accurate and more in line with the needs of target users.

[0095] In a possible implementation manner, the step of determining the dimensions and metric data of the ad hoc query according to the business demand data further includes:

[0096] selecting historical demand data of the same type as the demand data as target historical demand data;

[0097] Using the historical dimension and measurement data corresponding to the target historical demand data as the second prediction dimension and measurement data;

[0098] The second request information of the user is obtained, and the second prediction dimension and metric data are updated based on the second request information of the user to obtain the dimension and metric data of the ad hoc query.

[0099] Exemplarily, compared with the prediction process of dimension and metric data through the target prediction model, historical demand data that is the same as the demand data can also be selected as the target historical demand data, and the historical dimension and metric data corresponding to the target historical demand data can be used as the second prediction dimension and metric data. There is no need to train the model, which saves computing power. Through a simple comparison process, the dimension and metric data of the ad hoc query can be obtained more intuitively.

[0100] In a possible implementation, the method further includes:

[0101] Generating initial interpretation data for the query statement according to the dimension and measurement data and the association relationship;

[0102] The user's modification information for the initial interpretation data is obtained, and the initial interpretation data is updated according to the modification information to obtain target interpretation data.

[0103] Exemplarily, under the support of the data snapshot principle, the ad hoc tool will not pay attention to the impact of the database type on the statement interpretation. It will generate initial interpretation data for the query statement based on the dimension and measurement data and the association relationship, and update the initial interpretation data based on the modification information to obtain the target interpretation data, thereby improving the readability and comprehensibility of the interpretation data, reducing the technical staff's understanding cost of ad hoc construction, and ensuring that non-technical users can also quickly build queries.

[0104] In a possible implementation, the method further includes:

[0105] Acquire update request information of the target user, and acquire an update data table corresponding to the update request information;

[0106] According to the query statement, extracting the to-be-updated fields corresponding to the query statement in the update data table by using a streaming query technology to obtain update mirror data;

[0107] Establishing an update association relationship between the mirror data and the updated mirror data through the star data model, performing data query on the mirror data and the updated mirror data according to the update association relationship and the query statement, and obtaining an update query result;

[0108] The update query result is solidified to obtain an update data snapshot, and a single update query table is generated according to the update data snapshot.

[0109] Exemplarily, a single update query table is generated based on the update request information to track data changes, thereby ensuring the real-time and relevance of the analysis results.

[0110] As an example but not limiting, a data ad hoc query method based on data snapshot in a specific application scenario includes: the user selects relevant database tables of two different database types, MySQL and Doris, through the GUI, namely, the test score table, the test subject management table, and the course management table; selects the course name, course year, etc. as dimensions, and the test score as a measure; uses the subjectid field and modeid field of the three tables (test score table, test subject management table, and course management table) respectively, and uses the star data model to establish the association between the tables; generates a query statement based on the dimension and measure data and the association relationship; uses the streaming query technology to copy the data mirror of the test score table, the test subject management table, and the course management table; uses the query statement to take a data snapshot of the data mirror to generate a business wide table snapshot. The wide table contains data fields including student ID, course name, course year, course form, and test score. In the analysis GUI, the business wide table snapshot is dragged and dropped for dimension and measure selection. In this implementation example, the user first drags and drops the course name and course year, and finally drags and drops the course form. At this time, the course name and year will merge cells to form a data dimension. The user then drags and drops the test score counts to analyze the number of course enrollees; finally, drags and drops the test score average to analyze the course teaching quality; based on different business needs, the user clears the original analysis (analyzing the number of course enrollees, analyzing the course teaching quality), re-iterates the analysis, and conducts in-depth mining of the data value; the analysis results can be exported.

[0111] The order of execution of each step in the above embodiment does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0112] Reference Figure 2 A data ad hoc query method based on data snapshots includes: building ad hoc queries, generating ad hoc snapshots, and iterating ad hoc analysis; wherein, building ad hoc queries includes: 1. business construction, selecting libraries, tables, and fields used by the business (equivalent to determining the target data table based on the selection information of the target user for the preset graphical user interface); 2. attribute configuration, configuring the dimension or measurement attribute of the field (equivalent to determining the dimension and measurement data of the ad hoc query according to the business demand data, and generating a query statement according to the dimension and measurement data); 3. establishing associations, configuring the association relationship of each table (equivalent to establishing the association relationship of the mirror data through a star data model); generating ad hoc snapshots includes: 4. mirror copying, extracting and copying the tables and fields involved from each library to generate a mirror (equivalent to, according to the query statement, The flow-through query technology extracts the target field corresponding to the query statement in the target data table to obtain the mirror data); 5. Mirror query, generate a query statement according to the configuration relationship to query the mirror (equivalent to establishing the association relationship of the mirror data through the star data model, and querying the mirror data according to the association relationship and the query statement to obtain the query result); 6. Data snapshot, solidify the mirror query result into a wide table of the data snapshot (equivalent to solidifying the query result to obtain a data snapshot, and generating a single target query table based on the data snapshot); Ad hoc analysis iteration includes: 7. Ad hoc analysis, perform ad hoc analysis according to the business; 8. Optimization iteration, continuously iterate according to the results to obtain satisfactory results; 9. Snapshot update, the solidified data can be manually updated.

[0113] Corresponding to the data ad hoc query method based on data snapshot described in the above embodiment, Figure 5 A structural block diagram of a data ad hoc query device based on data snapshots provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0114] Reference Figure 3 The embodiment of the present application provides a data ad hoc query device based on a data snapshot, and the data ad hoc query device based on a data snapshot includes:

[0115] The information receiving module 201 is used to determine the target data table based on the selection information of the target user for the preset graphical user interface;

[0116] A first calculation module 202, used to determine the dimensions and metric data of the ad hoc query according to the business demand data, and generate a query statement according to the dimensions and metric data;

[0117] The extraction module 203 is used to extract the target field corresponding to the query statement in the target data table through the streaming query technology according to the query statement to obtain the mirror data;

[0118] An association module 204 is used to establish an association relationship of the mirror data through a star data model, and perform a data query on the mirror data according to the association relationship and the query statement to obtain a query result;

[0119] The snapshot module 205 is used to solidify the query results to obtain a data snapshot, and generate a single target query table according to the data snapshot.

[0120] In a possible implementation, Figure 4 As shown, an embodiment of the present application provides a terminal device 300, including: a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, it implements: determining a target data table based on selection information of a target user for a preset graphical user interface; determining dimensions and metric data for ad hoc queries according to business demand data, and generating a query statement according to the dimensions and metric data; extracting a target field corresponding to the query statement in the target data table through a streaming query technology according to the query statement to obtain mirror data; establishing an association relationship of the mirror data through a star data model, performing a data query on the mirror data according to the association relationship and the query statement to obtain a query result; solidifying the query result to obtain a data snapshot, and generating a single target query table according to the data snapshot.

[0121] In a possible implementation, Figure 5As shown, an embodiment of the present application provides a computer-readable storage medium 400, on which a computer program 411 is stored. When the computer program 411 is executed by a processor, it implements: determining a target data table based on selection information of a target user for a preset graphical user interface; determining dimensions and metric data for ad hoc queries based on business demand data, and generating a query statement based on the dimensions and metric data; extracting a target field corresponding to the query statement in the target data table through a streaming query technology based on the query statement to obtain mirror data; establishing an association relationship of the mirror data through a star data model, performing a data query on the mirror data based on the association relationship and the query statement to obtain a query result; solidifying the query result to obtain a data snapshot, and generating a single target query table based on the data snapshot.

[0122] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0123] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0124] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0125] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0127] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data ad hoc query method based on data snapshot, characterized in that: include: Determine a target data table based on selection information of a target user for a preset graphical user interface; Determine the dimensions and measurement data of ad hoc queries based on business demand data, and generate query statements based on the dimensions and measurement data; The step of determining the dimensions and measurement data of the ad hoc query according to the business demand data includes: Acquire historical demand data of historical ad hoc query services and historical dimension and measurement data corresponding to the historical demand data; Training a preset prediction model based on the historical demand data and the historical dimensions and measurement data to obtain a target prediction model, wherein the preset prediction model is a model built based on a deep learning network; Substituting the demand data of the ad hoc query business into the target prediction model to obtain a first prediction dimension and measurement data corresponding to the demand data; Acquire the user's first request information, and update the first prediction dimension and metric data based on the user's first request information to obtain the dimension and metric data of the ad hoc query; According to the query statement, the target field corresponding to the query statement in the target data table is extracted by using a streaming query technology to obtain mirror data; The step of extracting the target field corresponding to the query statement in the target data table by using a streaming query technology according to the query statement to obtain the mirror data includes: When different target data tables are stored in target databases of different data types, obtaining the data type of the target database; Determining a preset field extraction method corresponding to the data type in the preset field extraction rule as a target extraction method for the target field; According to the query statement, the target field of the target data table is extracted by the streaming query technology in the target extraction method to obtain the mirror data; Establishing an association relationship of the mirror data through a star data model, performing a data query on the mirror data according to the association relationship and the query statement, and obtaining a query result; Solidifying the query results to obtain a data snapshot, and generating a single target query table based on the data snapshot; The method further comprises: Acquire update request information of the target user, and acquire an update data table corresponding to the update request information; According to the query statement, extracting the to-be-updated fields corresponding to the query statement in the update data table by using a streaming query technology to obtain update mirror data; Establishing an update association relationship between the mirror data and the updated mirror data through the star data model, performing data query on the mirror data and the updated mirror data according to the update association relationship and the query statement, and obtaining an update query result; The update query result is solidified to obtain an update data snapshot, and a single update query table is generated according to the update data snapshot.

2. The data ad hoc query method based on data snapshot according to claim 1, characterized in that: Before the step of determining the target data table based on the selection information of the target user for the preset graphical user interface, the method further includes: Acquire the content data to be queried for the ad hoc query service and the authority information of the target user; Determine the data query scope of the target user based on the permission information and preset query rules; and selecting a target database storing the content data to be queried from databases within the data query range according to the content data to be queried, and obtaining identity identification information of the target database; Taking a data table in the target database containing the content data to be queried as a data table to be selected, and obtaining name information of the data table to be selected; Based on the identity recognition information and the name information, the preset graphical user interface is established through visualization technology, and the preset graphical user interface is displayed to the target user.

3. The data ad hoc query method based on data snapshot according to claim 1, characterized in that: The step of determining the dimensions and measurement data of the ad hoc query according to the business demand data also includes: selecting historical demand data of the same type as the demand data as target historical demand data; Using the historical dimension and measurement data corresponding to the target historical demand data as the second prediction dimension and measurement data; The second request information of the user is obtained, and the second prediction dimension and metric data are updated based on the second request information of the user to obtain the dimension and metric data of the ad hoc query.

4. The data ad hoc query method based on data snapshot according to claim 1, characterized in that: The method further comprises: Generating initial interpretation data for the query statement according to the dimension and measurement data and the association relationship; The user's modification information for the initial interpretation data is obtained, and the initial interpretation data is updated according to the modification information to obtain target interpretation data.

5. A data ad hoc query device based on data snapshot, characterized in that: The data ad hoc query device based on data snapshot includes: An information receiving module, used for determining a target data table based on selection information of a target user for a preset graphical user interface; A first calculation module is used to determine the dimensions and measurement data of the ad hoc query according to the business demand data, and generate a query statement according to the dimensions and measurement data; An extraction module is used to extract the target field corresponding to the query statement in the target data table by the streaming query technology according to the query statement to obtain the mirror data; the step of extracting the target field corresponding to the query statement in the target data table by the streaming query technology according to the query statement to obtain the mirror data includes: when different target data tables are stored in target databases of different data types, obtaining the data type of the target database; determining the preset field extraction method corresponding to the data type in the preset field extraction rule as the target extraction method for the target field; according to the query statement, extracting the target field of the target data table by the streaming query technology according to the target extraction method to obtain the mirror data; An association module, used to establish an association relationship of the mirror data through a star data model, perform data query on the mirror data according to the association relationship and the query statement, and obtain a query result; The snapshot module is used to solidify the query result to obtain a data snapshot, and generate a single target query table based on the data snapshot, and is also used to: obtain the update request information of the target user, and obtain the update data table corresponding to the update request information; according to the query statement, extract the to-be-updated fields corresponding to the query statement in the update data table through the streaming query technology to obtain the updated mirror data; establish an update association relationship between the mirror data and the updated mirror data through the star data model, perform data query on the mirror data and the updated mirror data according to the update association relationship and the query statement to obtain an update query result; solidify the update query result to obtain an update data snapshot, and generate a single update query table based on the update data snapshot.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data ad hoc query method based on data snapshot as described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data ad hoc query method based on data snapshot as described in any one of claims 1 to 4 is implemented.

Images