A data screening method and device, electronic equipment and readable storage medium

Abstract

The application provides a data screening method and device, electronic equipment and readable storage medium. In response to a feature screening instruction, at least two data cubes to be screened are obtained. For each data cube to be screened, a screening condition for screening a plurality of feature data in the data cube to be screened is determined. An SQL screening statement for screening the data cube to be screened is generated based on a target feature dimension, a target feature value and a relationship expression included in the screening condition. The SQL screening statement is executed to generate an execution result set of the data cube to be screened. The feature data in the execution result set of the at least two data cubes to be screened is aggregated by using a preset aggregation function indicated by the feature screening instruction to obtain a screening result of the at least two data cubes to be screened. In this way, the screening efficiency of data between different data cubes can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data screening method and device, electronic equipment and readable storage medium. BACKGROUND

[0002] A data cube is a technical architecture for data analysis and indexing. It is a processor for big data, which can perform real-time indexing on metadata with any number of keywords. After analyzing the metadata through the data cube, the data query and retrieval efficiency can be greatly improved. The data cube is above the data storage layer and the database system. After data cube analysis, data query and retrieval services can be greatly improved, and the system platform can have advantages such as real-time data storage, real-time query, real-time transmission of query results, etc.

[0003] There are two ways to build a data cube in the prior art. One is to use traditional wide table technology to build and store a data cube. The other is to use narrow table technology to process and aggregate data through the application end. Neither of the two methods can directly implement data screening between multiple data cubes by executing the physical execution plan involved in the SQL statement, making the data screening process between multiple data cubes complex and inefficient. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a data screening method, device, electronic equipment and readable storage medium to overcome the problem of complex data screening process and low screening efficiency between data cubes.

[0005] The data screening method provided by the present application comprises:

[0006] In response to a feature screening instruction, at least two to-be-screened data cubes indicated by the feature screening instruction are obtained. The to-be-screened data cubes include feature data with different feature values under different feature dimensions.

[0007] For each to-be-screened data cube, a screening condition for screening a plurality of feature data in the to-be-screened data cube is determined. The screening condition includes a target feature dimension, a target feature value and a relationship expression. The target feature value is the feature value of each feature data in the target feature dimension. The relationship expression represents the logical relationship between the target feature dimension and the target feature value.

[0008] generate an SQL filtering statement for filtering the data cube to be filtered based on the target feature dimension, the target feature value, and the relational expression;

[0009] execute the SQL filtering statement to generate an execution result set of the data cube to be filtered;

[0010] invoke a preset aggregation function indicated by the feature filtering instruction to perform aggregation processing on feature data in the execution result sets of the at least two data cubes to be filtered, to obtain filtering results of the at least two data cubes to be filtered.

[0011] In a possible implementation, the execution of the SQL filtering statement to generate an execution result set of the data cube to be filtered includes:

[0012] parsing the SQL filtering statement to obtain SQL node information and metadata information of the SQL filtering statement;

[0013] generating an initial logical plan of the SQL filtering statement based on the SQL node information and the metadata information;

[0014] performing optimization processing on the initial logical plan to obtain an executable SQL statement;

[0015] executing the executable SQL statement to generate an execution result set of the data cube to be filtered.

[0016] In a possible implementation, the generation of an SQL filtering statement for filtering the data cube to be filtered based on the target feature dimension, the target feature value, and the relational expression includes:

[0017] generating a logical expression for filtering the data cube to be filtered based on the target feature dimension, the target feature value, and the relational expression;

[0018] substituting the logical expression into a preset SQL statement template to generate an SQL filtering statement for filtering the data cube to be filtered.

[0019] In a possible implementation, the invocation of a preset aggregation function indicated by the feature filtering instruction to perform aggregation processing on feature data in the execution result sets of the at least two data cubes to be filtered, to obtain filtering results of the at least two data cubes to be filtered includes:

[0020] invoking the preset aggregation function indicated by the feature filtering instruction;

[0021] input the feature data in each execution result set into the preset aggregation function one by one, and use the preset aggregation function to merge the feature data in the at least two execution result sets to obtain a merged result set;

[0022] determine the output result as a screening result of the at least two data cubes to be screened.

[0023] In a possible implementation, the preset aggregation function indicated by the feature screening instruction is invoked by the following steps:

[0024] In response to the feature screening instruction, determine the data type, the display form, and the data screening operation of the screening result on the feature data in the at least two execution result sets;

[0025] Based on the data type, the display form, and the data screening operation, determine the preset aggregation function used for screening the feature data in the at least two execution result sets, and generate a SQL invocation statement for invoking the preset aggregation function;

[0026] execute the SQL invocation statement to invoke the preset aggregation function.

[0027] In a possible implementation, the preset aggregation function includes a union aggregation function; and the inputting of the feature data in each execution result set into the preset aggregation function and the use of the preset aggregation function to aggregate the feature data in the at least two execution result sets to obtain the output result of the preset aggregation function include:

[0028] input the feature data in each execution result set into the union aggregation function one by one, use the union aggregation function to merge the feature data in the at least two execution result sets to obtain a merged result set;

[0029] determine the merged result set as the output result of the union aggregation function.

[0030] In a possible implementation, the preset aggregation function includes an intersection aggregation function; and the inputting of the feature data in each execution result set into the preset aggregation function and the use of the preset aggregation function to aggregate the feature data in the at least two execution result sets to obtain the output result of the preset aggregation function include:

[0031] input the feature data in each execution result set into the intersection aggregation function one by one, use the intersection aggregation function to screen out at least one target feature data common in the at least two execution result sets;

[0032] use the at least one target feature data to form a common result set;

[0033] determining the common result set as the output result of the intersection aggregation function.

[0034] In a possible implementation, the preset aggregation function includes a difference set aggregation function; the aggregation processing of the feature data in the at least two execution result sets by using the preset aggregation function includes:

[0035] inputting the feature data in each execution result set into the difference set aggregation function one by one, and merging the feature data in the at least two execution result sets by using the difference set aggregation function to obtain a merged result set;

[0036] filtering out the feature data belonging to the first data cube from the merged result set to obtain the output result of the difference set aggregation function.

[0037] Embodiments of the present application further provide a data screening device, which includes:

[0038] a data acquisition module configured to acquire at least two data cubes to be screened indicated by a feature screening instruction in response to the feature screening instruction; wherein the data cube to be screened includes feature data having different feature values under different feature dimensions;

[0039] a screening item determination module configured to determine, for each data cube to be screened, a screening condition for screening a plurality of feature data in the data cube to be screened; wherein the screening condition includes a target feature dimension, a target feature value, and a relationship expression; the target feature value is a feature value of each feature data under the target feature dimension; and the relationship expression represents a logical relationship between the target feature dimension and the target feature value.

[0040] a statement generation module configured to generate a SQL screening statement for screening the data cube to be screened based on the target feature dimension, the target feature value, and the relationship expression;

[0041] a result generation module configured to execute the SQL screening statement to generate an execution result set of the data cube to be screened;

[0042] a result screening module configured to call a preset aggregation function indicated by the feature screening instruction to perform aggregation processing on feature data in the execution result sets of the at least two data cubes to be screened, and obtain a screening result of the at least two data cubes to be screened.

[0043] In one possible implementation, when the result generation module executes the SQL filtering statement to generate the execution result set of the data cube to be filtered, the result generation module is used to:

[0044] Parse the SQL filter statement to obtain the SQL node information and metadata information of the SQL filter statement;

[0045] Based on the SQL node information and the metadata information, an initial logical plan for the SQL filtering statement is generated;

[0046] The initial logical plan is optimized to obtain executable SQL statements;

[0047] Execute the executable SQL statement to generate the execution result set of the data cube to be filtered.

[0048] In one possible implementation, when the statement generation module generates an SQL filtering statement to filter the data cube based on the target feature dimension, the target feature value, and the relational expression, the statement generation module is used to:

[0049] Based on the target feature dimension, the target feature value, and the relational expression, a logical expression is generated to filter the data cube to be filtered.

[0050] Substitute the logical expression into a preset SQL statement template to generate an SQL filtering statement for filtering the data cube to be filtered.

[0051] In one possible implementation, when the result filtering module calls the preset aggregation function indicated by the feature filtering instruction to aggregate the feature data in the execution result sets of the at least two data cubes to be filtered, and obtains the filtering results of the at least two data cubes to be filtered, the result filtering module is used to:

[0052] Invoke the preset aggregation function indicated by the feature filtering instruction;

[0053] The feature data in each execution result set is used as the input data of the preset aggregation function. The preset aggregation function is used to aggregate the feature data in at least two execution result sets to obtain the output result of the preset aggregation function.

[0054] The output result is determined as the filtering result of the at least two data cubes to be filtered.

[0055] In one possible implementation, the result filtering module is used to invoke the preset aggregation function indicated by the feature filtering instruction through the following steps:

[0056] in response to the feature screening instruction, determining a data type, a display form, and a data screening operation for screening feature data in the at least two execution result sets;

[0057] based on the data type, the display form, and the data screening operation, determining a preset aggregation function used for screening feature data in the at least two execution result sets, and generating a SQL call statement for calling the preset aggregation function;

[0058] executing the SQL call statement to call the preset aggregation function.

[0059] In a possible implementation, the preset aggregation function includes a union aggregation function; when used for taking feature data in each execution result set as input data of the preset aggregation function, and performing aggregation processing on the feature data in the at least two execution result sets by using the preset aggregation function to obtain an output result of the preset aggregation function, the result screening module is configured to:

[0060] input the feature data in each execution result set into the union aggregation function one by one, and combine the feature data in the at least two execution result sets by using the union aggregation function to obtain a combined result set;

[0061] determine the combined result set as the output result of the union aggregation function.

[0062] In a possible implementation, the preset aggregation function includes an intersection aggregation function; when used for taking feature data in each execution result set as input data of the preset aggregation function, and performing aggregation processing on the feature data in the at least two execution result sets by using the preset aggregation function to obtain an output result of the preset aggregation function, the result screening module is configured to:

[0063] input the feature data in each execution result set into the intersection aggregation function one by one, and filter out at least one target feature data common in the at least two execution result sets by using the intersection aggregation function;

[0064] compose a common result set by using the at least one target feature data;

[0065] determine the common result set as the output result of the intersection aggregation function.

[0066] In a possible implementation, the preset aggregation function includes a difference set aggregation function; when the result screening module is used to take the feature data in each execution result set as input data of the preset aggregation function, aggregate the feature data in the at least two execution result sets by using the preset aggregation function, and obtain an output result of the preset aggregation function, the result screening module is configured to:

[0067] input the feature data in each execution result set into the difference set aggregation function one by one, merge the feature data in the at least two execution result sets by using the difference set aggregation function, and obtain a merged result set;

[0068] filter out the feature data belonging to the first data cube from the at least two to-be-screened data cubes in the merged result set, and obtain the output result of the difference set aggregation function.

[0069] The embodiments of the present application further provide an electronic device, including a processor, a memory, and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the data screening method described above.

[0070] The embodiments of the present application further provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the data screening method described above.

[0071] The data screening method, device, electronic device, and readable storage medium provided by the embodiments of the present application, in response to a feature screening instruction, obtain at least two to-be-screened data cubes indicated by the feature screening instruction; for each to-be-screened data cube, determine a screening condition for screening a plurality of feature data in the to-be-screened data cube; wherein the screening condition includes a target feature dimension, a target feature value, and a relationship expression; based on the target feature dimension, the target feature value, and the relationship expression, generate an SQL screening statement for screening the to-be-screened data cube; execute the SQL screening statement to generate an execution result set of the to-be-screened data cube; call a preset aggregation function indicated by the feature screening instruction to aggregate the feature data in the execution result sets of the at least two to-be-screened data cubes, and obtain a screening result of the at least two to-be-screened data cubes. In this way, when the feature data in different data cubes is screened as a whole, the SQL screening statement is generated, and the preset aggregation function is used to screen the feature data between different data cubes, which can simplify the data screening process between different data cubes, and help to improve the screening efficiency of data between different data cubes.

[0072] In order to make the above objectives, characteristics and advantages of the present application more apparent, more comprehensible, the following preferred embodiments are specifically described in detail below, together with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0074] Figure 1 A flow chart of a data filtering method provided by the embodiments of the present application;

[0075] Figure 2 A schematic diagram of a sentence execution process provided by the embodiments of the present application;

[0076] Figure 3 A structural schematic diagram of a data filtering device provided by the embodiments of the present application;

[0077] Figure 4 A structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0078] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more apparent, the following will clearly and completely describe the technical solutions of the embodiments of the present application in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, every other embodiment obtained by those skilled in the art without creative labor belongs to the scope of protection of the present application.

[0079] It is found through research that there are two construction methods of data cubes in the prior art. One is to realize data cube construction and storage by using traditional wide table technology. The other is to process data through application end and then summarize by using narrow table technology. Neither of the two methods can realize data filtering between multiple data cubes directly through physical execution plan involved in the execution of SQL statements, so that the data filtering process between multiple data cubes is relatively complex and the filtering efficiency is low.

[0080] Based on this, the embodiment of the present application provides a data filtering method, which can reduce the difficulty of data filtering between different data cubes and improve the filtering efficiency of the rate between different data cubes.

[0081] Please refer to Figure 1 , Figure 1 The flowchart of the data filtering method provided by the embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the data filtering method provided by the embodiment of the present application comprises the following steps.

[0082] S101, in response to a feature filtering instruction, at least two data cubes to be filtered indicated by the feature filtering instruction are acquired.

[0083] S102, for each data cube to be filtered, a filtering condition for filtering a plurality of feature data in the data cube to be filtered is determined; wherein the filtering condition comprises a target feature dimension, a target feature value and a relationship expression.

[0084] S103, based on the target feature dimension, the target feature value and the relationship expression, an SQL filtering statement for filtering the data cube to be filtered is generated.

[0085] S104, the SQL filtering statement is executed to generate an execution result set of the data cube to be filtered.

[0086] S105, a preset aggregation function indicated by the feature filtering instruction is called to aggregate the feature data in the execution result set of the at least two data cubes to be filtered, and the filtering result of the at least two data cubes to be filtered is obtained.

[0087] The data filtering method provided by the embodiment of the present application, in response to a feature filtering instruction, acquires at least two data cubes to be filtered indicated by the feature filtering instruction; and analyzes the filtering condition for filtering a plurality of feature data included in each data cube to be filtered; according to the filtering condition, an SQL filtering statement for filtering the data cube to be filtered is generated, and an execution result set of the data cube to be filtered is generated by executing the SQL filtering statement. At this time, the feature data between different data cubes can be filtered based on the execution result set of each data cube to be filtered by using a preset aggregation function, and the filtering result of the at least two data cubes to be filtered is obtained. In this way, the filtering process of the feature data between different data cubes can be simplified, and the filtering efficiency of the feature data between different data cubes can be improved.

[0088] In step S101, in response to a feature screening instruction issued by a user, at least two data cubes to be screened are obtained according to the screening indicated by the feature screening instruction; each data cube to be screened includes feature data with different feature values in different feature dimensions.

[0089] In the scheme provided in the present application, in response to a feature screening instruction, specifically, when a user presses a control corresponding to the feature screening instruction, the feature screening instruction can be triggered; the user can issue the feature screening instruction through the control corresponding to the feature screening instruction and / or a preset combination key; specifically, the user can press the control corresponding to the feature screening instruction by using a finger, a mouse, etc.; or a preset combination key in a keyboard, such as a Ctrl key, an Alt key, an A key, etc.; the preset key can be set artificially according to user demand.

[0090] For example, a three-dimensional data cube is taken as an example for illustration (in actual cases, the data cube can be a higher-dimensional data cube); the data cube to be screened has three feature dimensions, for example, the feature dimensions of the data cube to be screened can be age, gender, and occupation, respectively.

[0091] When the data cube to be screened is screened, the feature data included in the data cube to be screened can be screened according to any feature dimension of the data cube to be screened; for example, the feature data with a gender of “male” can be screened from the feature data included in the data cube to be screened according to the feature dimension of “gender”; or the feature data with an occupation of “teacher” can be screened from the feature data included in the data cube to be screened according to the feature dimension of “occupation”.

[0092] Here, the business personnel can issue a feature screening instruction to issue a feature dimension to be used when the data cube to be screened is screened, a feature value, and a relationship expression indicating the relationship between the feature dimension and the feature value; that is, the feature screening instruction issued by the business personnel carries dimension identification information indicating a target feature dimension when the data cube to be screened is screened, value identification information of a target feature value that the feature data in the target feature dimension should have, and expression identification information of a relationship expression associated with the target feature dimension and the target feature value.

[0093] In step S102, for each data cube to be screened, the feature screening instruction issued by the business personnel is analyzed to determine screening conditions for screening multiple feature data in the at least two data cubes to be screened.

[0094] The screening condition is condition information referred to for screening the plurality of feature data, and screening the plurality of feature data is to screen feature data meeting the screening condition from the plurality of feature data. Specifically, the screening condition can include a target feature dimension, a target feature value, and a relationship expression.

[0095] Here, the target feature dimension is pre-selected by the business personnel, the target feature value is a feature value that the feature data should have in the target feature dimension, and the relationship expression represents a logical relationship between the target feature dimension and the target feature value. That is, the relationship expression can indicate the association between the target feature dimension and the target feature value. The relationship expression can include greater than, equal to, less than, greater than or equal to, less than or equal to, and not equal to.

[0096] For example, when the target feature dimension is "gender" and the target feature value is "male", if the feature data with "gender" of "male" is to be screened, the association between the target feature dimension and the target feature value should be "equal to", that is, the feature data with "gender" "equal to" "male" needs to be screened. When the target feature dimension is "age" and the target feature value is "50", if the feature data with "age" greater than "50" is to be screened, the association between the target feature dimension and the target feature value should be "greater than", that is, the feature data with "age" "greater than" "50" needs to be screened.

[0097] In step S103, based on the target feature dimension, the target feature value, and the relationship expression included in the screening condition, a computer executable SQL screening statement for screening the feature data included in the to-be-screened data cube is generated, and then the screening of the to-be-screened data cube can be realized by executing the SQL screening statement.

[0098] In an embodiment, step S103 includes generating a logical expression for screening the to-be-screened data cube based on the target feature dimension, the target feature value, and the relationship expression; and substituting the logical expression into a preset SQL statement template to generate a SQL screening statement for screening the to-be-screened data cube.

[0099] In this step, the target feature dimension, the target feature value, and the relationship expression are used to generate a logical expression representing screening of the feature data with the target feature value in the to-be-screened data cube according to the target feature dimension.

[0100] The generated logical expression is substituted into a pre-set SQL statement template to generate a SQL screening statement for screening the feature data with the target feature value in the to-be-screened data cube according to the target feature dimension.

[0101] In step S104, the generated SQL filter statement is executed to obtain an execution result set after filtering the feature data included in the data cube to be filtered; in correspondence with the above embodiment, the feature data with "male" gender can be filtered from the feature data included in the data cube to be filtered according to the feature dimension of "gender", and the feature data with "male" gender is used to form the execution result set; or the feature data with "teacher" occupation can be filtered from the feature data included in the data cube to be filtered according to the feature dimension of "occupation", and the feature data with "teacher" occupation is used to form the execution result set.

[0102] Here, in order to accurately obtain the execution result of the SQL filter statement, the SQL filter statement needs to be parsed and optimized during execution.

[0103] In an embodiment, refer to Figure 2 , Figure 2 a statement execution process provided by the embodiment of the present application. As shown in Figure 2 , step S104 includes:

[0104] S1041, parsing the SQL filter statement to obtain SQL node information and metadata information of the SQL filter statement.

[0105] In this step, the SQL filter statement is parsed to convert the SQL filter statement into the form of abstract syntax tree (AST) and represented by SQL node information (SQLNode); the abstract syntax tree (AST) obtained by conversion is checked for syntax, and the abstract syntax tree (AST) is verified for syntax by using the metadata information in the database; after verification, the abstract syntax tree (AST) is still represented by SQL node information (SQLNode).

[0106] S1042, generating an initial logical plan of the SQL filter statement based on the SQL node information and the metadata information.

[0107] In this step, the SQL node information (SQLNode) and the metadata information of the abstract syntax tree (AST) are used to construct the initial logical plan (RelNode) of the SQL filter statement.

[0108] The rules of optimization mainly include removing unused fields, merging multiple projection lists, using JOIN instead of subquery, reordering the JOIN list, pushing down the projection item, and pushing down the filter condition;

[0109] S1043, optimizing the initial logical plan to obtain an executable SQL statement.

[0110] In this step, the initial logical plan (RelNode) is optimized, specifically, unused fields in the initial logical plan (RelNode) are removed, multiple projection lists in the initial logical plan (RelNode) are merged, a JOIN is used to replace a subquery statement in the initial logical plan (RelNode), a JOIN list is reordered, a projection item in the initial logical plan (RelNode) is pushed down, and a filter condition in the initial logical plan (RelNode) is pushed down, and the like, to obtain an executable SQL statement.

[0111] In S1044, the executable SQL statement is executed to generate an execution result set of the data cube to be filtered.

[0112] In this step, the execution result set of the data cube to be filtered is generated by executing the obtained executable SQL statement; that is, the feature data filtered by executing the executable SQL statement is used to form the execution result set of the data cube to be filtered.

[0113] In step S105, the execution result set of each data cube to be filtered is substituted into a preset aggregation function indicated by the feature filtering instruction, the preset aggregation function indicated by the feature filtering instruction is called, and aggregation processing of the feature data in the execution result sets of the at least two data cubes to be filtered is implemented, to obtain a filtering result of the at least two data cubes to be filtered.

[0114] The preset aggregation function (User Defined Aggregation Function, UDAF) is a user-defined aggregation function.

[0115] Here, the business personnel can explicitly indicate an operation mode of filtering the at least two data cubes to be filtered as a whole by issuing the feature filtering instruction, and then the corresponding preset aggregation function can be used to filter the overall data of the at least two data cubes to be filtered.

[0116] In an embodiment, step S105 includes: calling the preset aggregation function indicated by the feature filtering instruction; taking the feature data in each execution result set as input data of the preset aggregation function, and using the preset aggregation function to aggregate the feature data in the at least two execution result sets, to obtain an output result of the preset aggregation function; and determining the output result as the filtering result of the at least two data cubes to be filtered.

[0117] In this step, a preset aggregation function indicated by the feature filtering instruction issued by the business personnel is called; and the feature data in each execution result set is input into the called preset aggregation function as input data of the preset aggregation function.

[0118] The feature data in the at least two execution result sets is aggregated by using the called preset aggregation function to obtain an output result of the called preset aggregation function; and the output result of the preset aggregation function is determined as the filtering result of the at least two data cubes to be filtered.

[0119] In an implementation, the preset aggregation function indicated by the feature filtering instruction is called by the following steps:

[0120] In step a, in response to the feature filtering instruction, the data type, the display form, and the data filtering operation for filtering the feature data in the at least two execution result sets are determined.

[0121] In this step, in response to the feature filtering instruction, the data type (for example, the data type is "bitmap type") of the filtering result obtained after filtering the multiple feature data in the at least two data cubes to be filtered, the display form (for example, the display form is "displayed in an array in a row") of the filtering result, and the data filtering operation (for example, the data filtering operation is "intersection operation", "union operation", "complement operation", and "difference operation") for filtering the feature data in the at least two execution result sets are determined.

[0122] In step b, based on the data type, the display form, and the data filtering operation, a preset aggregation function used for filtering the feature data in the at least two execution result sets is determined, and a SQL calling statement for calling the preset aggregation function is generated.

[0123] In this step, based on the determined data type, the display form, and the data filtering operation of the filtering result, a preset aggregation function used for filtering the feature data in the at least two execution result sets is determined.

[0124] Meanwhile, in order to realize the calling of the preset aggregation function, a SQL calling statement for calling the preset aggregation function is generated.

[0125] In step c, the SQL calling statement is executed to call the preset aggregation function.

[0126] In this step, the calling of the preset aggregation function is realized by executing the SQL calling statement, so that the filtering of the feature data in the at least two data cubes to be filtered is completed by using the preset aggregation function.

[0127] In an embodiment, the preset aggregation function comprises a union aggregation function; and the step S105 comprises: inputting the feature data in each execution result set into the union aggregation function one by one, merging the feature data in the at least two execution result sets by using the union aggregation function to obtain a merged result set; and determining the merged result set as the output result of the union aggregation function.

[0128] In this step, when a union operation is needed on the feature data in the execution result sets of the at least two to-be-screened data cubes, the feature data in each execution result set is input into a union aggregation function one by one, the feature data in the execution result sets of the at least two to-be-screened data cubes is merged by using the union aggregation function, and a merged result set is obtained after the merging; and the obtained merged result set is determined as the output result of the union aggregation function.

[0129] In another embodiment, the preset aggregation function comprises an intersection aggregation function; and the step S105 comprises: inputting the feature data in each execution result set into the intersection aggregation function one by one, screening out at least one target feature data that is common in the at least two execution result sets by using the intersection aggregation function; using the at least one target feature data to form a common result set; and determining the common result set as the output result of the intersection aggregation function.

[0130] In this step, when an intersection operation is needed on the execution result sets of the at least two to-be-screened data cubes, the feature data in each execution result set is input into an intersection aggregation function one by one, at least one target feature data that is contained in each execution result set of each to-be-screened data cube is screened out by using the intersection aggregation function; that is, at least one target feature data that is common in each execution result set; the at least one target feature data that is common in each execution result set is merged to form a common result set; and the obtained common result set is determined as the output result of the intersection aggregation function.

[0131] In another embodiment, the preset aggregation function comprises a difference aggregation function; and the step S105 comprises: inputting the feature data in each execution result set into the difference aggregation function one by one, merging the feature data in the at least two execution result sets by using the difference aggregation function to obtain a merged result set; and filtering out the feature data in the merged result set that belongs to a first data cube in the at least two to-be-screened data cubes to obtain the output result of the difference aggregation function.

[0132] In this step, when a difference set operation needs to be performed on the execution result sets of the at least two data cubes to be screened, the feature data in each execution result set is input into the difference set aggregation function one by one; first, the feature data in the at least two execution result sets is merged by the difference set aggregation function, and a merged result set is obtained after the merging; then, the feature data belonging to the first data cube in the at least two data cubes to be screened in the obtained merged result set is filtered out by the difference set aggregation function, and an output result of the difference set aggregation function is obtained.

[0133] Here, the selection of the first data cube can be determined by analyzing the feature screening instruction; that is, the business personnel can select the first data cube in advance, and issue the feature screening instruction so that the computer can determine the first data cube from the at least two data cubes to be screened during the running process; specifically, when the business personnel selects the first data cube, the cube identifier of the selected first data cube can be added in the feature screening instruction.

[0134] In another embodiment, the preset aggregation function includes a complement set aggregation function; step S105 includes: inputting the feature data in each execution result set into the complement set aggregation function one by one, merging the feature data in the at least two execution result sets by the complement set aggregation function to obtain a merged result set, and filtering out the feature data in the merged result set that does not belong to the first data cube in the at least two data cubes to be screened to obtain an output result of the complement set aggregation function.

[0135] In this step, when a difference set operation needs to be performed on the execution result sets of the at least two data cubes to be screened, the feature data in each execution result set is input into the difference set aggregation function one by one; first, the feature data in the at least two execution result sets is merged by the difference set aggregation function, and a merged result set is obtained after the merging; then, the feature data belonging to the first data cube in the at least two data cubes to be screened in the obtained merged result set is filtered out by the difference set aggregation function, and an output result of the difference set aggregation function is obtained.

[0136] The preset aggregation function provided by the embodiment of the application is shown in Table 1, and Table 1 is a self-defined aggregation function table.

[0137] Table 1 Self-defined aggregation function table

[0138]

[0139]

[0140] The data screening method provided by the embodiments of the present application comprises the following steps: in response to a feature screening instruction, at least two data cubes to be screened are acquired, which are indicated by the feature screening instruction; for each data cube to be screened, a screening condition for screening a plurality of feature data in the data cube to be screened is determined; the screening condition comprises a target feature dimension, a target feature value and a relationship expression; based on the target feature dimension, the target feature value and the relationship expression, an SQL screening statement for screening the data cube to be screened is generated; the SQL screening statement is executed to generate an execution result set of the data cube to be screened; and the feature data in the execution result sets of the at least two data cubes to be screened is aggregated by using a preset aggregation function indicated by the feature screening instruction, so as to obtain a screening result of the at least two data cubes to be screened. In this way, when the feature data in different data cubes is screened as a whole, the SQL screening statement is generated and the preset aggregation function is used to screen the feature data between different data cubes, so that the data screening process between different data cubes can be simplified, and the screening efficiency of data between different data cubes can be improved.

[0141] Please refer to Figure 3 , Figure 3 The data screening device provided by the embodiments of the present application is shown in the structure diagram. As shown in Figure 3 The data screening device 300 comprises:

[0142] The data acquisition module 310 is configured to acquire at least two data cubes to be screened in response to a feature screening instruction, which are indicated by the feature screening instruction; wherein the data cubes to be screened comprise feature data with different feature values in different feature dimensions;

[0143] The screening condition determination module 320 is configured to determine, for each data cube to be screened, a screening condition for screening a plurality of feature data in the data cube to be screened; wherein the screening condition comprises a target feature dimension, a target feature value and a relationship expression; the target feature value is a feature value of each feature data in the target feature dimension; and the relationship expression represents a logical relationship between the target feature dimension and the target feature value;

[0144] The statement generation module 330 is configured to generate, based on the target feature dimension, the target feature value and the relationship expression, an SQL screening statement for screening the data cube to be screened;

[0145] The result generation module 340 is configured to execute the SQL screening statement to generate an execution result set of the data cube to be screened;

[0146] The result screening module 350 is configured to call a preset aggregation function indicated by the feature screening instruction, and aggregate feature data in the execution result set of the at least two data cubes to be screened, to obtain the screening result of the at least two data cubes to be screened.

[0147] Further, the result generation module 340 is configured to, when used to execute the SQL screening statement and generate the execution result set of the data cube to be screened:

[0148] parse the SQL screening statement to obtain SQL node information and metadata information of the SQL screening statement;

[0149] generate an initial logical plan of the SQL screening statement based on the SQL node information and the metadata information;

[0150] optimize the initial logical plan to obtain an executable SQL statement;

[0151] execute the executable SQL statement to generate the execution result set of the data cube to be screened.

[0152] Further, the statement generation module 330 is configured to, when used to generate the SQL screening statement for screening the data cube to be screened based on the target feature dimension, the target feature value, and the relationship expression:

[0153] generate a logical expression for screening the data cube to be screened based on the target feature dimension, the target feature value, and the relationship expression;

[0154] substitute the logical expression into a preset SQL statement template to generate the SQL screening statement for screening the data cube to be screened.

[0155] Further, the result screening module 350 is configured to, when used to call a preset aggregation function indicated by the feature screening instruction, aggregate feature data in the execution result set of the at least two data cubes to be screened, to obtain the screening result of the at least two data cubes to be screened:

[0156] call the preset aggregation function indicated by the feature screening instruction;

[0157] use the preset aggregation function to aggregate feature data in at least two execution result sets, to obtain an output result of the preset aggregation function;

[0158] determine the output result as the screening result of the at least two data cubes to be screened.

[0159] Further, the result screening module 350 is configured to invoke the preset aggregation function indicated by the feature screening instruction by the following steps:

[0160] In response to the feature screening instruction, determine the data type, display form and data screening operation of the screening result for screening the feature data in the at least two execution result sets;

[0161] Based on the data type, display form and data screening operation, determine the preset aggregation function used for screening the feature data in the at least two execution result sets, and generate a SQL call statement for calling the preset aggregation function;

[0162] Execute the SQL call statement to call the preset aggregation function.

[0163] Further, the preset aggregation function includes a union aggregation function; when the result screening module 350 is configured to use the preset aggregation function to aggregate the feature data in the at least two execution result sets by taking the feature data in each execution result set as the input data of the preset aggregation function, the result screening module 350 is configured to:

[0164] Input the feature data in each execution result set into the union aggregation function one by one, and use the union aggregation function to merge the feature data in the at least two execution result sets to obtain a merged result set;

[0165] Determine the merged result set as the output result of the union aggregation function.

[0166] Further, the preset aggregation function includes an intersection aggregation function; when the result screening module 350 is configured to use the preset aggregation function to aggregate the feature data in the at least two execution result sets by taking the feature data in each execution result set as the input data of the preset aggregation function, the result screening module 350 is configured to:

[0167] Input the feature data in each execution result set into the intersection aggregation function one by one, and use the intersection aggregation function to screen at least one target feature data common to the at least two execution result sets;

[0168] Use the at least one target feature data to form a common result set;

[0169] Determine the common result set as the output result of the intersection aggregation function.

[0170] Further, the preset aggregation function includes a difference set aggregation function; when the result screening module 350 inputs the feature data in each execution result set as input data of the preset aggregation function, and utilizes the preset aggregation function to perform aggregation processing on the feature data in at least two execution result sets to obtain an output result of the preset aggregation function, the result screening module 350 is configured to:

[0171] input the feature data in each execution result set into the difference set aggregation function one by one, and utilize the difference set aggregation function to combine the feature data in at least two execution result sets to obtain a combined result set;

[0172] filter out the feature data in the combined result set that belongs to the first data cube in the at least two data cubes to be screened, and obtain the output result of the difference set aggregation function.

[0173] The data screening apparatus provided in the embodiments of the present application, in response to a feature screening instruction, acquires at least two data cubes to be screened indicated by the feature screening instruction; for each data cube to be screened, determines a screening condition for screening a plurality of feature data in the data cube to be screened; wherein the screening condition includes a target feature dimension, a target feature value, and a relationship expression; generates an SQL screening statement for screening the data cube to be screened based on the target feature dimension, the target feature value, and the relationship expression; executes the SQL screening statement to generate an execution result set of the data cube to be screened; and calls a preset aggregation function indicated by the feature screening instruction to perform aggregation processing on the feature data in the execution result sets of the at least two data cubes to be screened, and obtain a screening result of the at least two data cubes to be screened. In this way, when the feature data in different data cubes is screened as a whole, the SQL screening statement is generated and the preset aggregation function is utilized to screen the feature data between different data cubes, which can simplify the data screening process between different data cubes and help improve the screening efficiency of data between different data cubes.

[0174] Please refer to Figure 4 , Figure 4 The electronic device 400 includes a processor 410, a memory 420, and a bus 430. Figure 4

[0175] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 through the bus 430. The machine-readable instructions executed by the processor 410 can perform the above-mentioned Figure 1 ​The steps of the data screening method in the method embodiment are described above, and the specific implementation manners can be referred to the method embodiment, which will not be repeated here.

[0176] The computer readable storage medium provided in the embodiments of the present application stores a computer program, and the computer program can execute the above-mentioned Figure 1 The steps of the data screening method in the method embodiment are described above, and the specific implementation manners can be referred to the method embodiment, which will not be repeated here.

[0177] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0178] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0179] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0180] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0181] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0182] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of data screening, characterized by, The data screening method comprises: in response to a feature screening instruction, obtaining at least two data cubes to be screened indicated by the feature screening instruction; wherein the data cubes to be screened comprise feature data with different feature values under different feature dimensions; for each data cube to be screened, determining a screening condition for screening a plurality of feature data in the data cube to be screened; wherein the screening condition comprises a target feature dimension, a target feature value, and a relationship expression; the target feature value is a feature value of each feature data in the target feature dimension; the relationship expression represents a logical relationship between the target feature dimension and the target feature value; based on the target feature dimension, the target feature value, and the relationship expression, generating an SQL screening statement for screening the data cube to be screened; executing the SQL screening statement to generate an execution result set of the data cube to be screened; calling a preset aggregation function indicated by the feature screening instruction to aggregate the feature data in the execution result set of the at least two data cubes to be screened, and obtaining a screening result of the at least two data cubes to be screened; wherein the SQL screening statement for screening each data cube to be screened is generated by: based on the target feature dimension, the target feature value, and the relationship expression, generating a logical expression for screening the data cube to be screened; substituting the logical expression into a preset SQL statement template to generate an SQL screening statement for screening the data cube to be screened.

2. The data screening method of claim 1, wherein, The execution of the SQL screening statement to generate the execution result set of the data cube to be screened comprises: parsing the SQL screening statement to obtain SQL node information and metadata information of the SQL screening statement; based on the SQL node information and the metadata information, generating an initial logical plan of the SQL screening statement; optimizing the initial logical plan to obtain an executable SQL statement; executing the executable SQL statement to generate the execution result set of the data cube to be screened.

3. The data screening method of claim 1, wherein, The calling of the preset aggregation function indicated by the feature screening instruction to aggregate the feature data in the execution result set of the at least two data cubes to be screened to obtain the screening result of the at least two data cubes to be screened comprises: calling the preset aggregation function indicated by the feature screening instruction; taking the feature data in each execution result set as input data of the preset aggregation function, and using the preset aggregation function to aggregate the feature data in the at least two execution result sets to obtain an output result of the preset aggregation function; determining the output result as the screening result of the at least two data cubes to be screened.

4. The data screening method according to claim 1 or 3, characterized by, The preset aggregation function indicated by the feature screening instruction is called by the following steps: in response to a feature screening instruction, determining the data type, display form, and data screening operation of the screening result for screening the feature data in the at least two execution result sets. determine a preset aggregation function used for filtering feature data in at least two execution result sets based on the data type, the display form, and the data filtering operation, and generate a SQL call statement for calling the preset aggregation function; execute the SQL call statement to call the preset aggregation function.

5. The data screening method of claim 3, wherein, The preset aggregation function includes a union aggregation function; the feature data in each execution result set is input into the union aggregation function one by one, and the union aggregation function is used to merge the feature data in at least two execution result sets to obtain a merged result set; and the merged result set is determined as the output result of the union aggregation function. The preset aggregation function includes an intersection aggregation function; the feature data in each execution result set is input into the intersection aggregation function one by one, and the intersection aggregation function is used to filter out at least one target feature data common in at least two execution result sets; the at least one target feature data is used to form a common result set; and the common result set is determined as the output result of the intersection aggregation function. The preset aggregation function includes an intersection aggregation function; the feature data in each execution result set is input into the intersection aggregation function one by one, and the intersection aggregation function is used to filter out at least one target feature data common in at least two execution result sets; the at least one target feature data is used to form a common result set; and the common result set is determined as the output result of the intersection aggregation function.

6. The data screening method of claim 3, wherein, The data filtering device includes: a data acquisition module configured to acquire at least two data cubes to be filtered in response to a feature filtering instruction, wherein the data cubes to be filtered include feature data having different feature values in different feature dimensions; a filtering condition determination module configured to determine, for each data cube to be filtered, a filtering condition for filtering a plurality of feature data in the data cube to be filtered, wherein the filtering condition includes a target feature dimension, a target feature value, and a relationship expression; the target feature value is a feature value of each feature data in the target feature dimension; and the relationship expression represents a logical relationship between the target feature dimension and the target feature value; a statement generation module configured to generate a SQL filtering statement for filtering the data cube to be filtered based on the target feature dimension, the target feature value, and the relationship expression; 7. A data screening device characterized by comprising: a result generation module configured to execute the SQL filtering statement to generate an execution result set of the data cube to be filtered; a result filtering module configured to call a preset aggregation function indicated by the feature filtering instruction to aggregate feature data in execution result sets of the at least two data cubes to be filtered, and obtain a filtering result of the at least two data cubes to be filtered. The statement generation module is further configured to: ​ ​ ​ ​ generate a logical expression for screening the data cube to be screened based on the target feature dimension, the target feature value and the relationship expression; substitute the logical expression into a preset SQL statement template to generate an SQL screening statement for screening the data cube to be screened.

8. An electronic device, comprising: comprise: a processor, a memory and a bus, the memory storing machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to perform the steps of the data screening method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the data screening method according to any one of claims 1 to 6.

Images