Data processing method, device, equipment and computer-readable storage medium

Abstract

The present invention discloses a data processing method, apparatus, device, and computer-readable storage medium. The method includes: receiving data source configuration parameters and establishing a connection with a target data source based on the data source configuration parameters; the target data source includes multiple data tables; receiving a data selection instruction and determining a metric table and a dimension table from the multiple data tables based on the data selection instruction; receiving a data association instruction and, based on the association relationship included in the data association instruction, associating the metric table and the dimension table according to a data association model; receiving a parameter setting instruction and generating expansion information based on the parameter setting instruction; and using the expansion information to perform data expansion on the associated metric table and dimension table to obtain a data cube. The present invention improves data processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] The multidimensional data model of the data warehouse is established to meet the needs of users to query and analyze data from multiple angles and levels. It is a database model based on fact tables and dimensions. Its basic application is to realize online analytical processing (OLAP).

[0003] Of course, the data display, query and acquisition through the multidimensional data model are the demonstration of its role, but its real role is realized in that various multidimensional models can be established according to different data needs through the data warehouse, and data marts can be formed to be open to different user groups. In other words, various data products customized according to needs are placed in the data mart for different data consumers to purchase.

[0004] In the related art, the implementation method of this multidimensional data model is generally based on the Hadoop ecosystem to complete data expansion of data dimensions and metrics. Usually, using the Hadoop ecosystem to complete data expansion requires deploying multiple nodes to complete data calculations, which is a specific implementation of the divide-and-conquer idea. However, Hadoop is usually used for data volumes in the hundreds of millions. For data volumes in the tens of millions, deploying multiple nodes will cause under-utilization of resources on some nodes, which will result in a waste of hardware resources. Moreover, if a relational data model is used to expand data, a large number of complex operations need to be performed manually when using structured query language for data volumes in the tens of millions, and the data expansion performance is poor. In other words, in the related art, there is a lack of high-performance processing methods for data processing in the tens of millions. Summary of the Invention

[0005] The present invention is intended to provide a data processing method, apparatus, device, and computer-readable storage medium that can improve data processing efficiency.

[0006] The technical solution of the present invention is achieved as follows:

[0007] A data processing method includes: receiving data source configuration parameters, establishing a connection with a target data source according to the data source configuration parameters; the data source configuration parameters are used to characterize descriptive information of the target data source, the target data source including multiple data tables; receiving a data selection instruction, determining at least one metric table and at least one dimension table from the multiple data tables according to the data selection instruction; receiving a data association instruction, and associating the at least one metric table and the at least one dimension table with each other according to a data association model according to the association relationship included in the data association instruction; receiving a parameter setting instruction, generating expansion information according to the parameter setting instruction; and using the expansion information to perform data expansion on the associated at least one metric table and the at least one dimension table to obtain a data cube.

[0008] A data processing device includes: a business data layer for receiving data source configuration parameters and establishing a connection with a target data source according to the data source configuration parameters; the data source configuration parameters are used to represent descriptive information of the target data source; the target data source includes multiple data tables; a data warehouse layer for receiving data selection instructions and determining at least one metric table and at least one dimension table from the multiple data tables according to the data selection instructions; a relationship layer for receiving data association instructions and, according to the association relationship included in the data association instructions, associating the at least one metric table and the at least one dimension table with each other according to a data association model; an engine layer for receiving parameter setting instructions and generating expansion information according to the parameter setting instructions; using the expansion information, performing data expansion on the associated at least one metric table and the at least one dimension table to obtain a data cube.

[0009] A data processing device comprises: a memory for storing executable data instructions; and a processor for implementing the above-mentioned data processing method when executing the executable instructions stored in the memory.

[0010] A computer-readable storage medium stores executable instructions for causing a processor to execute the instructions to implement the above-mentioned data processing method.

[0011] The data processing method, apparatus, device and computer-readable storage medium provided by the present invention receive data source configuration parameters and establish a connection with a target data source according to the data source configuration parameters; the target data source includes multiple data tables; receives a data selection instruction and determines at least one metric table and at least one dimension table from the multiple data tables according to the data selection instruction; receives a data association instruction and associates the at least one metric table and the at least one dimension table with each other according to a data association model according to the association relationship included in the data association instruction; receives a parameter setting instruction and generates expansion information according to the parameter setting instruction; uses the expansion information to perform data expansion on the at least one metric table and the at least one dimension table after association to obtain a data cube. By adopting the present technical solution, dimension tables and metric tables can be generated and associated according to multiple different instructions input by the user, the expansion information required by the user can be generated, and the data expansion processing of the associated dimension tables and metric tables can be performed, thereby obtaining the data cube required by the user. Therefore, the user only needs to input multiple different instructions to quickly obtain the required data cube, thereby improving the efficiency of data processing. Moreover, since the data processing method of the present invention can perform data expansion processing in one device, there will be no problem of undersaturated resource usage of some nodes due to deploying multiple nodes for data expansion, thereby causing waste of hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An optional flowchart of a data processing method provided by an embodiment of the present invention;

[0013] Figure 2 An exemplary snowflake model data cube provided by an embodiment of the present invention;

[0014] Figure 3 Another optional flowchart of the data processing method provided by the embodiment of the present invention;

[0015] Figure 4 A schematic diagram of another optional flow chart of the data processing method provided in an embodiment of the present invention;

[0016] Figure 5 A schematic diagram of another optional flow chart of the data processing method provided in the embodiment of the present invention;

[0017] Figure 6 Another optional flowchart of the data processing method provided by the embodiment of the present invention;

[0018] Figure 7 A schematic diagram of another optional flow chart of the data processing method provided in an embodiment of the present invention;

[0019] Figure 8A schematic structural diagram of a data processing device provided in an embodiment of the present invention;

[0020] Figure 9 A schematic structural diagram of a data processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0022] In the following description, reference is made to “embodiments of the present invention”, which describes a subset of all possible embodiments, but it will be understood that “embodiments of the present invention” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0023] In the following description, the terms "first\second\third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0024] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention pertains. The terms used in the embodiments of the present invention are for the purpose of describing the embodiments of the present invention only and are not intended to limit the present invention.

[0025] Figure 1 This is an optional flow chart of the data processing method provided by the embodiment of the present invention, which will be combined with Figure 1 The data processing method in the embodiment of the present invention is applied to a device installed with a data processing application, and the device implements data expansion by calling the data processing application; the data processing application can be stored in the device in the form of a plug-in.

[0026] S101 , receiving data source configuration parameters, and establishing a connection with a target data source according to the data source configuration parameters; the data source configuration parameters are used to characterize description information of the target data source, and the target data source includes multiple data tables.

[0027] In an embodiment of the present invention, a device can receive data source configuration parameters representing descriptive information of a target data source input by a user and establish a connection with the target data source based on the data source configuration parameters. The device can connect to various types of data sources based on the configuration information input by the user. For example, data sources may include MySQL, Oracle, and PostgreSQL, and each data source may include multiple data tables. The present invention does not specifically limit the types of data sources.

[0028] In an embodiment of the present invention, after the user configures the target data source to be connected through the data source configuration parameters and the device establishes a connection with the target data source, the device can receive a test instruction input by the user and test whether it is connected with the target data source according to the test instruction. When it is not connected, a prompt message can be sent to the user to enable the user to reconfigure the data source configuration parameters; when it is connected, the device displays multiple data tables in the data source.

[0029] In embodiments of the present invention, the data source configuration parameters may be a link to a target data source. The device may access the target data source based on the link input by the user, thereby establishing a connection with the target data source. In some embodiments of the present invention, the device may store the data source configuration parameters input by the user so that the next time the user inputs the data source configuration parameters, the stored data source configuration information is directly displayed, thereby improving data source connection efficiency.

[0030] In an embodiment of the present invention, after the device successfully connects to the target data source, it can display multiple data tables and data table structures in the target data source to the user, so that the user can view the data table structure, filter the fields in the data table, and modify the field type and field size in the data table.

[0031] S102: Receive a data selection instruction, and determine at least one metric table and at least one dimension table from multiple data tables according to the data selection instruction.

[0032] In an embodiment of the present invention, the device may receive a data selection instruction input by a user, parse the instruction information contained in the data selection instruction, and determine at least one metric table and at least one dimension table from multiple data tables according to the instruction information.

[0033] S103: Receive a data association instruction, and, according to the association relationship included in the data association instruction, associate at least one metric table and at least one dimension table with each other according to a data association model.

[0034] In an embodiment of the present invention, after determining at least one metric table and at least one dimension table and receiving a data association instruction input by the user, the device can parse the association relationship between the at least one metric table and the at least one dimension table set by the user from the data association instruction, and associate the at least one metric table and the at least one dimension table according to the data association model based on the association relationship.

[0035] In an embodiment of the present invention, the data association model may include at least one of a snowflake model, a star model, and a constellation model. Exemplarily, the apparatus may associate the at least one metric table with the at least one dimension table according to the snowflake model. Since the snowflake model has a parent-child relationship, adopting the snowflake model can achieve better scalability.

[0036] Figure 2 is an exemplary data cube of a snowflake model provided by an embodiment of the present invention; Figure 2 As shown in the figure, the fact table mainly contains two kinds of information: dimension and measure. The detailed description of the dimension is recorded in the dimension table. The dimension attribute in the fact table is just a key to the dimension table and does not record specific information. The measure generally records the corresponding value of the event, such as the sales quantity and sales amount of the product. The information in the dimension table can generally be hierarchical. For example, Figure 2 As shown in the figure, the "year", "month", and "day" in the dimension table of the "time dimension", and the "province", "city", and "county" in the dimension table of the "regional dimension" are hierarchical information to ensure that the metrics in the fact table can be expanded (i.e., aggregated) at different granularities. For example, after the dimensions and metrics are aggregated, multiple aggregated data such as "sales of goods in 2010" and "sales from Shanghai" can be formed.

[0037] S104: Receive a parameter setting instruction, and generate expansion information according to the parameter setting instruction.

[0038] Because different services and users require different data cubes, the required expansion information also varies. In an embodiment of the present invention, a device can receive a parameter setting instruction input by a user, parse the parameter information contained in the parameter setting instruction, and generate the expansion information required by the user based on the parameter information. In this embodiment of the present invention, the parameter setting instruction can include a time point, a frequency, and an expansion field.

[0039] In an embodiment of the present invention, the expansion information includes an expansion template and an expansion task; wherein the expansion template includes: an expansion field, a driving table, a non-driving table group, a driving table execution order, a non-driving table execution order, and an indicator calculation logic. The expansion task includes: the execution frequency and execution time of the expansion template. For example, when a measurement table and two dimension tables are generated, and the measurement table has been associated with the two dimension tables, the device can set the instruction according to the parameter sent by the user, select the expansion field pointed to by the instruction from the generated dimension table or measurement table, and set the dimension table or measurement table pointed to by the instruction as the driving table, and set the measurement table or dimension table pointed to by the instruction as the non-driving table, and set the specific execution order between the non-driving table and the driving table according to the instruction, and set the calculation logic between the various metrics in the measurement table according to the instruction.

[0040] Here, the execution time is the specific time point at which the expansion occurs, for example, 17:00. The expansion field refers to the field selected by the user from the linked dimension or metric table. The driving table group is the data table specified by the user from the linked dimension or metric table, and the non-driving table group is all data tables in the linked dimension or metric table that are not designated as driving tables. The driving table group execution order indicates the execution order of the driving tables, while the non-driving table group execution order indicates the execution order of the non-driving tables. The indicator calculation logic represents the calculation relationship between the metrics in the linked metric table, for example, (metric 1 + metric 2) * metric 3.

[0041] In some embodiments of the present invention, the driving table is the dimension table with the smallest amount of data among the associated dimension tables, or the metric table with the smallest amount of data among the associated metric tables; this can achieve a good data expansion effect. In some embodiments of the present invention, the data amount is the number of fields included in the dimension table or metric table.

[0042] S105 : Using the expansion information, perform data expansion on the at least one metric table and the at least one dimension table after association to obtain a data cube.

[0043] In an embodiment of the present invention, after generating an expansion template and expansion task, the device can perform a GroupBy calculation on the associated dimension table and metric table based on the generated expansion template and expansion task, thereby obtaining the data cube required by the user and inputting the data cube into a cube database for storage and user access. For example, after generating a metric table and two dimension tables, selecting the expansion fields, setting the specific execution order between the driver table, non-driver table, and non-driver table and driver table, and setting the calculation logic between the various metrics in the metric table, the device can perform a GroupBy calculation on the associated dimension table and metric table to ultimately generate a data cube.

[0044] In the above embodiment, since the device can generate and associate dimension tables and metric tables according to multiple different instructions input by the user, generate the expansion information required by the user, and perform data expansion processing on the associated dimension tables and metric tables, thereby obtaining the data cube required by the user, the user only needs to input multiple different instructions to quickly obtain the required data cube, thereby improving the efficiency of data processing; moreover, since the data processing method of the present invention can perform data expansion processing in one device, there will be no problem of undersaturated resource usage of some nodes and waste of hardware resources due to deploying multiple nodes for data expansion.

[0045] In some embodiments of the present invention, each metric table and each dimension table includes an association relationship field; the above S103 can be implemented by S1031, specifically:

[0046] S1031. Receive a data association instruction, and associate the at least one metric table and the at least one dimension table according to a data association model based on the association relationship included in the data association instruction and the association relationship fields in the at least one metric table and the at least one dimension table.

[0047] In an embodiment of the present invention, after receiving a data association instruction, the device can parse the association relationship set by the user in the data association instruction, and, based on the association relationship set by the user and the association relationship field in the at least one metric table and the association relationship field in the at least one dimension table, associate the at least one metric table with the at least one dimension table according to the data association model. For example, when the data association instruction includes field A in dimension table 1 and field B in the metric table, the device can associate field A in dimension table 1 with field B in dimension table 1. For another example, the generated dimension tables are: Table A, Table B, and Table C; the generated metric tables are: Table 1 and Table 2; and the user's association instruction is to associate Table A and Table B with Table 1, and Table C with Table 2, the device can associate Table A and Table B with Table 1, and Table C with Table 2, based on the association relationship fields in each table.

[0048] In some embodiments of the present invention, the above S105 may be implemented by S1051, specifically:

[0049] S1051. Using the expansion information, perform a multi-dimensional Cartesian product calculation on the at least one associated metric table and the at least one dimension table to obtain a data cube.

[0050] In an embodiment of the present invention, after generating an expansion template and an expansion task, the device can calculate a multi-dimensional Cartesian product of the associated dimension table and metric table based on the generated expansion template and expansion task, thereby obtaining the data cube required by the user, and inputting the data cube into a cube database for storage for user use. For example, after generating a metric table and two dimension tables, and selecting the expansion fields, setting the specific execution order between the driver table, non-driver table, and non-driver table and driver table, and setting the calculation logic between the various metrics in the metric table, the device can calculate the multi-dimensional Cartesian product of the associated dimension table and metric table, and use the final calculation result as the generated data cube.

[0051] Figure 3 is another optional flow chart of the data processing method provided by an embodiment of the present invention. Figure 1 S102 in the above can be realized by S1021-S1023, which will be combined with Figure 3 The steps shown are explained.

[0052] S1021. Receive a data selection instruction, and select a first target data table from multiple data tables according to the data selection instruction; the first target data table is the data table specified by the data selection instruction.

[0053] In an embodiment of the present invention, when the user determines that there is a data table among multiple data tables that is the same as the required metric table or dimension table, and inputs a data selection instruction, the device can parse the data table identifier contained in the data selection instruction and directly select the corresponding data table from the multiple data tables according to the data table identifier.

[0054] S1022: Use the first data table in the first target data table as a dimension table; the fields in the first data table represent dimension information.

[0055] In an embodiment of the present invention, when the user determines that the fields in the data table all represent dimension information and inputs a data selection instruction, the device can use the corresponding data table as a dimension table according to the data table identifier in the data selection instruction.

[0056] S1023. Use the second data table in the first target data table as a measurement table; the fields in the second data table represent measurement information.

[0057] In an embodiment of the present invention, when the user determines that all fields in the data table represent measurement information and inputs a data selection instruction, the device can use the corresponding data table as the measurement table according to the data table identifier in the data selection instruction.

[0058] Figure 4This is another optional flow chart of the data processing method provided by an embodiment of the present invention, wherein the data selection instruction includes a data table selection instruction and a field selection instruction; Figure 1 S102 in the above can be realized by S1024-S1026, which will be combined with Figure 4 The steps shown are explained.

[0059] S1024, receiving a data table selection instruction, and selecting a second target data table from multiple data tables according to the data table selection instruction; each data table includes fields; the second target data table is the data table specified by the data table selection instruction.

[0060] In an embodiment of the present invention, when a user determines that the required dimension information or measurement information exists in certain data tables by viewing the fields of the data tables, and inputs a data table selection instruction including the identifiers of these data tables, the device can select a corresponding data table from multiple data tables according to the data table selection instruction.

[0061] In the embodiment of the present invention, each data table includes fields, and the data in each data table are the fields included in the table.

[0062] S1025. Receive a field selection instruction, mark the target field in the second target data table according to the field selection instruction, and obtain a marked second target data table; the target field is the field specified by the field selection instruction.

[0063] In an embodiment of the present invention, when the device receives a data table viewing instruction input by the user, it displays the table structure of the corresponding data table and information such as the fields in the data table to the user, and supports the user to filter the fields in the data table, filter the field types, and modify the field size.

[0064] In an embodiment of the present invention, when a selected data table includes many fields not needed for data expansion, the user can filter the fields required for data expansion by sending a field selection instruction to the device. Based on the received field selection instruction, the device marks the fields in the corresponding data table to which the field selection instruction refers. Upon receiving a field selection end instruction from the user, the device locally stores the marked data table based on the field selection end instruction.

[0065] S1026. Generate at least one metric table including the target field and at least one dimension table including the target field according to the marked data table.

[0066] In an embodiment of the present invention, after locally storing the tagged data table, the device may perform data cleansing on the stored tagged data table to generate at least one metric table and at least one dimension table. In an embodiment of the present invention, a tagged data table generates a dimension table including a target field or a metric table including a target field.

[0067] In some embodiments of the present invention, the above S1026 may be implemented by S1, specifically as follows:

[0068] S1. Perform an extraction and conversion operation on a data table with a mark to generate at least one metric table including a target field and at least one dimension table including a target field.

[0069] In an embodiment of the present invention, the device can perform an extraction and conversion operation on each marked data table to generate and store a dimension table or a metric table. This metric table or dimension table includes all marked fields in the marked data table. For example, when all marked fields are dimension fields, the generated data table is a dimension table; when all marked fields are metric fields, the generated data table is a dimension table. In an embodiment of the invention, the device can locally store the generated dimension table and metric table.

[0070] In the above embodiment, since the device can intuitively display the data table and the corresponding data table structure, and supports the user to check the fields in the data table to generate the data table to be associated, compared with the related technology of selecting the data to be associated through commands in the command line in the database language, it is more operational, simpler and more intuitive.

[0071] Figure 5 This is another optional flow chart of the data processing method provided by the embodiment of the present invention. Figure 1 In the example where S100 is included before S102, S100 is included before S101. Figure 5 The steps shown are explained.

[0072] S100. Receive a creation instruction, and create at least one first resource set and at least one second resource set according to the creation instruction, wherein each first resource set is used to store a dimension table of the same type, and each second resource set is used to store a measurement table of the same type.

[0073] In the above embodiment, the device can create a first resource set for storing dimension tables of the same type and a second resource set for storing measurement tables of the same type according to a creation instruction input by the user, so as to store the measurement tables and dimension tables determined subsequently. Exemplarily, the first resource set and the second resource set can be folders. Here, creating a first resource set for storing dimension tables of the same type and a second resource set for storing measurement tables of the same type can facilitate the rapid location and search of the required dimension tables and measurement tables, and facilitate the search of dimension tables and measurement tables.

[0074] In some embodiments of the present invention, the type of dimension table or measurement table can be defined according to specific business; for example, when dimension table 1 and dimension table 2 belong to the same subject, it means that dimension table 1 and dimension table 2 are of the same type and can be stored in the same resource collection; for another example, when dimension table 1 and dimension table 2 are both virtual resources, dimension table 1 and dimension table 2 are of the same type and can be stored in the same resource collection.

[0075] Figure 6 is another optional flow chart of the data processing method provided by the embodiment of the present invention. Figure 5 S100 in the above example may include S201-S204, which will be combined with Figure 6 The steps shown are explained.

[0076] S201: Determine business information corresponding to at least one first resource set.

[0077] In an embodiment of the present invention, the device may determine the service information of the first resource set based on the service information corresponding to the user who created the first resource set. In an embodiment of the present invention, the service information is a service type. In an embodiment of the present invention, a tenant corresponds to at least one user, and the at least one user has the same service type.

[0078] S202: Set access permissions for at least one first resource set according to business information.

[0079] In an embodiment of the present invention, a device can set a tenant with the same service type as the first resource set, allowing access to the first resource set. For example, if the service type corresponding to tenant 1 is A, the service type corresponding to tenant 2 is B, and the service type of first resource set a is also A, tenant 1 can access first resource set a, but tenant 2 cannot.

[0080] S203: Determine business information corresponding to at least one second resource set.

[0081] In an embodiment of the present invention, the apparatus may determine the service information of the second resource set according to the service information corresponding to the user who created the second resource set.

[0082] S204: Set access permissions for at least one second resource set according to the business information; the access permissions are used to restrict data access operations of tenants without permission.

[0083] In an embodiment of the present invention, a device can set a tenant with the same service type as the second resource set to access the second resource set. For example, if the service type corresponding to tenant 1 is A, the service type corresponding to tenant 2 is B, and the service type of second resource set b is also A, tenant 1 can access second resource set b, but tenant 2 cannot.

[0084] In an embodiment of the present invention, the data access operation may be an operation such as data viewing and data storage.

[0085] In the embodiment of the present invention, the terminal sets access permissions for the created resource set, so that different tenants can only access and operate the resource set corresponding to themselves, thereby ensuring data security.

[0086] Figure 7 This is another optional flow chart of the data processing method provided by the embodiment of the present invention. Figure 6 S102 in the above example may include S205-S208, which will be combined with Figure 7 The steps shown are explained.

[0087] S205. Receive a data storage request, where the data storage request carries the identification information of the tenant, and the identification information of the target first resource set or the identification information of the target second resource set; the target first resource set is a first resource set in at least one first resource set; and the target second resource set is a second resource set in at least one second resource set.

[0088] In an embodiment of the present invention, when the user needs to store the determined dimension table to a first resource set, the user can input a data storage instruction including the identification information of the tenant corresponding to the user and the identification information of the target first resource set; and, when the user needs to store the determined measurement table to a second resource set, the user can input a data storage instruction including the identification information of the tenant corresponding to the user and the identification information of the target second resource set.

[0089] In an embodiment of the present invention, the tenant identification information may be the tenant's name, etc. The target first resource set identification information may be the target first resource set's name, etc. Similarly, the target second resource set identification information may be the target second resource set's name, etc.

[0090] S206: Determine the target access rights of the tenant based on the identification information of the tenant.

[0091] In an embodiment of the present invention, the device can determine the business type corresponding to the tenant based on the tenant's name, and determine the tenant's target access rights based on the correspondence between the business type and the target access rights. In an embodiment of the present invention, the correspondence between the business type and the target access rights can be pre-set.

[0092] S207: Determine the access rights of the target first resource set according to the identification information of the target first resource set, or determine the access rights of the target second resource set according to the identification information of the target second resource set.

[0093] In an embodiment of the present invention, the device can determine the business type corresponding to the target first resource set based on the name of the target first resource set, and determine the access rights of the target first resource set based on the correspondence between the business type and the access rights; determine the business type corresponding to the target second resource set based on the name of the target second resource set, and determine the access rights of the target second resource set based on the correspondence between the business type and the access rights.

[0094] S208. When the target access permission satisfies the access permission of the target first resource set, at least one dimension table is allowed to be stored in the target first resource set; or, when the target access permission satisfies the access permission of the target second resource set, at least one measurement table is allowed to be stored in the target second resource set.

[0095] In an embodiment of the present invention, when the device determines that the tenant's access rights meet the access rights of the target first resource set, the generated dimension table or the dimension table directly selected from multiple data tables according to the data selection instruction can be stored in the target first resource set; and when the device determines that the tenant's access rights meet the access rights of the target second resource set, the generated measurement table or the measurement table directly selected from multiple data tables according to the data selection instruction can be stored in the target second resource set.

[0096] In the above embodiment, when storing the generated dimension tables and metric tables, the tenants' permissions are verified, so that different tenants can only access and operate the resource sets corresponding to themselves, thereby ensuring data security.

[0097] In one embodiment of the present invention, Figure 7 S103 can be achieved through S301-S302:

[0098] S301. Receive a data association instruction, and select at least one dimension table from a target first resource set and at least one measurement table from a target second resource set according to identification information of at least one measurement table and identification information of at least one dimension table included in the data association instruction.

[0099] In an embodiment of the present invention, when a user inputs a data association instruction carrying identification information of a measurement table and identification information of a dimension table, the device can determine, based on the identification information of the dimension table, a target first resource set where the dimension table is located from multiple first resource sets, and select the dimension table from the target first resource set; and the device can determine, based on the identification information of the measurement table, a target second resource set where the measurement table is located from multiple first resource sets, and select the measurement table from the target second resource set.

[0100] S302: According to the association relationship included in the data association instruction, the selected at least one metric table and the selected at least one dimension table are associated with each other according to the data association model.

[0101] In an embodiment of the present invention, after receiving the data association instruction input by the user, the device can parse the association relationship set by the user in the data association instruction, and based on the association relationship set by the user, the association relationship field in the selected measurement table, and the association relationship field in the selected dimension table, the selected measurement table and dimension table are associated with each other according to the data association model.

[0102] An embodiment of the present invention further provides a device for implementing the above-mentioned data processing method. Figure 8 Schematic diagram of part of the structure of the data processing device provided by the embodiment of the present invention. Figure 8 As shown, the data processing device 1 includes: a business data layer 11, which is used to receive data source configuration parameters and establish a connection with a target data source according to the data source configuration parameters; the data source configuration parameters are used to characterize the descriptive information of the target data source; the target data source includes multiple data tables; a data warehouse layer 12, which is used to receive data selection instructions and determine at least one metric table and at least one dimension table from the multiple data tables according to the data selection instructions; a relationship layer 13, which is used to receive data association instructions and associate the at least one metric table and the at least one dimension table with each other according to a data association model according to the association relationship included in the data association instructions; an engine layer 14, which is used to receive parameter setting instructions and generate expansion information according to the parameter setting instructions; and use the expansion information to expand the data of the associated at least one metric table and the at least one dimension table to obtain a data cube.

[0103] In some embodiments, each metric table and each dimension table includes an association relationship field; the relationship layer 13 is also used to receive a data association instruction, and according to the association relationship included in the data association instruction and the association relationship fields in the at least one metric table and the at least one dimension table, associate the at least one metric table and the at least one dimension table with each other according to the data association model.

[0104] In some embodiments, the engine layer 14 is further configured to use the expansion information to perform a multi-dimensional Cartesian product calculation on the at least one metric table and the at least one dimension table after association to obtain a data cube.

[0105] In some embodiments, the data warehouse layer 12 is also used to receive data selection instructions, and select a corresponding data table from the multiple data tables according to the data selection instructions; use the first data table in the selected data table as a dimension table; the fields in the first data table represent dimension information; use the second data table in the selected data table as a measurement table; the fields in the second data table represent measurement information.

[0106] In some embodiments, the data selection instruction includes a data table selection instruction and a field selection instruction; the data warehouse layer 12 is further used to receive the data table selection instruction, and select a second target data table from the multiple data tables according to the data table selection instruction; each data table contains fields; the second target data table is the data table specified by the data table selection instruction; receive the field selection instruction, mark the target field in the second target data table according to the field selection instruction, and obtain a marked second target data table; the target field is the field specified by the field selection instruction; based on the marked data table, generate at least one metric table including the target field and at least one dimension table including the target field.

[0107] In some embodiments, the data warehouse layer 12 is further used to receive a creation instruction before determining at least one metric table and at least one dimension table from the multiple data tables according to the data selection instruction, and to create at least one first resource set and at least one second resource set according to the creation instruction, wherein each first resource set is used to store the determined dimension tables of the same type, and each second resource set is used to store the determined metric tables of the same type.

[0108] In some embodiments, the data warehouse layer 12 is also used to determine the business information corresponding to the at least one first resource set after creating at least one first resource set and at least one second resource set respectively according to the creation instruction; set access permissions for the at least one first resource set based on the business information; determine the business information corresponding to the at least one second resource set; set access permissions for the at least one second resource set based on the business information; the access permissions are used to limit data access operations of tenants without permission.

[0109] In some embodiments, the data warehouse layer 12 is also used to receive a data storage request after determining at least one metric table and at least one dimension table from the multiple data tables according to the data selection instruction, the data storage request carrying the identification information of the tenant, and the identification information of the target first resource set or the identification information of the target second resource set; the target first resource set is a first resource set in the at least one first resource set; the target second resource set is a second resource set in the at least one second resource set; determine the target access rights of the tenant based on the identification information of the tenant; determine the access rights of the target first resource set based on the identification information of the target first resource set, or determine the access rights of the target second resource set based on the identification information of the target second resource set; when the target access rights meet the access rights of the target first resource set, allow the at least one dimension table to be stored in the target first resource set; or, when the target access rights meet the access rights of the target second resource set, allow the at least one metric table to be stored in the target second resource set.

[0110] In some embodiments, the relationship layer 13 is also used to receive data association instructions, select the at least one dimension table from the target first resource set and select the at least one measurement table from the target second resource set according to the identification information of the at least one measurement table and the identification information of the at least one dimension table included in the data association instructions; and associate the selected at least one measurement table and the selected at least one dimension table with each other according to the data association model according to the association relationship included in the data association instruction.

[0111] In some embodiments, the expansion information includes: an expansion task and an expansion template; the expansion task includes the execution frequency and execution time of the expansion template; the expansion template includes: an expansion field, a driving table group, a non-driving table group, a driving table execution order, a non-driving table execution order and an indicator calculation logic; wherein, the driving table is a dimension table or a measurement table after association; the non-driving table is a dimension table and / or measurement table that is not selected as the driving table after association.

[0112] In some embodiments, the driving table is a dimension table with the smallest amount of data among the associated dimension tables, or a metric table with the smallest amount of data among the associated metric tables.

[0113] In some embodiments, the data association model includes at least one of the following: a snowflake model, a star model, and a constellation model.

[0114] In some embodiments, the data warehouse layer 12 is further configured to perform extraction and conversion operations on the marked data tables; and generate at least one metric table and at least one dimension table according to the extraction and conversion results.

[0115] An embodiment of the present invention further provides a data processing device for implementing the above-mentioned data processing method. Figure 9 Schematic diagram of part of the structure of the data processing device provided by the embodiment of the present invention. Figure 9 As shown, the data processing device 1 includes: a memory 21 and a processor 22, and the memory 21 is connected to the processor 22 via a bus 23; the memory 21 is used to store executable data instructions; the processor 22 is used to implement the above method when executing the executable instructions stored in the memory, such as the data processing method described in the above embodiment.

[0116] The embodiment of the present invention further provides a computer-readable storage medium storing executable instructions for causing a processor to execute the above method, such as the data processing method described in the above embodiment. The executable instructions in the embodiment of the present invention are computer programs.

[0117] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0118] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0121] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Receive data source configuration parameters, and establish a connection with a target data source according to the data source configuration parameters; The data source configuration parameters are used to characterize the description information of the target data source, and the target data source includes multiple data tables; receiving a data selection instruction, and determining at least one metric table and at least one dimension table from the plurality of data tables according to the data selection instruction; receiving a data association instruction, and according to the association relationship included in the data association instruction, associating the at least one metric table and the at least one dimension table with each other according to a data association model; receiving a parameter setting instruction, and generating expansion information according to the parameter setting instruction; The expansion information includes a driving table, which is a dimension table with the smallest amount of data in the associated dimension tables, or a metric table with the smallest amount of data in the associated metric tables; The expansion information is used to perform data expansion on the associated at least one metric table and the at least one dimension table in combination with the selected driving table in the same device to obtain a data cube, and the data cube is stored in a database.

2. The method according to claim 1, wherein Each metric table and each dimension table includes an association relationship field; the receiving a data association instruction and, according to the association relationship included in the data association instruction, associating the at least one metric table and the at least one dimension table with each other according to a data association model, include: A data association instruction is received, and according to the association relationship included in the data association instruction and the association relationship fields in the at least one metric table and the at least one dimension table, the at least one metric table and the at least one dimension table are associated with each other according to a data association model.

3. The method according to claim 2, wherein The receiving a data selection instruction and determining at least one metric table and at least one dimension table from the plurality of data tables according to the data selection instruction includes: receiving a data selection instruction, and selecting a first target data table from the plurality of data tables according to the data selection instruction; the first target data table being the data table specified by the data selection instruction; The first data table in the first target data table is used as a dimension table; the fields in the first data table represent dimension information; The second data table in the first target data table is used as a measurement table; the fields in the second data table represent measurement information.

4. The method according to claim 2, wherein The data selection instruction includes a data table selection instruction and a field selection instruction; the receiving of the data selection instruction and determining at least one metric table and at least one dimension table from the plurality of data tables according to the data selection instruction include: receiving the data table selection instruction, and selecting a second target data table from the plurality of data tables according to the data table selection instruction; each data table includes fields; the second target data table is the data table specified by the data table selection instruction; receiving the field selection instruction, marking a target field in the second target data table according to the field selection instruction, and obtaining a marked second target data table; the target field being a field specified by the field selection instruction; At least one metric table including the target field and at least one dimension table including the target field are generated according to the marked data table.

5. The method according to claim 1, wherein Before determining at least one metric table and at least one dimension table from the plurality of data tables according to the data selection instruction, the method further includes: Receive a creation instruction, and create at least one first resource set and at least one second resource set according to the creation instruction, wherein each first resource set is used to store a dimension table of the same type, and each second resource set is used to store a measurement table of the same type.

6. The method according to claim 5, wherein After creating at least one first resource set and at least one second resource set respectively according to the creation instruction, the method further includes: Determining service information corresponding to the at least one first resource set; Setting access permissions for the at least one first resource set according to the business information; Determining service information corresponding to the at least one second resource set; Access permissions are set for the at least one second resource set according to the business information; the access permissions are used to restrict data access operations of tenants without permission.

7. The method according to claim 6, wherein After determining at least one metric table and at least one dimension table from the plurality of data tables according to the data selection instruction, the method further includes: Receive a data storage request, the data storage request carrying identification information of the tenant and identification information of a target first resource set or identification information of a target second resource set; the target first resource set is one of the at least one first resource set; the target second resource set is one of the at least one second resource set; Determining target access rights of the tenant based on the identification information of the tenant; Determining the access rights of the target first resource set according to the identification information of the target first resource set, or determining the access rights of the target second resource set according to the identification information of the target second resource set; When the target access permission satisfies the access permission of the target first resource set, the at least one dimension table is allowed to be stored in the target first resource set; or, when the target access permission satisfies the access permission of the target second resource set, the at least one measurement table is allowed to be stored in the target second resource set.

8. The method according to claim 7, wherein The receiving the data association instruction and associating the at least one metric table and the at least one dimension table with each other according to the association relationship included in the data association instruction according to a data association model include: receiving a data association instruction, and selecting, based on identification information of the at least one metric table and identification information of the at least one dimension table included in the data association instruction, the at least one dimension table from the target first resource set, and the at least one metric table from the target second resource set; According to the association relationship included in the data association instruction, the selected at least one metric table and the selected at least one dimension table are associated with each other according to a data association model.

9. The method according to claim 1, wherein The expansion information includes: expansion tasks and expansion templates; The expansion task includes the execution frequency and execution time of the expansion template; The expansion template includes: expansion fields, a driving table group, a non-driving table group, a driving table execution order, a non-driving table execution order and indicator calculation logic; wherein the non-driving table is a dimension table and / or a metric table that is not selected as the driving table after association.

10. A data processing device, characterized in that: include: The business data layer is used to receive data source configuration parameters and establish a connection with the target data source according to the data source configuration parameters; The data source configuration parameters are used to characterize the description information of the target data source; The target data source includes multiple data tables; The data warehouse layer is configured to receive a data selection instruction and determine at least one metric table and at least one dimension table from the plurality of data tables according to the data selection instruction; a relationship layer, configured to receive a data association instruction and, based on the association relationship included in the data association instruction, associate the at least one metric table and the at least one dimension table with each other according to a data association model; The engine layer is configured to receive parameter setting instructions and generate expansion information according to the parameter setting instructions; The expansion information includes a driving table, which is a dimension table with the smallest amount of data in the associated dimension tables, or a metric table with the smallest amount of data in the associated metric tables; The expansion information is used to perform data expansion on the associated at least one metric table and the at least one dimension table in combination with the selected driving table in the same device to obtain a data cube, and the data cube is stored in a database.

11. A data processing device, characterized in that: include: A memory for storing executable data instructions; A processor, configured to implement the method according to any one of claims 1 to 9 when executing the executable instructions stored in the memory.

12. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to cause a processor to implement the method according to any one of claims 1 to 9 when executed.

Images