A data processing method and device, and storage medium

Abstract

The present application provides a data processing method, comprising: constructing an operator set for each data layer in a data warehouse to obtain an operator set corresponding to each data layer; configuring an operator link based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer; constructing a data processing link corresponding to the data warehouse based on the data processing sub-link corresponding to each data layer using a particle swarm algorithm; and processing source data based on the data processing link to obtain a data mart for the data warehouse. This method shortens data processing time, improves data processing efficiency, and thereby enhances the overall performance of the data processing link.

Description

Technical Field

[0001] The present invention relates to the field of data services, and in particular to a data processing method and device, and a storage medium. Background Art

[0002] In the big data world, data warehouses (DWs) are crucial as a common data foundation. When building a data warehouse, the typical approach is to first organize the data into layers and then process the data layer by layer. However, traditional data warehouses primarily use online analytical processing (OLAP) services to build data marts. While users can freely combine dimensions and metrics based on their analytical needs, the processing flow from data sources to data marts remains coupled, resulting in a lengthy and time-consuming process that can only be customized by developers. Consequently, common data processing methods are not widely applicable and time-consuming, impacting the overall performance of the data processing chain. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method and device, and a storage medium, which effectively improve applicability, shorten processing time, and enhance the overall performance of the data processing link.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a data processing method, the data processing method comprising:

[0006] Construct an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer;

[0007] Performing operator link configuration based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer;

[0008] Based on PSO, construct the data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer;

[0009] The source data is processed based on the data processing link to obtain a data mart of the data warehouse.

[0010] In a second aspect, an embodiment of the present application provides a data processing device, the data processing device comprising: a construction unit, a configuration unit, and a processing unit;

[0011] The construction unit is used to construct an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer;

[0012] The configuration unit is configured to perform operator link configuration based on the operator set corresponding to each data layer to obtain a data processing link corresponding to each data layer;

[0013] The construction unit is further configured to construct the data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer based on PSO;

[0014] The processing unit is configured to process the source data based on the data processing link to obtain the data mart of the data warehouse.

[0015] In a third aspect, an embodiment of the present application provides a data processing device, comprising: a processor and a memory; wherein:

[0016] The memory is used to store a computer program that can be run on the processor;

[0017] The processor is configured to execute the data processing method described above when running the computer program.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program code stored thereon. When the computer program code is executed by a computer, the data processing method as described above is implemented.

[0019] An embodiment of the present application provides a data processing method and device, and a storage medium. The method includes: a data processing device constructing an operator set for each data layer in a data warehouse to obtain an operator set corresponding to each data layer; configuring an operator link based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer; based on a particle swarm algorithm, constructing a data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer; and performing data processing on source data based on the data processing link to obtain a data mart of the data warehouse. It can be seen that in the embodiment of the present application, the data processing device first constructs an operator set for each data layer in the data warehouse, thereby obtaining an operator set corresponding to each data layer; then, based on the operator set corresponding to each data layer, an operator link is configured to obtain a data processing sub-link corresponding to each data layer. The data processing device can select operators from the operator set according to the needs of the user, and then can orchestrate the operator links of each layer, arbitrarily combine the data processing process, and improve the applicability of the data processing method; finally, based on the particle swarm algorithm, the data processing link corresponding to the data warehouse is constructed according to the data processing sub-link corresponding to each data layer, and the source data is processed based on the data processing link to obtain the data mart of the data warehouse; the data processing sub-links of each layer in the above-mentioned data warehouse are connected in series to obtain a data processing link template, and the dynamic data processing link orchestration template is executed according to the particle swarm algorithm to obtain the optimal data processing link corresponding to the data warehouse. After the nodes in the entire data processing process are optimized and accelerated according to the particle swarm algorithm, the data processing time is shortened, the data processing efficiency is improved, and the overall performance of the data processing link is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the data processing method proposed in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a method for creating a data source access layer operator set proposed in an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a method for creating an operator set for a source data layer proposed in an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of a method for creating an operator set for a source data layer proposed in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a method for creating a first operator set corresponding to the detailed data layer proposed in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of a method for creating a second operator set corresponding to the detailed data layer proposed in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a method for creating a third operator set corresponding to the data application layer proposed in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of a method for preparing a fourth operator set corresponding to the data application layer proposed in an embodiment of the present application;

[0028] Figure 9 Schematic diagram of the data processing sub-link corresponding to the data source access layer proposed in the embodiment of the present application;

[0029] Figure 10 This is a schematic diagram of the data processing sub-link corresponding to the source data layer proposed in an embodiment of the present application;

[0030] Figure 11 A schematic diagram of the data processing sub-link corresponding to the detailed data layer proposed in an embodiment of the present application;

[0031] Figure 12 Schematic diagram of the data processing sub-link corresponding to the data application layer proposed in the embodiment of this application;

[0032] Figure 13 A schematic diagram of the initial data processing link corresponding to the data warehouse proposed in the embodiment of this application;

[0033] Figure 14 A schematic diagram of the initial data processing link executed by the PSO proposed in an embodiment of the present application;

[0034] Figure 15 This is a schematic diagram of the operator market planning proposed in the embodiment of this application;

[0035] Figure 16 This is a schematic diagram of the overall logic block diagram of the data processing link orchestration solution proposed in the embodiment of the present application;

[0036] Figure 17 Schematic diagram of the operator lifecycle and data processing link orchestration design proposed in this application embodiment;

[0037] Figure 18 Schematic diagram of the structure of the data processing device proposed in this embodiment of the application Figure 1 ;

[0038] Figure 19 Schematic diagram of the structure of the data processing device proposed in this embodiment of the application Figure 2 ; DETAILED DESCRIPTION

[0039] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It should be understood that the specific embodiments described herein are only used to explain the related applications and are not intended to limit the applications. It should also be noted that for ease of description, only the portions relevant to the related applications are shown in the drawings.

[0040] In the big data world, data warehouses are crucial as a common data foundation. When building a data warehouse, the typical approach is to first layer the data and then process the data layer by layer. This approach typically involves establishing a data layer to connect to various data sources, including logs, files, databases, and Kafka. Next, a metamodel is established based on specific business needs. Source data from the data source is extracted, converted, and loaded to map it to metamodel data. Finally, a Cartesian product aggregation operation is performed on the data to create a data cube. This successfully establishes a dataset based on detailed and aggregated data, providing a strong foundation for data retrieval from external systems.

[0041] Currently, traditional data warehouses are designed to build data marts for OLAP services. Users can arbitrarily combine dimensions and metrics based on page analysis requirements. However, the processing flow from data sources to data marts still uses a coupled approach. Starting with data source selection, each operation needs to be customized and developed into the process. Operations are then linked together according to business data processing logic to ultimately achieve the connection of the entire process. When data sources, businesses, and algorithms differ, the process can only be sorted out and developed from beginning to end. The entire process is time-consuming and can only be customized by developers. Therefore, common data processing methods are not very applicable and are time-consuming, which in turn affects the overall performance of the data processing chain.

[0042] In order to solve the problems existing in the data processing methods in the prior art, the embodiments of the present application provide a data processing method and device, and a storage medium. The method includes: constructing an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer; configuring an operator link based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer; constructing a data processing link corresponding to the data warehouse based on the data processing sub-link corresponding to each data layer based on a particle swarm algorithm; processing the source data based on the data processing link to obtain a data mart of the data warehouse. In the present application, the data processing device can select operators from the operator set as needed based on the hierarchical design of the data warehouse according to the needs of the user, and then can orchestrate the operator links of each layer and arbitrarily combine the data processing flow, thereby improving the applicability of the data processing method; executing a dynamic data processing link orchestration template based on the particle swarm algorithm, so that the optimal node can be selected, shortening the data processing time and improving the data processing efficiency; thereby improving the overall performance of the data processing link.

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0044] Example 1

[0045] The present application provides a data processing method. Figure 1 Schematic diagram of the data processing method proposed in this application embodiment Figure 1 ,like Figure 1 As shown, the method for performing data processing by a data processing device may include the following steps:

[0046] Step 101: Construct an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer.

[0047] In an embodiment of the present application, the data processing device may first construct an operator set for each data layer in the data warehouse, thereby obtaining an operator set corresponding to each data layer, wherein one data layer corresponds to one operator set.

[0048] It should be noted that in the embodiments of the present application, the data warehouse is a strategic set that provides all types of data support for decision-making processes at all levels of the enterprise. It is created for analytical reporting and decision support purposes. It provides guidance for business process improvement, monitoring time, cost, quality and control for enterprises that need business intelligence. The input of the data warehouse is a variety of data sources, and the final output is used for enterprise data analysis, data mining, data reporting and other directions. The data cleaning process can be simplified through data hierarchical management, and the original one-step work is divided into multiple steps to complete, which is equivalent to breaking a complex task into multiple simple tasks. The processing logic of each layer is relatively simple and easy to understand, so it is easier for us to ensure the correctness of each step. When data errors occur, we often only need to adjust a certain step locally.

[0049] It should be noted that in the embodiments of the present application, the data warehouse includes multiple data layers, which may specifically include a data source access layer, a source data layer, a detailed data layer, and a data application layer. Among them, the data source access layer abstractly integrates the commonly used data connection capabilities in the industry into these four major operator sets through different data source connection operator sets. The data sources are divided into four types: relational databases, non-relational databases, message queues, and file servers. The source data layer is used to store data from a series of data sources. The detailed data layer extracts, transforms, and loads (Extract, Transform, Load, ETL) the data from the source data layer, and should generate consistent, accurate, and clean data. The data application layer is data constructed to meet specific analysis needs. It is mainly a data mart established by various business parties or departments based on the detailed data layer. The application layer only contains the data of the detailed layer that the department or business aspect is concerned about. It is star-shaped or snowflake-structured data. In terms of data granularity, it is highly aggregated data.

[0050] It should be noted that in the embodiments of this application, an operator refers to a mapping from one function space to another function space (or itself). In the field of data processing, computers use various operators to change the starting state of a data set during data processing, passing through various intermediate states and gradually reaching the target state, thereby completing data processing. The various operations in the processing flow are called operators. The operators in the embodiments of this application can include inputs, outputs, and function bodies. The function body corresponds to various operations, and the inputs and outputs can be various types of data.

[0051] Furthermore, in an embodiment of the present application, for the data source access layer, when the data processing device constructs an operator set for each data layer in the data warehouse and obtains the operator set corresponding to each data layer, it can first configure the initial data connection operator in the connection parameter package to obtain the configured data connection operator; then the configured data connection operator can be added to the data connection operator set; finally, the data connection operator set can be configured to generate the operator set corresponding to the data source access layer.

[0052] It should be noted that, in the embodiments of this application, Figure 2 Construct a schematic diagram for the operator set of the data source access layer, such as Figure 2As shown, when creating an operator set for the data source access layer, the data processing device can first select and design a data connection parameter package and configure the initial data connection operator in the connection parameter package. This configuration process can include configuring connection parameters and basic operator attributes (name, scope of application, and usage permissions) to obtain the configured data connection operator. The configured data connection operator can then be added to the data connection operator set. The data connection operator set can then be configured for online / offline use, and the operator set corresponding to the data source access layer can be retrieved from the database.

[0053] Furthermore, in an embodiment of the present application, for the source data layer, when the data processing device constructs an operator set for each data layer in the data warehouse and obtains the operator set corresponding to each data layer, it can first configure the initial design model mapping operator in the design model mapping operator package to obtain the configured design model mapping operator; add the configured design model mapping operator to the design model mapping operator set; configure the design model mapping operator set to generate the operator set corresponding to the source data layer.

[0054] It should be noted that, in the embodiments of this application, Figure 3 Construct a schematic diagram for the operator set of the source data layer, such as Figure 3 As shown, when creating an operator set for the source data layer, the data processing device can first select a design model mapping operator package and configure the initial design model mapping operator in the design model mapping operator package. This configuration process can include configuring mapping attributes and basic operator attributes (name, scope of application, and usage permissions) to obtain the configured design model mapping operator. The configured design model mapping operator can then be added to the design model mapping operator set. The design model mapping operator set can then be configured online or offline, and the operator set corresponding to the source data layer can be obtained from the library.

[0055] Furthermore, in an embodiment of the present application, for the source data layer, when the data processing device constructs an operator set for each data layer in the data warehouse and obtains the operator set corresponding to each data layer, it can also configure the initial batch mapping operator in the batch mapping operator package to obtain the configured batch mapping operator; and add the configured batch mapping operator to the design model mapping operator set;

[0056] The design model mapping operator set is configured and processed to generate the operator set corresponding to the source data layer.

[0057] It should be noted that, in the embodiments of this application, Figure 4 Construct a schematic diagram for the operator set of the source data layer, such as Figure 4As shown, when creating an operator set for the source data layer, the data processing device can also select a batch mapping operator package and configure the initial batch mapping operators in the batch mapping operator package. This configuration process can include configuring connection parameters and basic operator properties (name, scope of application, and usage permissions) to obtain the configured batch mapping operator. The configured batch mapping operator can then be added to the design model mapping operator set. The design model mapping operator set can then be configured for online / offline use, and the operator set corresponding to the source data layer can then be retrieved from the warehouse.

[0058] Furthermore, in an embodiment of the present application, for the detailed data layer, the constructed operator set includes a first operator set and a second operator set. The operator set of the detailed data layer in the data warehouse is constructed according to two different operator set construction methods to obtain the first operator set and the second operator set corresponding to the detailed data layer, wherein the first operator set can be a design model mapping operator set, and the second operator set can be an ETL mapping operator set.

[0059] Furthermore, in an embodiment of the present application, for the first operator set corresponding to the detailed data layer, when the data processing device constructs the operator set for each data layer in the data warehouse and obtains the operator set corresponding to each data layer, it can first configure the initial design model mapping operator in the design model mapping operator package to obtain the configured design model mapping operator; add the configured design model mapping operator to the design model mapping operator set; configure the design model mapping operator set to generate the first operator set corresponding to the detailed data layer.

[0060] It should be noted that, in the embodiments of this application, Figure 5 Construct a schematic diagram for the operator set of the detailed data layer, such as Figure 5 As shown, when creating an operator set for the detailed data layer, the data processing device can select a design model mapping operator package and configure the initial design model mapping operator in the package. This configuration process can include configuring mapping attributes and basic operator attributes (name, scope of application, and usage permissions) to obtain the configured design model mapping operator. The configured design model mapping operator can then be added to the design model mapping operator set. The design model mapping operator set can then be configured for online / offline use, and the first operator set corresponding to the detailed data layer can be retrieved from the database.

[0061] Furthermore, in an embodiment of the present application, for the second operator set corresponding to the detailed data layer, the data processing device can also configure the initial ETL operator in the extraction, transformation and loading ETL mapping operator package when constructing the operator set for each data layer in the data warehouse to obtain the configured ETL operator; then, the configured ETL operator can be added to the ETL mapping operator set; the ETL mapping operator set can be configured to generate the second operator set corresponding to the detailed data layer.

[0062] It should be noted that, in the embodiments of this application, Figure 6 Construct a schematic diagram for the operator set of the detailed data layer, such as Figure 6 As shown, when the data processing device constructs the operator set for the detailed data layer, the data processing device selects the ETL mapping operator package and performs configuration processing on the initial ETL operator in the ETL mapping operator package. The specific content of the configuration processing includes configuring the basic attributes of the operator, where the basic attributes may include but are not limited to name, scope of application, usage rights, etc., to obtain the configured ETL operator; then, the configured ETL operator can be added to the ETL mapping operator set; the ETL mapping operator set is configured for online / offline processing, and the second operator set corresponding to the detailed data layer is obtained from the warehouse.

[0063] Furthermore, in an embodiment of the present application, for the data application layer, the operator set constructed by the data processing device includes a third operator set and a fourth operator set. The operator set of the data application layer in the data warehouse is constructed according to two different operator set construction methods to obtain the third operator set and fourth operator set corresponding to the data application layer, wherein the third operator set can be a derived dimension operator set, and the fourth operator set can be a metric calculation rule operator set.

[0064] Furthermore, in an embodiment of the present application, for the third operator set corresponding to the data application layer, when the data processing device constructs the operator set for each data layer in the data warehouse and obtains the operator set corresponding to each data layer, it can first set the initial derived dimension operator in the derived dimension operator package as a forced dimension, and / or hierarchical dimension, and / or joint dimension to obtain the set derived dimension operator; configure the set derived dimension operator to obtain the configured derived dimension operator; then, the configured derived dimension operator can be added to the derived dimension operator set; the derived dimension operator set can be configured to generate the third operator set corresponding to the data application layer.

[0065] It should be noted that, in the embodiments of this application, Figure 7 Construct a schematic diagram for the operator set of the data application layer, such as Figure 7As shown, when the data processing device creates an operator set for the data application layer, the data processing device can select a derived dimension operator package and set the initial derived dimension operator in the derived dimension operator package as a mandatory dimension, hierarchical dimension, or joint dimension. The configuration process may include configuring basic operator attributes, where the basic attributes may include but are not limited to name, scope of application, and usage permissions. The configured derived dimension operator is obtained; then, the configured derived dimension operator can be added to the derived dimension operator set; the derived dimension operator set is configured for online / offline use, and the third operator set corresponding to the data application layer is obtained from the library.

[0066] Furthermore, in an embodiment of the present application, for the fourth operator set corresponding to the data application layer, when the data processing device constructs an operator set for each data layer in the data warehouse and obtains the operator set corresponding to each data layer, it can first perform aggregation calculation on the initial metric calculation rule operator package operators in the metric calculation rule operator package to obtain the calculated operators; configure the calculated operators to obtain the configured operators; then, the configured operators can be added to the metric calculation rule operator set; the metric calculation rule operator set can be configured to generate the fourth operator set corresponding to the data application layer.

[0067] It should be noted that, in the embodiments of this application, Figure 8 Construct a schematic diagram for the operator set of the data application layer, such as Figure 8 As shown, when creating the data application layer operator set: the data processing device selects the metric calculation rule operator package, and can first perform aggregation calculation on the initial metric calculation rule operator package operators in the metric calculation rule operator package, wherein the aggregation calculation includes but is not limited to using functions such as SUM / MIN / MAX / COUNT to perform calculations, to obtain the calculated operators; the basic attributes of the calculated operators are configured, wherein the basic attributes may include but are not limited to name, scope of application, usage permissions, etc. to obtain the configured operators; then, the configured operators can be added to the metric calculation rule operator set; the data processing device performs online / offline configuration processing on the metric calculation rule operator set, and obtains the fourth operator set corresponding to the data application layer from the warehouse.

[0068] It should be noted that in an embodiment of the present application, the data processing device performs forced dimension / hierarchical dimension / joint dimension settings on the initial derived dimension operator in the derived dimension operator package, and the forced dimension / hierarchical dimension / joint dimension settings can be any combination or any one of them. Derived dimensions are used to exclude non-primary key dimensions on the dimension table within the valid dimension and replace them with the primary key of the dimension table (that is, the corresponding foreign key on the fact table). For example: fact table A and dimension table B have three dimensions A, B, and E, totaling 7 cuboids; if A and E have some connection: A can find D in dimension table B, and D can find E, then A can be used instead of E to reduce the number of cuboids.

[0069] Step 102: Perform operator link configuration based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer.

[0070] In an embodiment of the present application, the data processing device may configure an operator link based on an operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer, wherein one data layer corresponds to one data processing sub-link.

[0071] Furthermore, in an embodiment of the present application, for the data source access layer, when the data processing device performs operator link configuration based on the operator set corresponding to each data layer and obtains the data processing sub-link corresponding to each data layer, it can first obtain the data connection operator based on the data connection operator set; configure the data source attributes for the data connection operator to obtain the configured data source; perform connectivity testing based on the configured data source to generate the data processing sub-link corresponding to the data source access layer.

[0072] It should be noted that, in the embodiments of this application, Figure 9 Construct a schematic diagram for the data processing sub-link corresponding to the data source access layer, such as Figure 9 As shown, the data processing device can first obtain a data connection operator based on a data connection operator set; configure the data source attributes for the data connection operator to obtain the configured data source; perform a connectivity test based on the configured data source, and the connectivity test is used to test whether the database is successfully connected and whether the parameters in the basic attributes of the operator (name, scope of application, and usage permissions) are consistent, and then generate a data processing sub-link corresponding to the data source access layer.

[0073] Furthermore, in an embodiment of the present application, an operator link is configured for the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer. The data processing device can first obtain a model mapping operator based on the design model mapping operator set; construct a data model mapping relationship for the model mapping operator; configure a storage address based on the configured data model mapping relationship, and generate a data processing sub-link corresponding to the source data layer.

[0074] It should be noted that, in the embodiments of this application, Figure 10 Construct a schematic diagram for the data processing sub-link corresponding to the source data layer, such as Figure 10 As shown, the data processing device can first obtain the model mapping operator based on the design model mapping operator set; construct a data model mapping relationship for the model mapping operator; configure the storage address based on the configured data model mapping relationship, and generate a data processing sub-link corresponding to the source data layer.

[0075] Furthermore, in an embodiment of the present application, an operator link is configured for the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer. The data processing device can first add a data wide table model; select the source layer-related table based on the data wide table model; select the design model mapping operator from the first operator set based on the source layer-related table; select the ETL mapping operator from the second operator set based on the design model mapping operator; perform model mapping and storage address configuration processing based on the ETL mapping operator to generate a data processing sub-link corresponding to the detailed data layer.

[0076] It should be noted that, in the embodiments of this application, Figure 11 Construct a schematic diagram for the data processing sub-link corresponding to the detailed data layer, such as Figure 11 As shown, the data processing device can first add a data wide table model; based on the data wide table model, select the source layer involved table; based on the source layer involved table, the data processing device can select corresponding tables from the source layer involved table according to user needs and specific business processing content, such as: user table, permission table, role table, etc., and select the design model mapping operator from the first operator set (design model mapping operator set); based on the design model mapping operator, select the ETL mapping operator from the second operator set (ETL mapping operator set); based on the ETL mapping operator, perform model mapping and storage address configuration processing to generate a data processing sub-link corresponding to the detailed data layer.

[0077] Furthermore, in an embodiment of the present application, an operator link is configured for the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer. The data processing device can select a derived dimension operator based on the third operator set; configure a derived dimension rule for the derived dimension operator to obtain the configured derived dimension; based on the configured derived dimension, a metric calculation rule operator can then be selected from the fourth operator set; and a data processing sub-link corresponding to the data application layer is generated based on the metric calculation rule operator.

[0078] It should be noted that, in the embodiments of this application, Figure 12 Construct a schematic diagram for the data processing sub-link corresponding to the data application layer, such as Figure 12As shown, the data processing device can first select a derived dimension operator based on the third operator set (derived dimension operator set); configure a derived dimension rule for the derived dimension operator to obtain the configured derived dimension; based on the configured derived dimension, select a metric calculation rule operator based on the fourth operator set (metric calculation rule operator set); and generate a data processing sub-link corresponding to the data application layer based on the metric calculation rule operator.

[0079] Step 103: Based on the particle swarm algorithm, the data processing device can construct a data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer.

[0080] In an embodiment of the present application, after the data processing device configures the operator link based on the operator set corresponding to each data layer and obtains the data processing sub-link corresponding to each data layer, it can continue to build the data processing link corresponding to the data warehouse based on the data processing sub-link corresponding to each data layer based on the particle swarm algorithm.

[0081] Furthermore, in an embodiment of the present application, when the data processing device constructs the data processing link corresponding to the data warehouse based on the PSO and the data processing sub-link corresponding to each data layer, it can first connect the data processing sub-link of the data source access layer, the data processing sub-link of the source data layer, the data processing sub-link of the detail data layer, and the data processing sub-link of the data application layer in series in sequence to form the initial data processing link corresponding to the data warehouse; and then perform configuration processing of the initial data processing link according to the PSO to obtain the data processing link corresponding to the data warehouse.

[0082] It is understood that in the embodiments of the present application, Figure 13 This is a schematic diagram of the construction method of the initial data processing link corresponding to the data warehouse, such as Figure 13 As shown, the data processing device can sequentially connect the data processing sub-links of the data source access layer, the data processing sub-links of the source data layer, the data processing sub-links of the detail data layer, and the data processing sub-links of the data application layer into the initial data processing link corresponding to the data warehouse; then perform the configuration processing of the initial data processing link according to PSO to obtain the data processing link corresponding to the data warehouse.

[0083] Furthermore, in an embodiment of the present application, the data processing device can perform configuration processing of the initial data processing link according to PSO, that is, based on the particle swarm chasing the local optimal solution and the global optimal solution to obtain the optimal data processing link, thereby accelerating the data processing efficiency of the data processing link

[0084] It should be noted that, in the embodiments of this application, Figure 14 The specific process diagram of the initial data processing link for PSO is as follows: Figure 14 As shown, the data processing device can first initialize the position and velocity of each particle, each operator calculation node is a particle, and then evaluate the fitness of each particle according to the fitness function. Secondly, the individual extreme value of each particle is obtained, that is, the best position that particle i has ever experienced: pbesti=( ), and the global extreme value of the entire group: gbest=( ), then compare the current fitness value of each particle with the corresponding value of its individual historical best position (pbest). If the current fitness value is higher, the historical best position pbest will be updated with the current position. Similarly, compare the current fitness value of each particle with the corresponding value of the global best position (gbest). If the current fitness value is higher, the historical global best position gbest will be updated with the current position. Update and optimize the speed and position of each particle according to formulas (1) and (2). If the end condition is not met, return to step 2. Usually, the algorithm stops when the maximum number of iterations is reached or the increment of the best fitness value is less than a given threshold. In other words, PSO is initialized as a group of random particles (random solution). Then the optimal solution is found through iteration. In each iteration, the particle updates itself by tracking two "extreme values" (pbest, gbest). After finding these two optimal values, the particle updates its speed and position according to the following formula.

[0085] (1)

[0086] (2)

[0087] Formula (1) is the formula for updating the d-dimensional velocity of particle i; Formula (2) is the formula for updating the d-dimensional position of particle i, where the variables have the following meanings: is the d-th dimension component of the velocity vector of particle i at the k-th iteration, is the d-th dimension component of the position vector of particle i at the k-th iteration, is the particle acceleration constant, The two random functions have a value range of [0, 1] to increase the randomness of the search. w is the inertia weight, a non-negative number that adjusts the search range. A larger w value indicates a stronger global optimization capability and a weaker local optimization capability. Conversely, a smaller w value indicates a stronger local optimization capability.

[0088] It should be noted that in the embodiment of the present application, each operator computing node is a particle. The position and speed of each node are determined based on the node position and processing performance. A fitness value is assigned to each operator node based on the machine load, network conditions, etc. Then, according to equations (1) and (2), the speed and position of each operator node are updated and optimized, and finally the optimal solution that meets the termination conditions is obtained. In this way, the optimal data processing link corresponding to the above-mentioned data warehouse is obtained. After the nodes in the entire data processing flow are optimized and accelerated according to the particle swarm algorithm, the data processing time is shortened and the overall performance of the data processing link is improved.

[0089] Step 104: Process the source data based on the above data processing link to obtain a data mart of the data warehouse.

[0090] In an embodiment of the present application, after constructing the data processing link corresponding to the data warehouse, the data processing device may continue to process the source data based on the data processing link, thereby obtaining a data mart of the data warehouse.

[0091] That is to say, based on the above data processing link, the source data is processed layer by layer to obtain a data processing method with flexible process orchestration from data source to data mart.

[0092] In an embodiment of the present application, the data processing device can select operators from the operator set as needed based on user needs and under the premise of a hierarchical design of the data warehouse, and then can orchestrate the operator links of each layer and arbitrarily combine the data processing flow, thereby improving the applicability of the data processing method; according to the particle swarm algorithm, a dynamic data processing link orchestration template is executed, so that the optimal node can be selected, the data processing time is shortened, and the data processing efficiency is improved; thereby, the overall performance of the data processing link is improved.

[0093] An embodiment of the present application provides a data processing method, in which a data processing device constructs an operator set for each data layer in a data warehouse to obtain an operator set corresponding to each data layer; configures an operator link based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer; based on a particle swarm algorithm, constructs a data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer; and processes source data based on the data processing link to obtain a data mart of the data warehouse. In an embodiment of the present application, the data processing device can select operators from the operator set as needed according to user needs, and then can orchestrate the operator links of each layer and arbitrarily combine data processing processes, thereby improving the applicability of the data processing method; the data processing device can first select the required operator model in the operator market planning and make a simple configuration according to user needs, and then perform calculation orchestration of the selected operator model in each layer, and finally save the configuration template of the entire data processing operator link, and execute the entire data processing operator link template according to the particle swarm algorithm, thereby achieving the expected codeless development effect. At the same time, based on the particle swarm algorithm, the optimal node of the data processing link can be selected, which shortens the data processing time and improves the data processing efficiency; thereby improving the overall performance of the data processing link.

[0094] Example 2

[0095] Based on the above embodiments, the data processing method proposed in the embodiments of the present application can first construct operator sets for the data source access layer, source data layer, detailed data layer, and data application layer in the data warehouse. Among them, when the data processing device constructs the data source access layer operator set, it is necessary to use the design data connection parameter package; when the data processing device constructs the source data layer operator set, it is necessary to use the design data connection parameter package and the batch mapping operator package; when the data processing device constructs the detailed data layer operator set, it is necessary to use the design model mapping operator package and the ETL mapping operator package; when the data processing device constructs the data application layer operator set, it is necessary to use the derived dimension operator package and the metric calculation rule operator package.

[0096] Furthermore, it should be noted that when the data processing device constructs the data source access layer operator set, the selected connection integration parameter package comes from the data source access layer in the operator mart planning; when the data processing device constructs the source data layer operator set, the selected design data connection parameter package and batch mapping operator package come from the source data layer in the operator mart planning; when the data processing device constructs the detailed data layer operator set, the selected design model mapping operator package and ETL mapping operator package come from the detailed data layer in the operator mart planning; when the data processing device constructs the data application layer operator set, the selected derived dimension operator package and metric calculation rule operator package come from the data application layer in the operator mart planning.

[0097] It should be noted that in an embodiment of the present application, when the data processing device constructs the operator set according to the data source access layer, source data layer, detail data layer, and data application layer respectively, the data packets selected when constructing the operator set at each layer can come from the operator market planning.

[0098] Further, if Figure 15 As shown in the diagram, the operator marketplace plan includes four major data warehouse processing flows, each with a corresponding set of operators for subsequent data processing. The data source access layer is divided into four connection operator sets based on the data source: relational databases, non-relational databases, message queues, and file servers. Commonly used data connection capabilities in the industry are abstracted and integrated into these four major subsets, making them easier to manage and use.

[0099] Among them, the data source access layer - relational database connection operator: the data processing device encapsulates the common relational database connection configuration to facilitate the connection of relational databases; the data source access layer - non-relational database connection operator: the data processing device encapsulates the common non-relational database connection configuration to facilitate the connection of non-relational databases; the data source access layer - message queue connection operator: the data processing device encapsulates the common message queue connection configuration to facilitate the connection of message queues; the data source access layer - file service connection operator: the data processing device encapsulates the common file service connection configuration to facilitate the connection of file services. Thus, the data processing device configures the connections of different data sources to facilitate the connection of different data sources, that is, to facilitate the connection of relational databases, non-relational databases, message queues, and file servers; the data processing device integrates the connection parameters of different data sources of relational databases, non-relational databases, message queues, and file servers (that is, designs a data connection integration parameter package) to facilitate the data processing device to select corresponding connection parameters for connection when connecting different data sources.

[0100] Among them, the source data layer - extraction service operator: the data processing device encapsulates the data access logic of heterogeneous data sources to facilitate the docking of data source models; the source data layer - conversion service operator: encapsulates the mapping logic of heterogeneous data sources to source layer data sources, and unifies the source data standards; the source data layer - loading service operator: encapsulates the execution data flow operation of heterogeneous data sources to source layer models; the source data layer - storage service operator: encapsulates the storage path of source layer data and the automatic expansion capability of the storage layer. The data processing device encapsulates the data access logic of heterogeneous data sources to facilitate the docking of data source models, and then can encapsulate the mapping logic of heterogeneous data sources to source layer data sources and unify the source data standards. Secondly, the data processing device can select a suitable model mapping operator package. In an embodiment of the present application, the data processing device can choose to design a model mapping operator package or a batch model mapping operator package.

[0101] Among them, the detailed data layer - extraction service operator: the data processing device encapsulates the source layer data access logic to facilitate the docking of the source layer data model; the detailed data layer - conversion service operator: encapsulates the mapping logic of the source layer data to the detailed layer data, and unifies the data standard of the detailed data wide table model; the detailed data layer - loading service operator: encapsulates the execution data flow operation of the source layer data model to the detailed wide table data model; the detailed data layer - storage service operator: encapsulates the storage path of the detailed layer data and the automatic expansion capability of the storage layer; among them, the data processing device encapsulates the mapping logic of the source layer data to the detailed layer data, and unifies the data standard of the detailed data wide table model. The data processing device can choose to design a model mapping operator package, and load the operator to encapsulate the execution data flow operation of the source layer data model to the detailed wide table data model, thereby generating an ETL mapping operator package, so that the data processing device can select the ETL mapping operator package.

[0102] Among them, the data application layer - derived dimension configuration service operator: the data processing device encapsulates the mandatory dimension, hierarchical dimension, and joint dimension configuration of the data cube, and directly outputs the corresponding cuboid group through this operator. The data processing device encapsulates the mandatory dimension, hierarchical dimension, and joint dimension configuration of the data cube, so that the derived dimension operator package can be selected; the data service layer - calculation rule configuration service operator: encapsulates the calculation rules of aggregation functions such as sum / minimum / maximum / count (SUM / MIN / MAX / COUNT) to facilitate data aggregation calculation. The data processing device can encapsulate the calculation rules of aggregation functions, so that the metric calculation rule operator package can be selected, and the initial metric calculation rule operator package in the metric calculation rule operator package can be aggregated to obtain the calculated operator; the data service layer - indicator definition configuration service operator: freely input indicator calculation formulas, encapsulate calculation processing logic, and output corresponding indicator items; the data service layer - expansion depth configuration service operator: encapsulates expansion depth configuration logic, reduces the number of expansions of the input data cube, improves expansion latency, and reduces memory and hard disk overhead;

[0103] For example, the pre-calculated result of each dimension combination is called Cuboid. Assuming there are 4 dimensions, there will be = 16 Cuboids need to be calculated.

[0104] Furthermore, in the embodiments of the present application, Figure 16 This is the overall logic block diagram of the data processing link arrangement solution in the embodiment of this application, such as Figure 16As shown, the overall design logic of this data processing link orchestration solution can include five major components: data source access, data source layer creation, data detail layer creation, data application layer creation, and operational state. Data processing devices can introduce the concept of design state (similar to a draft) into the first four layers, subdividing the data processing process into two, same-level operations: operator set creation and data processing link orchestration. The operator marketplace is solely responsible for operator lifecycle management, including creation, creation, launch / delisting, updates, and destruction. Link orchestration is solely responsible for processing the data processing flow at the corresponding data layer, utilizing operators from the operator marketplace. The two complement each other to complete the overall data processing link orchestration.

[0105] It should be noted that data processing devices can introduce the concept of design state (similar to a draft) into the first four layers, subdividing the data processing flow into two identical operations: operator creation and data processing link orchestration. Specifically, the design state of the data source access layer includes operator set creation and link orchestration in the operator marketplace.

[0106] Furthermore, the operator set creation method at the data source access layer includes: processing new operators, adding the processed new operators to an operator group by a data processing device, and configuring the operator group for online and offline operations. The link orchestration method at the data source access layer includes: selecting operators based on requirements, customizing parameters based on the selected operators, and then adding the operators with customized parameters to the data source access processing flow, completing the overall flow creation.

[0107] It should be noted that data processing devices can introduce the concept of design state (similar to a draft) in the first four layers, subdividing the data processing flow into two identical operations: operator creation and data processing link orchestration. Specifically, the design state of the source data layer includes operator set creation and link orchestration in the operator marketplace.

[0108] Furthermore, the operator set creation method for the source data layer includes: processing new operators, where the data processing device can add the processed new operators to an operator group, and then configure the operator group for online and offline operations. The link orchestration method for the source data layer includes: selecting operators based on requirements, customizing parameters based on the selected operators, and then adding the operators with customized parameters to the source data layer processing flow, finally completing the overall flow creation.

[0109] It should be noted that data processing devices can introduce the concept of design state (similar to a draft) in the first four layers, subdividing the data processing flow into two operations at the same level: operator creation and data processing link orchestration. Specifically, the design state of the detailed data layer includes operator set creation and link orchestration in the operator market.

[0110] Furthermore, the operator set creation method for the detailed data layer includes: processing new operators, adding the processed new operators to an operator group by a data processing device, and configuring the operator group for online and offline operations. The link orchestration method for the detailed data layer includes: selecting operators based on requirements, customizing parameters based on the selected operators, and then adding the operators with customized parameters to the detailed data layer processing flow, ultimately completing the overall flow creation.

[0111] It should be noted that data processing devices can introduce the concept of design state (similar to a draft) into the first four layers, subdividing the data processing flow into two identical operations: operator creation and data processing link orchestration. Specifically, the design state of the data application layer includes operator set creation and link orchestration in the operator marketplace.

[0112] Furthermore, the operator set creation method for the data application layer includes: processing new operators, adding the processed new operators to an operator group by a data processing device, and configuring the operator group for online and offline operations. The link orchestration method for the data application layer includes: selecting operators based on requirements, customizing parameters based on the selected operators, and then adding the operators with customized parameters to the data application layer processing flow, ultimately completing the overall flow creation.

[0113] It should be noted that the last part is the running state. The data processing device arranges the links of the first four layers in series into the data processing link corresponding to the data warehouse, and uses the particle swarm algorithm to execute the above data processing link to improve the overall performance of the data processing link.

[0114] An embodiment of the present application provides a data processing method, which includes: a data processing device constructing an operator set for each data layer in a data warehouse to obtain an operator set corresponding to each data layer; configuring an operator link based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer; based on a particle swarm algorithm, constructing a data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer; and performing data processing on source data based on the data processing link to obtain a data mart of the data warehouse. In this application, the data processing device can select operators from the operator set as needed according to the needs of the user, and then can orchestrate the operator links of each layer and arbitrarily combine the data processing processes, thereby improving the applicability of the data processing method; the data processing device can first select the required operator model in the operator market planning and make a simple configuration according to the needs of the user, and then perform calculation orchestration of the selected operator model in each layer, and finally save the configuration template of the entire data processing operator link, and execute the entire data processing operator link template according to the particle swarm algorithm, thereby achieving the expected code-free development effect. At the same time, based on the particle swarm algorithm, the optimal node of the data processing link can be selected, which shortens the data processing time and improves the data processing efficiency; thereby improving the overall performance of the data processing link.

[0115] Example 3

[0116] Based on this, the embodiment of the present application proposes an operator market solution that supports automatic orchestration of data links, that is, a data processing method with a graphical interface, componentized or modularized calculation rules, and flexible process orchestration. When data from any data source needs to be processed in a data warehouse, there is no need to understand code development. When calculations need to be performed in the source data layer, data processing layer, and data aggregation layer, first select the required operator model in the operator market and perform a simple configuration. Then, the selected operator model is used to perform calculation orchestration in each layer. Finally, the configuration template of the entire data processing operator link is saved and executed, thereby achieving the expected code-free development effect.

[0117] In the embodiments of this application, an operator mart solution that supports automatic orchestration of data links is proposed. Based on the existing hierarchical architecture of data warehouses, each type of operation is abstracted into an operator capability. These are then hashed into different operator sets based on usage function, module division, and capability level (basic / advanced). When using an operator, users can select operators from the operator set as needed, based on the hierarchical design of the data warehouse, and customize the permutation and combination of operator links. In the embodiments of this application, an operator refers to a mapping from one function space to another function space (or itself). In the field of data processing, computers use various operators to change the starting state of a dataset, passing through various intermediate states, and gradually reaching the target state, thereby completing the data processing. The various operations in the processing flow are called operators. In the embodiments of this application, operators include inputs, outputs, and function bodies. Function bodies correspond to various operations. Inputs and outputs can be various types of data. The underlying system uses particle swarms to pursue local and global optimal solutions based on user-configured links, service distribution, service usage (busyness), and computational concurrency, to determine the optimal data processing link. The overall logic diagram of the solution (data processing link orchestration solution) is as follows Figure 16 .

[0118] The overall design logic of this solution is shown in the figure. It includes five major components: data source access, data source layer production, data detail layer production, data application layer production, and runtime execution. The concept of design state (similar to a draft) is introduced in the first four layers, subdividing the data processing process into two equally layered operations: operator production (operator set) and link orchestration. The operator marketplace is solely responsible for operator lifecycle management, including creation, production, online / offline, update, and destruction. Link orchestration is solely responsible for the data processing flow at the corresponding data layer, utilizing operators from the operator marketplace. These two components complement each other to complete the design of the entire data processing process. The final component is the execution process, which utilizes a particle swarm algorithm to accelerate process orchestration.

[0119] For the operator market, such as Figure 15 As shown, the following planning was made.

[0120] The operator marketplace plan includes four major data warehouse processing flows, each with a corresponding set of operators for data processing. The data source access process is divided into four connection operator sets based on the data source: relational databases, non-relational databases, message queues, and file servers. Commonly used data connection capabilities in the industry are abstracted and integrated into these four major subsets, making them easier to manage and use.

[0121] Data source access process (data source access layer) - relational database connection operator: encapsulates common relational database connection configurations to facilitate relational database connectivity; data source access process - non-relational database connection operator: encapsulates common non-relational database connection configurations to facilitate non-relational database connectivity; data source access process - message queue connection operator: encapsulates common message queue connection configurations to facilitate message queue connectivity; data source access process - file service connection operator: encapsulates common file service connection configurations to facilitate file service connectivity;

[0122] Source Data Layer - Extraction Service Operator: Encapsulates the data access logic of heterogeneous data sources to facilitate data source model docking; Source Data Layer - Conversion Service Operator: Encapsulates the mapping logic from heterogeneous data sources to source layer data sources and unifies source data standards; Source Data Layer - Loading Service Operator: Encapsulates the execution data flow operations from heterogeneous data sources to source layer models; Source Data Layer - Storage Service Operator: Encapsulates the storage path of source layer data and the automatic expansion capability of the storage layer;

[0123] Data processing layer (detailed data layer) - extraction service operator: encapsulates the source layer data access logic to facilitate the connection with the source layer data model; data processing layer - conversion service operator: encapsulates the mapping logic from source layer data to detail layer data, unifying the data standards of the detail data wide table model; data processing layer - loading service operator: encapsulates the execution data flow operations from the source layer data model to the detail wide table data model; data processing layer - storage service operator: encapsulates the storage path of the detail layer data and the automatic expansion capability of the storage layer;

[0124] Data Service Layer (Data Application Layer) - Derived Dimension Configuration Service Operator: Encapsulates mandatory dimensions, hierarchical dimensions, and joint dimension configurations for data cubes, and directly outputs corresponding cuboid groups through this operator; Data Service Layer - Calculation Rule Configuration Service Operator: Encapsulates calculation rules for operators such as SUM / MIN / MAX / COUNT to facilitate data aggregation calculations; Data Service Layer - Indicator Definition Configuration Service Operator: Freely input indicator calculation formulas, encapsulates calculation processing logic, and outputs corresponding indicator items; Data Service Layer - Expansion Depth Configuration Service Operator: Encapsulates expansion depth configuration logic, reduces the number of expansions of the input data cube, improves expansion latency, and reduces memory and disk overhead;

[0125] Furthermore, for the operator life cycle (operator set production) and data processing link orchestration design, such as Figure 17 shown.

[0126] The ultimate goal of the overall design process is to dynamically generate operator link orchestration templates. The left side shows the operator marketplace processing process. To match the orchestration of the data processing process, the required operator atomic capabilities are encapsulated and hashed according to their scope of application and usage permissions. These capabilities are then deployed on different nodes. The same operator can be deployed on multiple nodes. The operator processing process is divided into design and operational states. This multi-state process is designed to better support the smooth expansion of operator capabilities. The design state primarily involves three parts: operator capability encapsulation, testing, and rollout. Operator capability encapsulation is essentially a code block that provides API debugging capabilities. The underlying computing foundation uses Flink operators. Once the operator in the design state passes testing, the rollout process is initiated. Only operators that have initiated the rollout process can be used in the data processing process. The operational state is essentially a set of link orchestration templates. The deployed operator capabilities are integrated into the orchestration templates based on the data processing logic for batch / stream process execution. For example, multiple operators on multiple nodes can be integrated into an orchestration template based on the data processing logic. In an embodiment of the present invention, the configuration of the dynamic operator link orchestration template may also be performed according to the particle swarm algorithm (PSO), so that the optimal node operation may be selected.

[0127] Once the data processing flow template is successfully configured, the flow can be started. The flow is executed in a cluster environment, and the operator capabilities in the flow are distributed across different machines based on machine load, network conditions, and other factors.

[0128] To improve data processing efficiency, this proposal uses an algorithm based on particle swarms to pursue local and global optimal solutions to obtain the optimal data processing link. The principle of the particle swarm algorithm (PSO) is as follows: PSO seeks the optimal solution by continuously iterating among a group of random particles (random solutions). In each iteration, particles continuously update their values ​​by tracking two extreme values ​​(individual extreme value and global extreme value), forcing themselves to continuously tend towards the optimal solution. The specific process of the algorithm is as follows: Figure 14 As shown in the figure, each operator computing node is a particle. The position and speed of each node are determined according to the node position and processing performance. A fitness value is assigned to each operator node according to the machine load, network conditions, etc. Then, the speed and position of each particle are updated and optimized according to formulas (1) and (2). After the nodes in the entire data processing flow are accelerated according to the particle swarm algorithm, the overall performance of the data processing link is improved.

[0129] An embodiment of the present application provides a data processing method, in which a data processing device constructs an operator set for each data layer in a data warehouse to obtain an operator set corresponding to each data layer; configures an operator link based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer; based on a particle swarm algorithm, constructs a data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer; and processes source data based on the data processing link to obtain a data mart of the data warehouse. In an embodiment of the present application, the data processing device can select operators from the operator set as needed according to user needs, and then can orchestrate the operator links of each layer and arbitrarily combine data processing processes, thereby improving the applicability of the data processing method; the data processing device can first select the required operator model in the operator market planning and make a simple configuration according to user needs, and then perform calculation orchestration of the selected operator model in each layer, and finally save the configuration template of the entire data processing operator link, and execute the entire data processing operator link template according to the particle swarm algorithm, thereby achieving the expected codeless development effect. At the same time, based on the particle swarm algorithm, the optimal node of the data processing link can be selected, which shortens the data processing time and improves the data processing efficiency; thereby improving the overall performance of the data processing link.

[0130] Example 4

[0131] Based on the above embodiments, the present invention provides a data processing device. Figure 18 Schematic diagram of the structure of the data processing device Figure 1 ,like Figure 18 As shown, the data processing device 10 includes: a construction unit 11, a configuration unit 12, and a processing unit 13;

[0132] The construction unit 11 is used to construct an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer;

[0133] The configuration unit 12 is configured to perform operator link configuration based on the operator set corresponding to each data layer to obtain a data processing link corresponding to each data layer;

[0134] The construction unit 11 is further configured to construct the data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer based on PSO;

[0135] The processing unit 13 is configured to process the source data based on the data processing link to obtain the data mart of the data warehouse.

[0136] Furthermore, in an embodiment of the present application, each data layer in the data warehouse includes: a data source access layer, a source data layer, a detail data layer, and a data application layer.

[0137] Furthermore, in an embodiment of the present application, the construction unit 11 is also used to configure the initial data connection operator in the connection parameter package to obtain the configured data connection operator; add the configured data connection operator to the data connection operator set; configure the data connection operator set to generate an operator set corresponding to the data source access layer.

[0138] Furthermore, in an embodiment of the present application, the construction unit 11 is also used to configure the initial design model mapping operator in the design model mapping operator package to obtain the configured design model mapping operator; add the configured design model mapping operator to the design model mapping operator set; configure the design model mapping operator set to generate an operator set corresponding to the source data layer.

[0139] Furthermore, in an embodiment of the present application, the construction unit 11 is also used to configure the initial batch mapping operator in the batch mapping operator package to obtain the configured batch mapping operator; add the configured batch mapping operator to the design model mapping operator set; configure the design model mapping operator set to generate the operator set corresponding to the source data layer.

[0140] Furthermore, in an embodiment of the present application, the operator set of the detailed data layer includes a first operator set, and the construction unit 11 is also used to configure the initial design model mapping operator in the design model mapping operator package to obtain the configured design model mapping operator; add the configured design model mapping operator to the design model mapping operator set; configure the design model mapping operator set to generate the first operator set corresponding to the detailed data layer.

[0141] Furthermore, in an embodiment of the present application, the operator set of the detailed data layer includes a second operator set, and the construction unit 11 is also used to configure the initial ETL operator in the extraction, transformation and loading ETL mapping operator package to obtain the configured ETL operator; add the configured ETL operator to the ETL mapping operator set; configure the ETL mapping operator set to generate the second operator set corresponding to the detailed data layer.

[0142] Furthermore, in an embodiment of the present application, the operator set of the data application layer includes a third operator set, and the construction unit 11 is also used to set the initial derived dimension operator in the derived dimension operator package as a forced dimension, and / or a hierarchical dimension, and / or a joint dimension to obtain the set derived dimension operator; perform configuration processing on the set derived dimension operator to obtain the configured derived dimension operator; add the configured derived dimension operator to the derived dimension operator set; perform configuration processing on the derived dimension operator set to generate the third operator set corresponding to the data application layer.

[0143] Furthermore, in an embodiment of the present application, the operator set of the data application layer includes a fourth operator set, and the construction unit 11 is also used to perform aggregation calculation on the operators of the initial metric calculation rule operator package in the metric calculation rule operator package to obtain the calculated operator; perform configuration processing on the calculated operator to obtain the configured operator; add the configured operator to the metric calculation rule operator set; perform configuration processing on the metric calculation rule operator set to generate the fourth operator set corresponding to the data application layer.

[0144] Furthermore, in an embodiment of the present application, the configuration unit 12 is also used to obtain a data connection operator based on a data connection operator set; configure the data source attributes of the data connection operator to obtain a configured data source; perform a connectivity test based on the configured data source to generate a data processing sub-link corresponding to the data source access layer.

[0145] Furthermore, in an embodiment of the present application, the configuration unit 12 is also used to obtain a model mapping operator based on a design model mapping operator set; construct a data model mapping relationship for the model mapping operator; configure a storage address based on the configured data model mapping relationship, and generate a data processing sub-link corresponding to the source data layer.

[0146] Furthermore, in an embodiment of the present application, the configuration unit 12 is also used to add a new data wide table model; based on the data wide table model, a table involved in the source layer is selected; based on the table involved in the source layer, a design model mapping operator is selected from the first operator set; based on the design model mapping operator, an ETL mapping operator is selected from the second operator set; based on the ETL mapping operator, model mapping and storage address configuration processing are performed to generate a data processing sub-link corresponding to the detailed data layer.

[0147] Furthermore, in an embodiment of the present application, the configuration unit 12 is also used to select a derived dimension operator based on the third operator set; configure a derived dimension rule for the derived dimension operator to obtain the configured derived dimension; based on the configured derived dimension, select a metric calculation rule operator from the fourth operator set; and generate a data processing sub-link corresponding to the data application layer based on the metric calculation rule operator.

[0148] Furthermore, in an embodiment of the present application, the processing unit 13 is also used to connect the data processing sub-link of the data source access layer, the data processing sub-link of the source data layer, the data processing sub-link of the detail data layer, and the data processing sub-link of the data application layer in series in sequence to form an initial data processing link corresponding to the data warehouse; and perform configuration processing of the initial data processing link according to PSO to obtain the data processing link corresponding to the data warehouse.

[0149] In the embodiments of the present application, further, Figure 19 Schematic diagram of the structure of the data processing device Figure 2 ,like Figure 19 As shown, the data processing device 10 proposed in the embodiment of the present application may also include a processor 14, a memory 15 storing executable instructions of the processor 14, and further, the data processing device 10 may also include a communication interface 16, and a bus 17 for connecting the processor 14, the memory 15 and the communication interface 16.

[0150] In the embodiment of the present application, the processor 14 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiment of the present application is not specifically limited. The data processing device 10 may also include a memory 15, which may be connected to the processor 14, wherein the memory 15 is used to store executable program code, which includes computer operating instructions. The memory 15 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0151] In the embodiment of the present application, the bus 17 is used to connect the communication interface 16, the processor 14 and the memory 15, and to facilitate mutual communication between these devices.

[0152] In the embodiment of the present application, the memory 15 is used to store instructions and data.

[0153] Furthermore, in an embodiment of the present application, the above-mentioned processor 14 is used to construct an operator set for each data layer in the data warehouse to obtain the operator set corresponding to each data layer; perform operator link configuration based on the operator set corresponding to each data layer to obtain the data processing sub-link corresponding to each data layer; based on PSO, construct the data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer; perform data processing on the source data based on the data processing link to obtain the data mart of the data warehouse.

[0154] In practical applications, the memory 15 may be a volatile memory, such as a random-access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 17.

[0155] An embodiment of the present application provides a data processing device that constructs an operator set for each data layer in a data warehouse to obtain an operator set corresponding to each data layer; configures an operator link based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer; constructs a data processing link corresponding to the data warehouse based on the data processing sub-link corresponding to each data layer based on a particle swarm algorithm; and processes source data based on the data processing link to obtain a data mart of the data warehouse. In an embodiment of the present application, the data processing device can select operators from the operator set as needed according to user needs, and then can orchestrate the operator links of each layer and arbitrarily combine data processing processes, thereby improving the applicability of the data processing method; the data processing device can first select the required operator model in the operator market planning and make a simple configuration according to user needs, and then perform calculation orchestration of the selected operator model in each layer, and finally save the configuration template of the entire data processing operator link, and execute the entire data processing operator link template according to the particle swarm algorithm, thereby achieving the expected codeless development effect. At the same time, based on the particle swarm algorithm, the optimal node of the data processing link can be selected, which shortens the data processing time and improves the data processing efficiency; thereby improving the overall performance of the data processing link.

[0156] An embodiment of the present application provides a computer-readable storage medium having a program stored thereon, which implements the data processing method described above when the program is executed by a processor.

[0157] Specifically, the program instructions corresponding to a data processing method in this embodiment may be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to a software workload prediction method in the storage medium are read or executed by an electronic device, the following steps are included:

[0158] Construct an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer;

[0159] Performing operator link configuration based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer;

[0160] Based on PSO, construct the data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer;

[0161] The source data is processed based on the data processing link to obtain a data mart of the data warehouse.

[0162] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage) containing computer-usable program code.

[0163] The present application is described with reference to the implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the flowchart. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which is implemented in the implementation flow diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process described in the flowchart. Figure 1 a process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0166] The above description is merely a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method includes: Construct an operator set for each data layer in the data warehouse to obtain the operator set corresponding to each data layer; each data layer in the data warehouse includes: a data source access layer, a source data layer, a detail data layer, and a data application layer; The step of constructing an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer includes: Selecting a design model mapping operator package from the operator market plan, configuring an initial design model mapping operator in the design model mapping operator package, and obtaining a configured design model mapping operator; Adding the configured design model mapping operator to a design model mapping operator set; Performing configuration processing on the design model mapping operator set to generate an operator set corresponding to the source data layer; or, Selecting a batch model mapping operator package from the operator market plan, configuring an initial batch mapping operator in the batch model mapping operator package, and obtaining a configured batch mapping operator; Adding the configured batch mapping operator to the design model mapping operator set; Performing configuration processing on the design model mapping operator set to generate an operator set corresponding to the source data layer; Accordingly, performing operator link configuration based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer includes: Acquire a model mapping operator based on an operator set corresponding to the source data layer; Constructing a data model mapping relationship for the model mapping operator; Configuring a storage address based on the data model mapping relationship, and generating a data processing sub-link corresponding to the source data layer; Based on the particle swarm algorithm PSO, construct the data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer; The source data is processed based on the data processing link to obtain a data mart of the data warehouse.

2. The method according to claim 1, characterized in that The step of constructing an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer includes: Pre-acquire a connection parameter package, configure the initial data connection operator in the connection parameter package, and obtain the configured data connection operator; Adding the configured data connection operator to the data connection operator set; Performing configuration processing on the data connection operator set to generate an operator set corresponding to the data source access layer; Accordingly, performing operator link configuration based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer includes: Acquire the data connection operator based on the operator set corresponding to the data source access layer; Configuring data source attributes for the data connection operator to obtain the configured data source; A connectivity test is performed based on the configured data source to generate a data processing sub-link corresponding to the data source access layer.

3. The method according to claim 1, characterized in that The step of constructing an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer includes: When the operator set of the detailed data layer includes the first operator set, selecting a design model mapping operator package from the operator market plan, configuring the initial design model mapping operator in the design model mapping operator package, and obtaining a configured design model mapping operator; adding the configured design model mapping operator to the design model mapping operator set; Performing configuration processing on the design model mapping operator set to generate the first operator set corresponding to the detailed data layer; When the operator set of the detailed data layer includes the second operator set, an extraction, transformation and loading ETL mapping operator package is selected from the operator market plan, and the initial ETL operator in the ETL mapping operator package is configured to obtain a configured ETL operator; Add the configured ETL operator to the ETL mapping operator set; Performing configuration processing on the ETL mapping operator set to generate the second operator set corresponding to the detailed data layer; Accordingly, performing operator link configuration based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer includes: Added data wide table model; Selecting tables involved in the source layer based on the data wide table model; Based on the source layer reference table, a design model mapping operator is selected from the first operator set corresponding to the detail data layer; Based on the design model mapping operator, selecting an ETL mapping operator from the second operator set corresponding to the detailed data layer; Model mapping and storage address configuration processing are performed based on the ETL mapping operator to generate a data processing sub-link corresponding to the detailed data layer.

4. The method according to claim 1, wherein Each data layer in the data warehouse includes a data application layer. Constructing an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer includes: When the operator set of the data application layer includes the third operator set, a derived dimension operator package is selected from the operator market plan, and dimension setting is performed on the initial derived dimension operator in the derived dimension operator package to obtain a derived dimension operator after setting; the dimension setting includes at least one of the following: mandatory dimension setting; hierarchical dimension setting; and joint dimension setting. Performing configuration processing on the set derived dimension operator to obtain a configured derived dimension operator; Adding the configured derived dimension operator to a derived dimension operator set; Performing configuration processing on the derived dimension operator set to generate the third operator set corresponding to the data application layer; When the operator set of the data application layer includes a fourth operator set, a metric calculation rule operator package is selected from the operator market plan, and an aggregation calculation is performed on the operators of the initial metric calculation rule operator package in the metric calculation rule operator package to obtain a calculated operator; Performing configuration processing on the calculated operator to obtain a configured operator; Adding the configured operator to the metric calculation rule operator set; Performing configuration processing on the metric calculation rule operator set to generate the fourth operator set corresponding to the data application layer; Accordingly, performing operator link configuration based on the operator set corresponding to each data layer to obtain a data processing sub-link corresponding to each data layer includes: Selecting a derived dimension operator from the third operator set corresponding to the data application layer; Configuring a derived dimension rule for the derived dimension operator to obtain a configured derived dimension; Based on the configured derived dimension, selecting a metric calculation rule operator from the fourth operator set corresponding to the data application layer; A data processing sub-link corresponding to the data application layer is generated based on the metric calculation rule operator.

5. The method according to claim 1, characterized in that The method of constructing a data processing link corresponding to the data warehouse based on the data processing sub-link corresponding to each data layer based on the particle swarm algorithm PSO includes: The data processing sub-link of the data source access layer, the data processing sub-link of the source data layer, the data processing sub-link of the detail data layer, and the data processing sub-link of the data application layer are sequentially connected in series to form the initial data processing link corresponding to the data warehouse; The configuration process of the initial data processing link is performed according to the particle swarm algorithm PSO to obtain the data processing link corresponding to the data warehouse.

6. A data processing device, characterized in that: The data processing device includes: a construction unit, a configuration unit, and a processing unit; The construction unit is used to construct an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer; each data layer in the data warehouse includes: a data source access layer, a source data layer, a detail data layer, and a data application layer; wherein, the construction of an operator set for each data layer in the data warehouse to obtain an operator set corresponding to each data layer includes: selecting a design model mapping operator package from the operator market planning, configuring the initial design model mapping operator in the design model mapping operator package, and obtaining the configured design model mapping operator. operator; adding the configured design model mapping operator to the design model mapping operator set; performing configuration processing on the design model mapping operator set to generate an operator set corresponding to the pasting source data layer; or, selecting a batch model mapping operator package from the operator market planning, performing configuration processing on the initial batch mapping operator in the batch model mapping operator package to obtain a configured batch mapping operator; adding the configured batch mapping operator to the design model mapping operator set; performing configuration processing on the design model mapping operator set to generate an operator set corresponding to the pasting source data layer; The configuration unit is configured to obtain a model mapping operator based on an operator set corresponding to the source data layer; Constructing a data model mapping relationship for the model mapping operator; configuring a storage address based on the data model mapping relationship, and generating a data processing sub-link corresponding to the source data layer; The construction unit is further configured to construct the data processing link corresponding to the data warehouse according to the data processing sub-link corresponding to each data layer based on a particle swarm algorithm (PSO); The processing unit is configured to process the source data based on the data processing link to obtain the data mart of the data warehouse.

7. A data processing device, characterized in that: The data processing device includes: a processor and a memory; wherein, The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 5 when running the computer program.

8. A computer-readable storage medium, characterized in that The storage medium stores computer program code, and when the computer program code is executed by a computer, the method according to any one of claims 1 to 5 is executed.

Images