Data processing method, data processing device and data warehouse

Abstract

The invention provides a data processing method, and relates to the technical field of data modeling, in particular to the technical field of data warehouses and data processing. According to the specific implementation scheme, relation modeling is conducted on all data of an original data source, and a logic entity table containing the logic incidence relation of all the data is obtained; and performing dimension modeling based on each data in the logic entity table to obtain a fact table and a dimension table. According to the method provided by the invention, relationship modeling and dimension modeling can be combined, the logic association relationship between the data is increased while the multi-dimensional detail information of the data is reserved, and the business process between the data is conveniently obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of data modeling technology, and in particular to the field of data warehouse and data processing technology. Background Art

[0002] Existing data modeling methods are mostly constructed in the form of dimensional modeling, which can create models more efficiently and include detailed information of multiple dimensions of data. However, most data modeling systems based on dimensional modeling still have some problems. For example, existing dimensional modeling systems focus on analyzing a certain business according to different dimensions, but ignore the relationship between data in the business process, and therefore cannot meet the scenarios that require data application based on data relationships. Summary of the invention

[0003] The present disclosure provides a data processing method, a data processing device, and a data warehouse for solving at least one of the above technical problems.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, wherein the method comprises:

[0005] Perform relational modeling on each data in the original data source to obtain a logical entity table containing the logical association relationship of each data;

[0006] Dimension modeling is performed based on each data in the logical entity table to obtain a fact table and a dimension table.

[0007] According to a second aspect of the present disclosure, a data processing device is provided, wherein the device comprises:

[0008] The relationship modeling module is used to perform relationship modeling on each data of the original data source to obtain a logical entity table containing the logical association relationship of each data;

[0009] The logical modeling module is used to perform dimensional modeling based on the data in the logical entity table to obtain a fact table and a dimension table.

[0010] According to a third aspect of the present disclosure, a data warehouse is provided, which at least includes:

[0011] The data relationship layer is used to model the relationships between the data in the original data source and obtain a logical entity table containing the logical relationships between the data.

[0012] The detailed data layer is connected to the data relationship layer and is used to perform dimensional modeling based on the data in the logical entity table to obtain a fact table and a dimension table.

[0013] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0017] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the above method.

[0018] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 is a flowchart of a data processing method provided by the first embodiment of the present disclosure;

[0022] Figure 2 is an exemplary flow chart of S1011;

[0023] Figure 3 is a schematic diagram of table item information of any data in an exemplary user-defined logical entity table;

[0024] Figure 4 is a schematic diagram of a hierarchical structure of a data warehouse provided by the second embodiment of the present disclosure;

[0025] Figure 5 is a flowchart diagram of another data processing method provided by the first embodiment of the present disclosure;

[0026] Figure 6 is a schematic diagram of an exemplary topic division;

[0027] Figure 7 is a schematic diagram of a graphical interface of an exemplary business process relationship diagram;

[0028] Figure 8is a structural schematic diagram of a data processing device provided in the third embodiment of the present disclosure;

[0029] Fig. 9 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0032] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0033] The terminology used herein is for describing particular embodiments only and is not intended to limit the present disclosure.As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0034] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0035] The data processing method according to the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a vehicle-mounted device, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the data processing method provided by the present disclosure can be executed by a server.

[0036] In the first embodiment disclosed, see Figure 1 , Figure 1 A flowchart of a data processing method provided by the first embodiment of the present disclosure is shown, and the method includes:

[0037] S101: Perform relational modeling on each data of the original data source to obtain a logical entity table containing the logical association relationship of each data.

[0038] S102: Perform dimensional modeling based on the data in the logical entity table to obtain a fact table and a dimension table.

[0039] The table item information of any data in the logical entity table at least includes: basic information, field information and logical association relationship information.

[0040] In business scenarios, each business usually has its own database, i.e., the original data source. In the present disclosure, the original data source may include various data sources, and the data of each data source is uniformly processed. The original data source may include: EDAPDataLake, HIVE, DORIS, MYSQL, ORACLE.

[0041] Among them, regarding dimensional modeling, the components of the dimensional model evolve from the information package defined by the requirements. The dimension provides environmental information, and the dimension column set contains rich detailed information, which can be used to collect statistics from multiple dimensions. Dimensional modeling mainly obtains fact tables and dimension tables.

[0042] Fact table: The fact table is the core table in the data warehouse, which records the measurement events in the business process. A fact table is a structure that contains a large number of data values. Granularity: Each row in the fact table represents a business event. The finer the granularity, the more detailed the data. For example, our sales fact table may have a granularity of "each book in each order". The characteristics of a fact table may include: Foreign Keys: Fact tables usually contain multiple foreign keys, which are associated with dimension tables; Measures: These are numeric data that can be aggregated, such as sales amount, quantity, etc.; Additive: Most facts are additive, that is, they can be aggregated on all dimensions.

[0043] Dimension tables: They are placed around the fact table and are used to describe an important aspect of the fact table. Dimension tables are tables used in the data warehouse to describe the characteristics of business objects. They provide context for the numerical measurements in the fact table, making the data easier to understand and analyze. Dimension tables usually contain descriptive, text-based fields that are used for filtering, grouping, and tagging. Characteristics of dimension tables may include, for example: Descriptive attributes: Dimension tables contain a large number of descriptive text fields that are used to provide business context; Hierarchy: Dimension tables contain hierarchies, such as country-province-city in the geographic dimension; Relatively few rows: Dimension tables usually have fewer rows than fact tables, but each row may contain many columns; Primary key: Usually a surrogate key is used as the primary key, which will be used as a foreign key in the fact table.

[0044] Regarding relational modeling, relational data exists in a form called normalization, which uses entities and the logical relationships between them to describe data, thereby describing the business architecture.

[0045] The method provided by the present disclosure performs relational modeling on each data of the original data source to obtain a logical entity table containing the logical association relationship of each data, and then performs dimensional modeling based on each data in the logical entity table to obtain a fact table and a dimension table, so that relational modeling and dimensional modeling can be combined, while retaining the multi-dimensional detailed information of the data, the logical association relationship between the data is increased to facilitate the acquisition of the business process between the data. In other words, while still being able to output multi-dimensional data reports through fact tables and dimension tables, the logical association relationship between each data can also be associated to connect the business process, and is suitable for various scenarios that require data application based on data relationships.

[0046] In some examples, S101 includes multiple methods.

[0047] Method 1: Reverse data relationship modeling.

[0048] Among them, S101 includes:

[0049] S1011. Determine the logical association relationship between each data based on the foreign key table and the primary key table of each data in the original data source.

[0050] Usually, in a relational database, a foreign key table and a primary key foreign key table are used to express the relationship between data. In S1011, the foreign key table and the primary key table in the original data source are directly read and analyzed to sort out the logical association relationship between each data.

[0051] Among them, the primary key is a field or combination of fields used to uniquely identify each row of data in a relational table. The role of the primary key is to ensure that each row of data in the table can be uniquely identified and accessed. Common primary key choices are to use a self-increasing integer or a unique field.

[0052] A foreign key is a field used to establish a relationship between tables. A foreign key is a field that associates another table's primary key, thereby creating a relationship between tables. The role of a foreign key is to achieve association between tables. Through foreign keys, data can be queried and operated between related tables. Foreign keys can be used to implement data integrity constraints, such as implementing cascading updates or cascading deletes.

[0053] Specifically, see Figure 2 , Figure 2 An exemplary flow chart of S1011 is shown, where S1011 includes:

[0054] S1011a. Determine whether the foreign key table exists.

[0055] S1011b1. If the foreign key table exists, determine whether the foreign key table has been published.

[0056] S1011b2. If the foreign key table does not exist, end.

[0057] S1011c1. When it is determined that the foreign key table has been published, determine whether the foreign key table is the latest version.

[0058] Make sure that the foreign key table obtained is a published version to ensure that the obtained logical association is correct.

[0059] Wherein, S1011c1 includes the following sub-steps:

[0060] Sub-step 1: Determine whether the foreign key table includes the foreign key table of the child version.

[0061] Sub-step 2a: If the foreign key table of the sub-version is not included, the current foreign key table is used as the latest version.

[0062] Sub-step 2b: If a foreign key table of a sub-version is included, obtain the foreign key table of the sub-version as the foreign key table of the latest version.

[0063] In some cases, foreign keys may be modified to generate a foreign key table for a sub-version. Therefore, it is necessary to determine whether there is a new foreign key table for a sub-version that has taken effect, so as to obtain the latest foreign key for relationship modeling and ensure the accuracy of the logical association of the data.

[0064] S1011c2. If it is determined that the foreign key table is not published, end.

[0065] S1011d1. When it is determined that the foreign key table is the latest version, determine whether the primary key table exists and whether the primary key field and the foreign key field exist.

[0066] After obtaining the correct foreign key table, further obtain the primary key table and determine whether the foreign key field in the foreign key table and the primary key field in the primary key table exist. If they do exist, then build a logical association relationship.

[0067] S1011d2. When it is determined that the foreign key table is not the latest version, find the latest version of the foreign key table.

[0068] The specific method can be searched according to the methods of sub-step 1, sub-step 2a, and sub-step 2b.

[0069] S1011e1. When it is determined that the primary key table exists and the primary key field and the foreign key field exist, a logical association relationship between each data is constructed based on the latest version of the foreign key table and the primary key table.

[0070] S1011e2. When it is determined that the primary key table does not exist and / or when it is determined that the primary key field and the foreign key field exist, the process ends.

[0071] Through the above method, it is ensured that the correct and latest version of the foreign key table and primary key table are found to build the relationship between the data, so as to ensure the accuracy of the relationship modeling.

[0072] S1012. Perform relational modeling on each data based on the logical association relationship to obtain a logical entity table.

[0073] In the related technology, databases often use relational databases to store data. In the one-point method, a data reverse method can be provided to automatically migrate data from the relational database of the original data source to the data warehouse, and then perform dimensional modeling.

[0074] Method 2: Relational modeling of custom data relationships.

[0075] Among them, S101 includes:

[0076] S1011. Based on the user-defined information of each data input by the user, determine the basic information, field information and logical association relationship information of each data contained therein.

[0077] S1012: Perform relational modeling on each data based on basic information, field information and logical association relationship information of each data to obtain a logical entity table.

[0078] See also Figure 3 , Figure 3A schematic diagram showing the table item information of any data in an exemplary user-defined logical entity table. As an example, basic information may include: data hierarchy, subject domain, logical entity name, description, etc.; field information may include: name, Chinese name, field type, description, etc.; logical association relationship information may include: logical association entities, logical association relationship fields, etc., which are not limited here.

[0079] In this way, users can customize data logical entities and logical relationships.

[0080] It should be noted that the above method 1 and method 2 can be used in combination, which is not limited here.

[0081] See also Figure 4 , Figure 4 A schematic diagram of a hierarchical structure of an exemplary data warehouse is shown. The method provided by the present disclosure is applied in a data warehouse. The data warehouse may include: an operation data store (ODS), a data relationship layer (DWR), a detail data layer (DWD), a summary data layer (DWS), and an application data service (ADS).

[0082] The data introduction layer saves the data of the original database and plays the role of data backup. It can use various types of compression formats to save the original data as a source table. In the present disclosure, the data introduction layer can not only store the original data, but also send the data with logical associations to the data introduction layer through data synchronization after the detailed data layer processes the data through relationship modeling. The data introduction layer stores various data with logical associations.

[0083] Data relationship layer: can directly access the original data source, clean the data through the original data source, and obtain the logical relationship between the data based on the relationship modeling method, use the relationship modeling method to model the data, convert the data into logical entities, and obtain the logical entity table. Among them, the data relationship layer is connected to the original data source, data introduction layer, detailed data layer, summary data layer, and data application layer. When generating reports in the detailed data layer, summary data layer, and data application layer, the logical relationship in the data relationship layer can be used, or the input data relationship layer in the detailed data layer, summary data layer, and data application layer can be relationship modeled based on the data table processed by each layer.

[0084] Detailed data layer: represents the smallest granularity layer of the business. The detailed data layer is the detail layer of the data warehouse, storing detailed data that has been initially cleaned and converted. The detailed data layer will further clean the data and determine the data specifications. The data is stored in a standardized manner, so that the data has a certain standardization and consistency, and is usually used for subsequent data aggregation and analysis. In the detailed data layer of the present disclosure, each data is remodeled using a dimensional model to obtain fact tables and dimension tables, in preparation for subsequent layers.

[0085] Summary data layer: The summary data layer stores aggregated data. The data in the summary data layer is usually to meet the needs of enterprise-level reports and data analysis, with high computing density and query efficiency. When data enters the summary data layer from the detailed data layer, it is usually necessary to perform data aggregation calculations, such as processing and use according to each subject, which is the aggregate value of the fact table, such as statistics on the behavior of each subject object for 7 days, 30 days, and 90 days, or aggregating the behavior of the subject object on a single day to form a multi-column wide table. After associating the dimension information, statistical analysis is performed from different angles.

[0086] Data application layer: provides data for various statistical reports. It is the top layer of the data warehouse and is usually used for application-oriented data storage. The main function of this layer is to save the result data and provide a query interface for external systems. It is usually customized data for specific business needs and supports real-time query and fast response. In the data application layer, various business statistical indicators are preset according to business needs. For different business statistical indicators, multiple application tables are generated, so that data with different needs can be responded to inquiries and feedback.

[0087] See also Figure 5 , Figure 5 A flowchart of another data processing method provided by the first embodiment of the present disclosure is shown, the method comprising:

[0088] S200: Import and save various data in the original data source.

[0089] This step is the processing step of the data introduction layer. The data introduction layer can directly access the original data source and store the original data. It can also send the data with logical associations to the data introduction layer through data synchronization after the detailed data layer processes the data through relational modeling. The data introduction layer stores each data with logical associations.

[0090] S201. Set a theme for each data.

[0091] Data can be reasonably divided according to business themes to organize business data and achieve data classification and aggregation. Users can build subject domains and themes based on their business, which makes it convenient for users to divide different business processes according to their needs.

[0092] Among them, S201 includes:

[0093] Step 1: Read each data in the original data source and set the subject domain for each data.

[0094] Step 2: For each data in any subject domain, set the subject to which each data belongs.

[0095] See also Figure 6 , Figure 6 A schematic diagram showing an exemplary topic division.

[0096] Among them, the subject domain: the subject domain is a domain that further abstracts and divides the subjects of the same nature or type, and contains several closely related subjects.

[0097] Topic: A topic is a central summary of a business expressed based on a business scenario or data. A topic domain can include multiple topics.

[0098] In the present disclosure, a subject or subject domain may be applied at each layer and participate in data processing at each layer.

[0099] S203: Perform relationship modeling on each data of the original data source to obtain a logical entity table containing the logical association relationship of each data.

[0100] This step is the processing step of the data relationship layer, see above.

[0101] S204: Perform dimensional modeling based on the data in the logical entity table to obtain a fact table and a dimension table.

[0102] This step is the processing step of the detailed data layer. The detailed data layer will further clean the data, determine the data specifications, store the data in a standardized manner, make the data have certain specifications and consistency, and perform dimension modeling on the data to obtain fact tables and dimension tables. In it, you can add topics and subject domains to obtain fact tables and dimension tables exclusive to the subject of the business, or you can not introduce topics and subject domains, which is not limited here.

[0103] S205: Based on the logical associations among the data in the subject and / or subject domain, the fact table and the dimension table, and the logical entity table, aggregate calculations are performed on the data to obtain a summary table of the data.

[0104] This step is the processing step of the summary data layer. When data enters the summary data layer from the detailed data layer, it is usually necessary to perform data aggregation calculations to aggregate the data of multiple fact tables and dimension tables, obtain aggregate values, and then form a summary table in the form of a wide table. In it, you can add topics and subject domains to obtain a summary table exclusive to the subject of the business, or you can not introduce topics and subject domains, which is not limited here.

[0105] S206: Based on the preset business statistical indicators, combined with the logical association relationship between each data in the summary table, the subject and / or subject domain, and the logical entity table, an application table corresponding to the business statistical indicators is generated.

[0106] At the data application layer, various business statistical indicators are preset according to business needs. For different business statistical indicators, multiple statistical application tables are generated, so that data with different needs can be responded to and fed back. In it, topics and subject domains can be added to obtain application tables dedicated to business topics, or topics and subject domains can be omitted, which is not limited here.

[0107] In some examples, after S204, the method provided by the present disclosure further includes:

[0108] Step 1: According to the logical association relationship of each data in the logical entity table, the data in the fact table and dimension table are connected in series to obtain the business process corresponding to each data.

[0109] Step 2: Generate a business process relationship diagram based on the business process.

[0110] See also Figure 7 , Figure 7 A schematic diagram of a graphical interface of an exemplary business process relationship diagram is shown. The business process relationship diagram may be an entity relationship (ER) diagram or other types of diagrams. The diagram includes multiple logical entities and the logical associations therein. Through such a business process relationship diagram, the business process can be intuitively seen.

[0111] The method provided by the present disclosure combines dimensional modeling and relational modeling. On the one hand, it can facilitate the direct migration of data from a relational database to a data warehouse. On the other hand, it can introduce logical associations between data to meet scenarios where data applications are performed based on data relationships. For example, a business process relationship diagram can be generated to directly display the business process.

[0112] In disclosing the second embodiment, see Figure 4 , Figure 4 A schematic diagram of the structure of a data warehouse provided by the second embodiment of the present disclosure is shown, and the data warehouse at least includes:

[0113] The data relationship layer is used to perform relationship modeling on the data of the original data source to obtain a logical entity table containing the logical association relationship of each data.

[0114] Furthermore, the data relationship layer can be directly connected to the original data source, and data cleaning is performed on each data through the original data source, and the logical association relationship of each data is obtained based on the relationship modeling method, and the data modeling is performed using the relationship modeling method to convert each data into a logical entity to obtain a logical entity table. Among them, the data relationship layer is connected to the original data source, the data introduction layer, the detailed data layer, the summary data layer, and the data application layer. When the detailed data layer, the summary data layer, and the data application layer generate reports, the logical association relationship in the data relationship layer can be used, or the input data relationship layer in the detailed data layer, the summary data layer, and the data application layer can be relationally modeled based on the data table processed by each layer.

[0115] The detailed data layer is connected to the data relationship layer and is used to perform dimensional modeling based on the data in the logical entity table to obtain fact tables and dimension tables.

[0116] Furthermore, the detailed data layer represents the smallest granularity layer of the business. The detailed data layer is the detail layer of the data warehouse, storing detailed data that has been initially cleaned and converted. The detailed data layer will further clean the data and determine the data specifications. The data is stored in a standardized manner, so that the data has a certain standardization and consistency, which is usually used for subsequent data aggregation and analysis. In the detailed data layer of the present disclosure, each data is remodeled in the form of a dimensional model to obtain a fact table and a dimension table, in preparation for the subsequent layers.

[0117] Continue to see Figure 4 , the data warehouse may also include:

[0118] The data introduction layer is connected to the data relationship layer and the detailed data layer respectively, and is used for various data of the original data source.

[0119] Furthermore, the data introduction layer saves the data of the original database and plays the role of data backup. It can use various types of compression formats to save the original data as a source table. In the present disclosure, the data introduction layer can not only store the original data, but also send the data with logical associations to the data introduction layer through data synchronization after the detailed data layer processes the data through relationship modeling. The data introduction layer stores each data with logical associations.

[0120] The summary data layer is connected to the data relationship layer and the detail data layer respectively, and is used to aggregate and calculate each data based on the logical association relationship between each data in the subject and / or subject domain, fact table and dimension table, and logical entity table to obtain a summary table of each data.

[0121] Furthermore, the summary data layer stores aggregated data. The data in the summary data layer is usually to meet the needs of enterprise-level reports and data analysis, with high computing density and query efficiency. When data enters the summary data layer from the detailed data layer, it is usually necessary to perform data aggregation calculations, such as processing and use according to each subject, which is the aggregate value of the fact table, such as statistics on the behavior of each subject object for 7 days, 30 days, and 90 days, or aggregating the behavior of the subject object on a single day to form a multi-column wide table, and after associating the dimension information, statistical analysis is performed from different angles.

[0122] The data application layer is connected to the summary data layer and the detailed data layer respectively, and is used to generate application tables corresponding to business statistical indicators based on preset business statistical indicators and the logical association relationship between each data in the summary table, subject and / or subject domain, and logical entity table.

[0123] Furthermore, the data application layer provides data for various statistical reports. It is the top layer of the data warehouse and is usually used for application-oriented data storage. The main function of this layer is to save the result data and provide a query interface for external systems. It is usually customized data for specific business needs and supports real-time query and fast response. In the data application layer, various business statistical indicators are preset according to business needs. For different business statistical indicators, multiple application tables are generated, so that data with different needs can be responded to inquiries.

[0124] Based on Figure 1 The same principle, Figure 8 A data processing device 80 provided in the third embodiment of the present disclosure is shown, and the device includes:

[0125] The relationship modeling module 801 is used to perform relationship modeling on each data of the original data source to obtain a logical entity table containing the logical association relationship of each data;

[0126] The logical modeling module 802 is used to perform dimensional modeling based on the data in the logical entity table to obtain a fact table and a dimension table.

[0127] In some examples, the relationship modeling module is specifically used to:

[0128] A relationship determination submodule is used to determine the logical association relationship of each data based on the foreign key table and the primary key table of each data in the original data source;

[0129] The modeling submodule is used to perform relational modeling on each data based on the logical association relationship to obtain a logical entity table.

[0130] In some examples, the relationship determination submodule is specifically used to:

[0131] Determine whether the foreign key table exists;

[0132] In the case of yes, determine whether the foreign key table is published;

[0133] If yes, determine whether the foreign key table is the latest version;

[0134] When it is determined that the foreign key table is the latest version, determine whether the primary key table exists and whether the primary key field and the foreign key field exist;

[0135] If yes, build the logical association relationship of each data based on the latest version of the foreign key table and primary key table.

[0136] In some examples, the relationship determination submodule determines whether the foreign key table is the latest version when the condition is yes, specifically for:

[0137] Determine whether the foreign key table includes the child version's foreign key table;

[0138] If not, the current foreign key table is used as the latest version;

[0139] If so, get the foreign key table of the child version as the latest version.

[0140] In some examples, the table item information of any data in the logical entity table includes at least: basic information, field information, and logical association relationship information.

[0141] In some examples, the relationship modeling module is specifically used to:

[0142] Based on the user-defined information of each data input by the user, determine the basic information, field information and logical association relationship information of each data contained therein;

[0143] Based on the basic information, field information and logical association information of each data, relational modeling is performed on each data to obtain a logical entity table.

[0144] In some examples, the apparatus further comprises:

[0145] The subject domain module is used to read each data in the original data source and set the subject domain to which each data belongs;

[0146] The subject module is used to set the subject for each data in any subject domain.

[0147] In some examples, the apparatus further comprises:

[0148] The summary module is used to perform aggregation calculations on each data based on the logical associations of each data in the subject and / or subject domain, fact table and dimension table, and logical entity table to obtain a summary table of each data.

[0149] In some examples, the apparatus further comprises:

[0150] The application module is used to generate application tables corresponding to business statistical indicators based on preset business statistical indicators and in combination with the logical associations of various data in summary tables, themes and / or subject domains, and logical entity tables.

[0151] In some examples, the apparatus further comprises:

[0152] The drawing module is used to connect the data in the fact table and dimension table according to the logical association relationship of each data in the logical entity table to obtain the business process corresponding to each data;

[0153] Generate a business process relationship diagram based on the business process.

[0154] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0155] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, a computer program product and an autonomous driving vehicle.

[0156] The computer instructions of the non-transitory computer-readable storage medium storing computer instructions are used to make a computer execute the above method.

[0157] A computer program product comprises a computer program, which implements the above method when executed by a processor.

[0158] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0159] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0160] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0161] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured as a data processing method in any other appropriate manner (e.g., by means of firmware).

[0162] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0164] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0165] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0166] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0167] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0168] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0169] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, wherein: The method comprises: Perform relational modeling on each data in the original data source to obtain a logical entity table containing the logical association relationship of each data; Dimension modeling is performed based on each data in the logical entity table to obtain a fact table and a dimension table.

2. The method according to claim 1, wherein: The relational modeling is performed on each data of the original data source to obtain a logical entity table containing the logical association relationship of each data, including: Determine the logical association relationship of each data based on the foreign key table and the primary key table of each data in the original data source; Relationship modeling is performed on each data based on the logical association relationship to obtain the logical entity table.

3. The method according to claim 2, wherein: The determining the logical association relationship of each data based on the foreign key table and the primary key table of each data in the original data source includes: Determine whether the foreign key table exists; In the case of yes, determine whether the foreign key table is published; If yes, determine whether the foreign key table is the latest version; When it is determined that the foreign key table is the latest version, determine whether the primary key table exists and whether the primary key field and the foreign key field exist; If yes, a logical association relationship between the various data is constructed based on the latest version of the foreign key table and the primary key table.

4. The method according to claim 3, wherein: In the case of yes, determining whether the foreign key table is the latest version includes: Determine whether the foreign key table includes a sub-version foreign key table; If not, the current foreign key table is used as the latest version; If so, get the foreign key table of the child version as the latest version.

5. The method according to any one of claims 1 to 4, wherein: The table entry information of any data in the logical entity table at least includes: basic information, field information and logical association relationship information.

6. The method according to claim 5, wherein: The relational modeling is performed on each data of the original data source to obtain a logical entity table containing the logical association relationship of each data, including: Based on the user-defined information of each data input by the user, determine the basic information, field information and logical association relationship information of each data contained therein; Relationship modeling is performed on each data based on the basic information, field information and logical association relationship information of each data to obtain the logical entity table.

7. The method according to any one of claims 1 to 6, wherein: Before performing relationship modeling on each data of the original data source to obtain a logical entity table containing logical association relationships of each data, the method further includes: Read each data in the original data source, and set the subject domain to which each data belongs; For each data in any subject domain, set the subject to which each data belongs.

8. The method according to claim 7, wherein: After performing dimensional modeling based on each data in the logical entity table to obtain a fact table and a dimension table, the method further includes: Based on the logical association relationship between the subject and / or subject domain, the fact table, the dimension table and the data in the logical entity table, aggregate calculation is performed on each data to obtain a summary table of each data.

9. The method according to claim 8, wherein: After performing aggregation calculation on each data based on the logical association relationship between each data in the subject, the fact table, the dimension table, and the logical entity table to obtain a summary table of each data, the method further includes: Based on preset business statistical indicators, combined with the summary table, the subject and / or the subject domain, and the logical association relationship among the data in the logical entity table, an application table corresponding to the business statistical indicators is generated.

10. The method according to any one of claims 1 to 9, wherein: After performing dimensional modeling based on each data in the logical entity table to obtain a fact table and a dimension table, the method further includes: According to the logical association relationship of each data in the logical entity table, the data in the fact table and the dimension table are connected in series to obtain the business process corresponding to each data; A business process relationship diagram is generated based on the business process.

11. A data processing device, wherein: The device comprises: The relationship modeling module is used to perform relationship modeling on each data of the original data source to obtain a logical entity table containing the logical association relationship of each data; The logical modeling module is used to perform dimensional modeling based on the data in the logical entity table to obtain a fact table and a dimension table.

12. The device according to claim 11, wherein The relationship modeling module is specifically used for: A relationship determination submodule, used to determine the logical association relationship of each data based on the foreign key table and the primary key table of each data in the original data source; The modeling submodule is used to perform relational modeling on each data based on the logical association relationship to obtain the logical entity table.

13. The device according to claim 12, wherein: The relationship determination submodule is specifically used for: Determine whether the foreign key table exists; In the case of yes, determine whether the foreign key table is published; If yes, determine whether the foreign key table is the latest version; When it is determined that the foreign key table is the latest version, determine whether the primary key table exists and whether the primary key field and the foreign key field exist; If yes, a logical association relationship between the various data is constructed based on the latest version of the foreign key table and the primary key table.

14. The device according to claim 13, wherein: The relationship determination submodule determines whether the foreign key table is the latest version in the case of yes, specifically for: Determine whether the foreign key table includes a sub-version foreign key table; If not, the current foreign key table is used as the latest version; If so, get the foreign key table of the child version as the latest version.

15. The device according to any one of claims 11 to 14, wherein: The table item information of any data in the logical entity table at least includes: basic information, field information and logical association relationship information.

16. The device according to claim 15, wherein: The relationship modeling module is specifically used for: Based on the user-defined information of each data input by the user, determine the basic information, field information and logical association relationship information of each data contained therein; Relationship modeling is performed on each data based on the basic information, field information and logical association relationship information of each data to obtain the logical entity table.

17. The device according to any one of claims 11 to 16, wherein: The device also includes: A subject domain module is used to read each data in the original data source and set the subject domain to which each data belongs; The subject module is used to set the subject for each data in any subject domain.

18. The device according to claim 17, wherein: The device also includes: The summary module is used to perform aggregation calculation on each data based on the logical association relationship between each data in the subject and / or the subject domain, the fact table and the dimension table, and the logical entity table to obtain a summary table of each data.

19. The device according to claim 18, wherein: The device also includes: The application module is used to generate an application table corresponding to the business statistical indicators based on preset business statistical indicators, combined with the logical association relationship between the summary table, the subject and / or the subject domain, and the data in the logical entity table.

20. The device according to any one of claims 11 to 19, wherein: The device also includes: A drawing module, used to perform relational concatenation on each data in the fact table and the dimension table according to the logical association relationship of each data in the logical entity table, so as to obtain the business process corresponding to each data; A business process relationship diagram is generated based on the business process.

21. A data warehouse, wherein: At least: The data relationship layer is used to model the relationships between the data in the original data source and obtain a logical entity table containing the logical relationships between the data. The detailed data layer is connected to the data relationship layer and is used to perform dimensional modeling based on the data in the logical entity table to obtain a fact table and a dimension table.

22. The data warehouse according to claim 21, wherein: The data warehouse also includes: The summary data layer is connected to the data relationship layer and the detailed data layer respectively, and is used to perform aggregation calculation on each data based on the logical association relationship between each data in the subject and / or subject domain, the fact table and the dimension table, and the logical entity table to obtain a summary table of each data.

23. The data warehouse according to claim 21 or 22, wherein: The data warehouse also includes: The data application layer is connected to the summary data layer and the detailed data layer respectively, and is used to generate an application table corresponding to the business statistical indicators based on preset business statistical indicators and in combination with the logical association relationship between the summary table, the subject and / or the subject domain, and the logical entity table.

24. The data warehouse according to any one of claims 21 to 23, wherein: The data warehouse also includes: The data introduction layer is connected to the data relationship layer and the detailed data layer respectively, and is used for each data of the original data source.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images