Data processing method, apparatus, computer-readable medium, and computer program product

Abstract

The application discloses a data processing method, device, computer readable medium and computer program product. The method obtains data in a data source by using a first data warehouse in a hybrid data warehouse architecture, obtains source data, loads the source data to a second data warehouse in the hybrid data warehouse architecture for processing, transmits processing result data to the first data warehouse, stores the processing result data in the first data warehouse, and provides data services based on the stored processing result data. The data storage and performance of the first data warehouse are higher than those of the second data warehouse, and the computing performance of the first data warehouse is lower than that of the second data warehouse. By fusing data warehouses with performance advantages in different aspects, the advantages of the warehouse systems are fully utilized, and the problems of large expansion difficulty, slow performance, difficult mining of various data types, strict data compliance requirements, high security maintenance cost and the like are effectively solved.

Description

Technical Field

[0001] This application belongs to the field of Internet data processing technology, and in particular relates to a data processing method, apparatus, computer-readable medium and computer program product. Background Technology

[0002] In today's era of internet finance, data is expanding rapidly, and the demand for expansion is strong. As the amount of data grows larger and the collection of multi-source data continues to improve, the financial sector is accumulating massive amounts of structured, semi-structured, and unstructured data, which can no longer meet the challenges of large-scale distributed data processing at low cost.

[0003] At the same time, as the amount of data continues to increase and the sources of data become more diverse, data-based application systems are also constantly being updated and upgraded, leading to a rapid increase in the demand for underlying IT. Traditional IT (Internet Technology) construction models and architectures can no longer meet these demands. Summary of the Invention

[0004] In view of this, this application provides a data processing method, apparatus, computer-readable medium, and computer program product, which provides and implements a hybrid data warehouse architecture to address problems such as high scalability, slow performance, strict data compliance requirements, and high security maintenance costs.

[0005] The specific plan is as follows:

[0006] A data processing method, the method comprising:

[0007] Data from the data source is obtained using the first data warehouse in the hybrid data warehouse architecture to obtain the source data;

[0008] The source data is loaded into the second data warehouse in the hybrid data warehouse architecture, where the second data warehouse processes the source data and transmits the processed data to the first data warehouse.

[0009] The first data warehouse is used to store the processing results data, and data services are provided to external parties based on the stored processing results data;

[0010] The data storage performance and data service performance based on the stored data of the first data warehouse are higher than those of the second data warehouse, respectively, while the computing performance of the first data warehouse is lower than that of the second data warehouse.

[0011] Optionally, obtaining source data by utilizing the first data warehouse in the hybrid data warehouse architecture to acquire data from the data source includes:

[0012] Data is collected from the data source using the first data warehouse, the collected data is parsed, and the parsed data is stored by data pasting to the source to obtain pasted source data.

[0013] Optionally, loading the source data into the second data warehouse in the hybrid data warehouse architecture includes:

[0014] Write the source data into a data table of the target table structure, and store the data table in the specified directory;

[0015] The data table is loaded from the specified directory to the second data warehouse through a preset data loading module; wherein the target table structure is a table structure that matches the data structure of the data file in the first data warehouse.

[0016] Optionally, the processing of the loaded patch source data by the second data warehouse includes:

[0017] Based on the data warehouse logical model and data warehouse physical model provided by the second data warehouse, the loaded source data is processed.

[0018] The data warehouse logical model is used to describe the data and the relationships between data, while the data warehouse physical model is used to describe the data layering and processing links and the database table structure.

[0019] Optionally, the processing of the loaded source data based on the data warehouse logical model and data warehouse physical model provided by the second data warehouse includes:

[0020] Based on the functions of each data layer in the data layering processing link provided by the data warehouse physical model, the loaded source data is processed.

[0021] During the processing, based on the database table structure provided by the data warehouse logical model and the data warehouse physical model, the field mapping and database table association relationships between different layers in the data layering processing link are determined.

[0022] Optionally, each data layer in the data layering processing chain includes: a second basic topic layer, a second summary and sharing layer, and a second application data layer;

[0023] The second basic topic layer is used to classify and integrate the loaded post source data according to the topic;

[0024] The second summary and sharing layer is used to extract, summarize, and integrate the data output by the model layer according to business needs;

[0025] The second application data layer is used to perform application-level reorganization and processing on the data output by the summary sharing layer.

[0026] Optionally, the data layering of the first data warehouse includes a first basic theme layer, a first summary and sharing layer, and a first application data layer;

[0027] The step of transmitting the processing result data to the first data warehouse includes:

[0028] Through the preset data download module:

[0029] The processed result data generated by the second basic theme layer of the second data warehouse is transmitted down to the first basic theme layer of the first data warehouse;

[0030] The processed result data generated by the second aggregation and sharing layer of the second data warehouse is transmitted down to the first aggregation and sharing layer of the first data warehouse;

[0031] The processed result data generated by the second application data layer of the second data warehouse is transmitted down to the first application data layer of the first data warehouse.

[0032] Optionally, the step of using the first data warehouse to store processing result data and providing data services externally based on the stored processing result data includes:

[0033] The first basic theme layer, the first summary and sharing layer, and the first application data layer of the first data warehouse respectively store the processed result data received.

[0034] Data services are provided to the outside world through the data interface layer in the first data warehouse that is connected to the first basic theme layer, the first summary and sharing layer and the first application data layer.

[0035] A data processing apparatus, the apparatus comprising:

[0036] The data acquisition unit is used to acquire data from the data source using the first data warehouse in the hybrid data warehouse architecture to obtain the source data.

[0037] A data loading unit is used to load source data into the second data warehouse in the hybrid data warehouse architecture;

[0038] The processing unit is used to process the source data in the second data warehouse;

[0039] The data transmission unit is used to transmit the processing results data to the first data warehouse;

[0040] The data storage and data service unit is used to store the processing result data in the first data warehouse and provide data services to external parties based on the stored processing result data.

[0041] The data storage performance and data service performance based on the stored data of the first data warehouse are higher than those of the second data warehouse, respectively, while the computing performance of the first data warehouse is lower than that of the second data warehouse.

[0042] A computer-readable medium having a computer program stored thereon, the computer program comprising program code for performing the methods described in any of the preceding descriptions.

[0043] A computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods described in any of the preceding descriptions.

[0044] In summary, the data processing method, apparatus, computer-readable medium, and computer program product provided in this application utilize a first data warehouse in a hybrid data warehouse architecture to obtain data from a data source, obtain source data, load the source data into a second data warehouse in the hybrid data warehouse architecture for processing, transmit the processed result data to the first data warehouse, store the processed result data in the first data warehouse, and provide data services based on the stored processed result data; wherein, the data storage and service performance of the first data warehouse is higher than that of the second data warehouse, and the computing performance of the first data warehouse is lower than that of the second data warehouse.

[0045] As can be seen, this application proposes a hybrid data warehouse architecture based on a first and a second data warehouse. The first data warehouse is used to carry the data storage and data service functions of the hybrid data warehouse architecture, while the second data warehouse is used to carry the highly complex data computing functions of the hybrid data warehouse architecture. By integrating multiple data warehouses with different performance advantages and giving full play to the advantages of each warehouse system, it effectively solves the problems of high expansion difficulty, slow performance, difficulty in mining diverse data types, strict data compliance requirements, and high security maintenance costs. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1This is a flowchart illustrating the data processing method provided in this application;

[0048] Figure 2 This is an exemplary business process of the hybrid data warehouse architecture based on Hadoop-MPP provided in this application;

[0049] Figure 3 This is a structural diagram of the data processing apparatus provided in this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In today's era of internet finance, traditional IT construction models and architectures cannot meet the demands, resulting in problems such as difficulty in scaling up, slow performance, strict data compliance requirements, and high security maintenance costs. Furthermore, the applicant discovered that industries such as finance possess massive amounts of structured, semi-structured, and unstructured data. The convergence of semi-structured and unstructured data forms "dark data" that is difficult to mine, and how to more effectively utilize this data has become a major challenge for industries such as finance.

[0052] Therefore, this application provides a data processing method, apparatus, computer-readable medium, and computer program product, and solves the above-mentioned problems by proposing and implementing a hybrid data warehouse architecture.

[0053] See Figure 1 The data processing method flowchart shown in this application includes the following processing flow:

[0054] Step 101: Use the first data warehouse in the hybrid data warehouse architecture to obtain data from the data source to obtain the source data.

[0055] This application proposes a hybrid data warehouse architecture, which includes a first data warehouse and a second data warehouse.

[0056] Among them, the data storage performance and data service performance based on the stored data of the first data warehouse are higher than those of the second data warehouse, respectively, while the computing performance of the first data warehouse is lower than that of the second data warehouse.

[0057] In the operation of this hybrid data warehouse architecture, the first data warehouse is used to carry the data storage and data service functions of the architecture, and the second data warehouse is used to carry the high-complexity data computing functions of the architecture for data processing. In other words, the use of functions of the first and second data warehouses that do not have performance advantages is avoided as much as possible. The advantages of the first and second data warehouses are integrated in a targeted manner to form the functions required for the complete workflow of the hybrid data warehouse architecture, so as to give full play to the advantages of the first and second data warehouses.

[0058] It should be understood that the first data warehouse involves some computational processing (such as SQL assembly, transformation, and querying) in its data storage and data service functions based on the stored data, while the second data warehouse still needs a certain storage capacity (such as cached source data and related intermediate data on the processing path) in its data processing and computation process. However, these are all within the capacity of the respective data warehouse systems.

[0059] Alternatively, the server nodes of the first data warehouse can be expanded more than those of the second data warehouse, and the first data warehouse can handle structured, semi-structured and unstructured data. All these functions of the first data warehouse are integrated and applied in the hybrid data warehouse architecture.

[0060] In practical applications, one or more additional features can be designed as needed on the first and / or second data warehouse sides of the hybrid data warehouse architecture.

[0061] When applying this application, a first data warehouse may be obtained by adapting a series of functions to the hybrid data warehouse architecture of this application, based on the Hadoop data warehouse architecture as a prototype; or a second data warehouse may be obtained by adapting a series of functions to the hybrid data warehouse architecture of this application, based on the MPP (Massively Parallel Processing) data warehouse architecture as a prototype.

[0062] Among them, Hadoop's server nodes have strong scalability and can handle unstructured data, supporting data volumes of more than 10P. However, its system management, development and maintenance costs are high, and its efficiency in operating relational data is not as good as MPP.

[0063] The Hadoop ecosystem comprises a file storage system, a data acquisition system, an analytics and computing system, and a task scheduling system. The file storage system divides massive amounts of data into fixed-size file blocks and stores them on DataNodes within a distributed physical cluster of numerous inexpensive machines. The NameNode manages data operations on HDFS (HDFS Distributed File System) through metadata management. The data acquisition system moves data from traditional databases, queues, and various other storage methods to the Hadoop ecosystem or other data processing frameworks. The analytics and computing system maps structured data files to database tables and provides simple SQL query functionality, converting SQL statements into MapReduce tasks for execution. The task scheduling system is crucial after the scale of computing tasks and the establishment of the data warehouse have been expanded, as daily data processing may rely on different processing logics, forming business workflows.

[0064] MPP is a distributed parallel structured database cluster with a shared-nothing architecture, featuring high performance, high availability, and high scalability. MPP is highly efficient at processing relational SQL and supports transactional data processing, but its server node expansion capabilities are limited.

[0065] The main difference between MPP and the Hadoop architecture is that MPP fully utilizes the physical resources of each machine, including storage, CPU, and memory. Specifically, it uses each machine's resources for parallel computation and then returns the results. Each machine's resources are independent and not shared. In contrast, the Hadoop architecture shares resources across multiple machines, such as disk, CPU, and memory. Furthermore, recent MPP architectures have primarily adopted columnar storage (or it's an optional feature). Columnar storage offers better compression performance for massive datasets and provides better performance in OLAP (Online Analytical Processing), where only a few columns need to be analyzed. Finally, in terms of computation, MPP architecture uses a vectorized computing engine, leveraging the CPU's SIMD (Single Instruction Multiple Data) instruction set to accelerate the entire computation process, resulting in faster speeds.

[0066] In this embodiment, data is collected from multiple data sources (e.g., traditional databases, queues, and other storage) at the first data warehouse. The collected data is parsed, and the parsed data is stored in a source-attached manner using the distributed file system provided by the first data warehouse to obtain source-attached data. The parsing and processing of the collected data includes, but is not limited to, decompressing, transcoding, and decrypting the collected data.

[0067] For example, the following uses Hadoop and MPP as examples to provide the composition structure of a hybrid data warehouse architecture. This Hadoop-MPP-based hybrid data warehouse architecture mainly includes five modules:

[0068] 11) File acquisition module based on Hadoop;

[0069] 12) Hadoop to MPP data loading module (Hadoop2MPP);

[0070] 13) Data warehouse model module;

[0071] 14) Big Data Scheduling Module;

[0072] 15) MPP to Hadoop data download module (MPP2Hadoop).

[0073] The Hadoop-based file acquisition module can be used to decompress, transcode, and decrypt files from OLTP (Online Transaction Processing) systems. It uses Hadoop's distributed file system to store the data in source-pasting format, preserving the cleaned chain and slice data from the OLTP system. For example, by using source-pasting data time-point data, it can specifically retain the full cross-section of the most recent 7 days or the full cross-section at the end of each month.

[0074] The Hadoop-based file acquisition module is a function provided by the first data warehouse in the hybrid data warehouse architecture. In this step 101, the Hadoop-based file acquisition module on the first data warehouse can be used to acquire, decompress, transcode, and decrypt data from multiple data sources, and then store the data in the first data warehouse through the distributed file system.

[0075] Step 102: Load the source data into the second data warehouse in the hybrid data warehouse architecture, process the source data in the second data warehouse, and transmit the processed data to the first data warehouse.

[0076] Then, by creating a target table structure that matches the data structure of the data files in the first data warehouse, the source data is loaded into the second data warehouse.

[0077] Specifically, the source data can be written into a data table of the target table structure and stored in a specified directory. Then, a preset data loading module loads the source data data table from the specified directory into the second data warehouse, thereby enabling the second data warehouse to load the source data in the first data warehouse.

[0078] Taking a Hadoop-MPP hybrid data warehouse architecture as an example, the default data loading module can be the Hadoop-to-MPP data loading module in this architecture. This module writes ORC format Hive files (files stored in Hadoop) into the MPP database. In practical applications, a table structure matching the ORC format file can be created first using a method such as `create table`. Then, on the Hadoop side, the ORC format data file is written into the data table of this table structure, placed in a specific FTP (File Transfer Protocol) directory, and the `loaddata infile` method is executed by the data loading module to load the ORC format file into the MPP database.

[0079] This application pre-designs a data warehouse model within a hybrid architecture. The data warehouse model includes a logical model and a physical model. The logical model describes the data and the relationships between data points, while the physical model describes the data layering process and the database table structure. The data layering in the physical model includes a basic subject layer (integration model layer, M layer), a summary and sharing layer (summary commonality layer, C layer), and an application data layer (metrics layer, E layer).

[0080] The first and second data warehouses maintain their respective data warehouse logical models and data layering structures.

[0081] The first data warehouse has a data layering system comprising a first basic theme layer, a first summary and sharing layer, and a first application data layer. The second data warehouse has a data layering system comprising a second basic theme layer, a second summary and sharing layer, and a second application data layer. The second data warehouse is matched with corresponding data layering processing links for each layer. These processing links are formed by the processing functions at each layer. Specifically, the second basic theme layer is used to classify and integrate the loaded source data according to themes; the second summary and sharing layer is used to extract and summarize the data output from the model layer based on business needs; and the second application data layer is used to perform application-level reorganization and processing on the data output from the summary and sharing layer. The first data warehouse, however, only retains its data layering structure and does not process data within its own data warehouse; therefore, it does not match corresponding processing links to its data layering structure.

[0082] After loading the source data from the first data warehouse to the second data warehouse, the loaded source data can be processed based on the data warehouse logical model and data warehouse physical model provided by the second data warehouse. Specifically, the loaded source data can be processed based on the functions of each data layer in the data layering processing chain provided by the data warehouse physical model. During the processing, the field mappings and database table relationships between different layers in the data layering processing chain are determined based on the database table structure provided by the data warehouse logical model and data warehouse physical model of the second data warehouse.

[0083] Taking a Hadoop-MPP-based hybrid data warehouse architecture as an example, the data warehouse model module can provide the necessary data warehouse model functions for both the first and second data warehouses. In this Hadoop-MPP-based hybrid data warehouse architecture, when designing and developing / building this data warehouse model module, the following steps are taken: first, the logical and physical models of the data warehouse are designed; then, based on the model design results, the mapping relationship between the model and the data source is determined, and the data warehouse table creation and corresponding scripts (such as ETL scripts) are developed. For example, the further model design, table creation, and related script development process includes:

[0084] 21) Data Warehouse Logical Model Design: By collecting and researching OLTP system product information, conducting questionnaires, and carrying out business and technical research interviews, we understand the meaning of the data business transmitted by the OLTP system. Guided by ER (Entity-Relationship) and 3NF (Third Normal Form) modeling theories, and combined with the technical definitions of entities and relationships in the OLTP system by the data dictionary platform, and referring to the model elements (entity name, attributes, value range and rules, data type) defined in the data warehouse summary model, we design the logical model of the data warehouse.

[0085] 22) Data Warehouse Logical Model Design: Clearly define the data layering and processing links. Based on data processing characteristics, divide the data into an integration model layer (M layer), a summary commonality layer (C layer), and a measurement indicator layer (E layer) to form the main data processing links and data layers of the data warehouse. Combining the MPP database data sharding principle, best practices for sharded table replication, data access characteristics, and data business similarity, rationally construct the data warehouse physical model, design the relevant database table structure, and complete the mapping of fields at each layer of the data warehouse processing links from the OLTP system down to the data lake source and the determination of the main table associations. Also, save the relevant information to the ETL development tool to facilitate the ETL code generation engine to generate ETL scripts and batch program scheduling processes.

[0086] 23) Data warehouse table creation and ETL script implementation: Based on the ETL script generation engine, according to the model mapping relationship in the modeling process, the data layer processing design, and the abstracted structure mapping template and algorithm template, generate M / C / E layer ETL scripts and batch scheduling processes, develop complex table processing ETL programs, realize M / C / E layer data processing on the layer processing link, and complete data warehousing and data integration processing.

[0087] Meanwhile, for the construction of the big data scheduling module, the scheduling engine can be used to configure the ETL program flow in the data warehouse (based on the business priority of the warehouse processing table, data processing dependencies, and data processing order), and generate an execution plan according to the program running rules (the execution plan includes data extraction, cleaning, transformation, and loading processes). The data cluster logical instances are divided according to legal entities, and daily batches are scheduled by instance. A date switching and batch node running status management mechanism is established, and the data warehouse ETL workflow is reasonably designed to ensure the correctness of batch scheduling data processing, while meeting the timeliness of upper-level business data, and supporting cross-workflow engine query of batch running status.

[0088] Step 103: Use the first data warehouse to store the processing result data, and provide data services to external parties based on the stored processing result data.

[0089] The second basic theme layer, the second summary and sharing layer, and the second application data layer in the data layering of the second data warehouse are connected to the first basic theme layer, the first summary and sharing layer, and the first application data layer in the data layering of the first data warehouse in a one-to-one relationship.

[0090] When the second data warehouse processes data based on the functions of each layer in its data layering processing chain and generates corresponding processing result data, it downloads the result data of each data layer to the corresponding layer of the first data warehouse through a preset data download module. Specifically, the data download module downloads the processing result data generated by the second basic theme layer of the second data warehouse to the first basic theme layer of the first data warehouse, downloads the processing result data generated by the second summary and sharing layer of the second data warehouse to the first summary and sharing layer of the first data warehouse, and downloads the processing result data generated by the second application data layer of the second data warehouse to the first application data layer of the first data warehouse.

[0091] The first data warehouse stores the processed data it receives in its first basic subject layer, first summary and sharing layer, and first application data layer, respectively, and provides data services to the outside world through the data interface layer in the first data warehouse. This data interface layer is connected to the first basic subject layer, the first summary and sharing layer, and the first application data layer to provide data services based on the data stored in each layer, allowing external systems or users to query on demand.

[0092] In a hybrid data warehouse architecture based on Hadoop-MPP, the aforementioned pre-defined data download module is the MPP-to-Hadoop data download module. Based on this module, the processed data from each data layer in the MPP data warehouse can be downloaded to the corresponding data layer in the Hadoop data warehouse. For example, after the batch processing of the relevant tables in the M / C / E layer of the MPP is completed on the same day, the relevant data can be directly written to HDFS in Hadoop in TXTFile format through the MPP-to-Hadoop data download module.

[0093] See also Figure 2 An exemplary business process based on a hybrid data warehouse architecture using Hadoop-MPP specifically includes:

[0094] 31) For the Hadoop part of the data warehouse, daily batch downloads of files from OLTP source systems are obtained and loaded into the source data layer;

[0095] 32) In the data warehouse MPP part, the Hadoop2MPP data loading module is used as needed to load data files from the Hadoop part of the data warehouse into the MPP, that is, to load the source data layer data in the Hadoop part into the basic topic layer of the MPP.

[0096] 33) The basic theme layer of the data warehouse MPP, based on the data warehouse model design and data warehouse layering design strategy, completes the integration and processing of detailed data warehouse data, and performs layer-by-layer aggregation, integration, processing and standardized calculation of data according to the link from the basic theme layer to the aggregation and sharing layer and then to the application data layer.

[0097] 34) The data warehouse MPP uses the MPP2Hadoop data download module to instantly synchronize the processing results of its basic topic layer, summary and sharing layer, and application data layer to the data warehouse Hadoop.

[0098] 35) The result data of each layer in the data warehouse Hadoop is provided to the outside world through the data interface layer.

[0099] As can be seen, this application proposes a hybrid data warehouse architecture based on a first and a second data warehouse. The first data warehouse is used to carry the data storage and data service functions of the hybrid data warehouse architecture, while the second data warehouse is used to carry the highly complex data computing functions of the hybrid data warehouse architecture. By integrating multiple data warehouses with different performance advantages and giving full play to the advantages of each warehouse system, it effectively solves the problems of high expansion difficulty, slow performance, difficulty in mining diverse data types, strict data compliance requirements, and high security maintenance costs.

[0100] Corresponding to the above-described data processing method, this application also provides a data processing apparatus, the structure of which is as follows: Figure 3 As shown, it includes:

[0101] Data acquisition unit 10 is used to acquire data from the data source using the first data warehouse in the hybrid data warehouse architecture to obtain the source data;

[0102] Data loading unit 20 is used to load source data into the second data warehouse in the hybrid data warehouse architecture;

[0103] Processing unit 30 is used to process the patch source data in the second data warehouse;

[0104] The data transmission unit 40 is used to transmit the processing result data to the first data warehouse;

[0105] The data storage and data service unit 50 is used to store the processing result data in the first data warehouse and provide data services to the outside world based on the stored processing result data.

[0106] The data storage performance and data service performance based on the stored data of the first data warehouse are higher than those of the second data warehouse, respectively, while the computing performance of the first data warehouse is lower than that of the second data warehouse.

[0107] In one embodiment, the data acquisition unit 10 is specifically used to: collect data from the data source using the first data warehouse, parse the collected data, and store the parsed data in a data source pasting manner to obtain source pasting data.

[0108] In one embodiment, the data loading unit 20, when loading source data into the second data warehouse in the hybrid data warehouse architecture, is specifically used for:

[0109] Write the source data into a data table of the target table structure, and store the data table in the specified directory;

[0110] The data table is loaded from the specified directory to the second data warehouse through a preset data loading module; wherein the target table structure is a table structure that matches the data structure of the data file in the first data warehouse.

[0111] In one embodiment, the processing unit 30, when processing the loaded patch source data in the second data warehouse, specifically performs the following functions:

[0112] Based on the data warehouse logical model and data warehouse physical model provided by the second data warehouse, the loaded source data is processed.

[0113] The data warehouse logical model is used to describe the data and the relationships between data, while the data warehouse physical model is used to describe the data layering and processing links and the database table structure.

[0114] In one embodiment, the processing unit 30, when processing the loaded source data based on the data warehouse logical model and data warehouse physical model provided by the second data warehouse, is specifically used to: process the loaded source data based on the functions of each data layer on the data layer processing link provided by the data warehouse physical model.

[0115] During the processing, based on the database table structure provided by the data warehouse logical model and the data warehouse physical model, the field mapping and database table association relationships between different layers in the data layering processing link are determined.

[0116] In one embodiment, each data layer in the data layering processing chain includes: a second basic topic layer, a second summary and sharing layer, and a second application data layer;

[0117] The second basic topic layer is used to classify and integrate the loaded post source data according to the topic;

[0118] The second summary and sharing layer is used to extract, summarize, and integrate the data output by the model layer according to business needs;

[0119] The second application data layer is used to perform application-level reorganization and processing on the data output by the summary sharing layer.

[0120] In one embodiment, the data layering of the first data warehouse includes a first basic theme layer, a first summary and sharing layer, and a first application data layer;

[0121] Data download unit 40, when transmitting the processing result data to the first data warehouse, is specifically used for:

[0122] The processed result data generated by the second basic theme layer of the second data warehouse is transmitted down to the first basic theme layer of the first data warehouse;

[0123] The processed result data generated by the second aggregation and sharing layer of the second data warehouse is transmitted down to the first aggregation and sharing layer of the first data warehouse;

[0124] The processed result data generated by the second application data layer of the second data warehouse is transmitted down to the first application data layer of the first data warehouse.

[0125] In one embodiment, the data storage and data service unit 50 is specifically used for:

[0126] The first basic theme layer, the first summary and sharing layer, and the first application data layer of the first data warehouse respectively store the processed result data received.

[0127] Data services are provided to the outside world through the data interface layer in the first data warehouse that is connected to the first basic theme layer, the first summary and sharing layer and the first application data layer.

[0128] The data processing apparatus provided in this application is described simply because it corresponds to the data processing method provided in the above method embodiments. For any similarities, please refer to the description of the above method embodiments, which will not be detailed here.

[0129] This application also provides a computer-readable medium having a computer program stored thereon, the computer program comprising program code for performing the data processing methods provided in the above method embodiments.

[0130] In the context of this application, a computer-readable medium (machine-readable medium) can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0131] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0132] The aforementioned computer-readable medium may be contained within an electronic device or may exist independently without being assembled into an electronic device.

[0133] This application also provides a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the data processing method provided in the above method embodiments.

[0134] Specifically, according to embodiments of this application, the processes described in the above-described reference flowcharts can be implemented as computer software programs. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments of this application.

[0135] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0136] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0137] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0138] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Data from the data source is obtained using the first data warehouse in the hybrid data warehouse architecture to obtain the source data; the data layering of the first data warehouse includes a first basic theme layer, a first summary and sharing layer, and a first application data layer; the first data warehouse only retains its data layering structure, does not process the data on its own data warehouse side, and does not match the corresponding processing link for its data layering structure. The source data is loaded into the second data warehouse in the hybrid data warehouse architecture. The second data warehouse processes the source data and transmits the processed data to the first data warehouse. The data layers of the second data warehouse include a second basic theme layer, a second summary and sharing layer, and a second application data layer. The second data warehouse is matched with data layer processing links corresponding to each layer. The processing links are links formed by the processing functions on each layer. The second basic theme layer is used to classify and integrate the loaded source data according to themes. The second summary and sharing layer is used to extract and summarize the data output by the model layer according to business needs. The second application data layer is used to perform application-level reorganization and processing on the data output by the summary and sharing layer. The transmission of the processed data to the first data warehouse includes: transmitting the processed data generated by the second basic theme layer of the second data warehouse to the first basic theme layer of the first data warehouse through a preset data download module; transmitting the processed data generated by the second summary and sharing layer of the second data warehouse to the first summary and sharing layer of the first data warehouse; and transmitting the processed data generated by the second application data layer of the second data warehouse to the first application data layer of the first data warehouse. The first basic theme layer, the first summary and sharing layer, and the first application data layer of the first data warehouse respectively store the processed result data received; and provide data services to the outside world through the data interface layer in the first data warehouse that is connected to the first basic theme layer, the first summary and sharing layer, and the first application data layer. The data storage performance and data service performance based on the stored data of the first data warehouse are higher than those of the second data warehouse, respectively. The computing performance of the first data warehouse is lower than that of the second data warehouse. During the operation of the hybrid data warehouse architecture, the first data warehouse is used to carry the data storage and data service functions of the hybrid data warehouse architecture, while the second data warehouse is used to carry the highly complex data computing functions of the hybrid data warehouse architecture for data processing.

2. The method according to claim 1, characterized in that, The process of obtaining source data by utilizing the first data warehouse in the hybrid data warehouse architecture includes: Data is collected from the data source using the first data warehouse, the collected data is parsed, and the parsed data is stored by data pasting to the source to obtain pasted source data.

3. The method according to claim 1, characterized in that, The step of loading the source data into the second data warehouse in the hybrid data warehouse architecture includes: Write the source data into a data table of the target table structure, and store the data table in the specified directory; The data table is loaded from the specified directory to the second data warehouse through a preset data loading module; wherein the target table structure is a table structure that matches the data structure of the data file in the first data warehouse.

4. The method according to claim 1, characterized in that, The processing of the loaded source data by the second data warehouse includes: Based on the data warehouse logical model and data warehouse physical model provided by the second data warehouse, the loaded source data is processed. The data warehouse logical model is used to describe the data and the relationships between data, while the data warehouse physical model is used to describe the data layering and processing links and the database table structure.

5. The method according to claim 4, characterized in that, The process of processing the loaded source data based on the data warehouse logical model and data warehouse physical model provided by the second data warehouse includes: Based on the functions of each data layer in the data layering processing link provided by the data warehouse physical model, the loaded source data is processed. During the processing, based on the database table structure provided by the data warehouse logical model and the data warehouse physical model, the field mapping and database table association relationships between different layers in the data layering processing link are determined.

6. A data processing apparatus, characterized in that, The device includes: The data acquisition unit is used to acquire data from the data source using the first data warehouse in the hybrid data warehouse architecture to obtain the source data. The data layering of the first data warehouse includes a first basic theme layer, a first summary and sharing layer, and a first application data layer. The first data warehouse only retains its data layering structure and does not process the data on its own data warehouse side, nor does it match the corresponding processing link for its data layering structure. A data loading unit is used to load source data into the second data warehouse in the hybrid data warehouse architecture. The data layers of the second data warehouse include a second basic theme layer, a second summary and sharing layer, and a second application data layer. The second data warehouse is matched with data layer processing links corresponding to each layer. The processing links are links formed by the processing functions on each layer. The second basic theme layer is used to classify and integrate the loaded source data according to themes. The second summary and sharing layer is used to extract and summarize the data output by the model layer according to business needs. The second application data layer is used to perform application-level reorganization and processing on the data output by the summary and sharing layer. The processing unit is used to process the source data in the second data warehouse; A data download unit is used to transmit processed result data to the first data warehouse; including: downloading processed result data generated by the second basic theme layer of the second data warehouse to the first basic theme layer of the first data warehouse through a preset data download module; downloading processed result data generated by the second summary and sharing layer of the second data warehouse to the first summary and sharing layer of the first data warehouse; and downloading processed result data generated by the second application data layer of the second data warehouse to the first application data layer of the first data warehouse. The data storage and data service unit is used to store the processed result data received respectively in the first basic theme layer, the first summary and sharing layer and the first application data layer of the first data warehouse; and to provide data services to the outside world through the data interface layer in the first data warehouse that is connected to the first basic theme layer, the first summary and sharing layer and the first application data layer. The data storage performance and data service performance based on the stored data of the first data warehouse are higher than those of the second data warehouse, respectively. The computing performance of the first data warehouse is lower than that of the second data warehouse. During the operation of the hybrid data warehouse architecture, the first data warehouse is used to carry the data storage and data service functions of the hybrid data warehouse architecture, while the second data warehouse is used to carry the highly complex data computing functions of the hybrid data warehouse architecture for data processing.

7. A computer-readable medium, characterized in that, It stores a computer program thereon, the computer program containing program code for performing the method as described in any one of claims 1-5.

8. A computer program product, characterized in that, It includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method as described in any one of claims 1-5.

Images