Configuration data warehouse design method based on data center

Abstract

The invention discloses a configuration data warehouse design method based on a data medium platform. The method comprises the following steps: deploying the data medium platform and a bottom layer big data cluster; acquiring data, and making a data collection scheme; cleaning the data; designing a topic model library, compiling a data integration script, and integrating data into the topic model library; developing an HBase wide table; and designing a data center configuration module. According to the method, an efficient data warehouse system is constructed through a data center configuration management method, flexible access, cleaning and query are achieved, and the method has the advantages of being high in expansibility and adaptability and easy and convenient to operate.

Description

Technical Field

[0001] The present invention relates to the field of data warehouse design, and in particular to a configurable data warehouse design method based on a data middle platform. Background Art

[0002] Since the goal of "big middle platform, small front desk" was proposed in 2015, the concept of "middle platform" has gradually entered the public eye and has been widely used in enterprises. The term "middle platform" refers to a data platform between the front desk and the back desk. Although "middle platform" did not have a clear definition at first, its connotation has become increasingly clear as major companies have begun to build middle platforms. The middle platform is essentially a platform that supports flexible response of front-end business and efficient integration of back-end resources. It provides support for rapid iteration of enterprise business by reducing redundancy and optimizing resource allocation. As an important part of the middle platform system, the "data middle platform" is a platform that can collect, process, calculate and service data. Its core goal is to provide unified and efficient data support for the front desk and back desk to meet the data needs of different scenarios. With the development of big data technology, the data middle platform plays an irreplaceable role in the data governance and intelligent transformation of enterprises.

[0003] In the field of big data, Hadoop, as an open source framework developed by the Apache Foundation, is one of the important underlying technologies supporting the data center. Hadoop achieves efficient management and processing of massive data through the distributed storage system HDFS and the distributed computing framework MapReduce. Hadoop's distributed architecture enables it to be deployed in cheap server clusters, which not only expands the data storage capacity, but also significantly improves the efficiency of data reading and writing. Therefore, Hadoop has been widely used in data processing scenarios in multiple industries. However, Hadoop itself is an underlying framework, and support for actual business scenarios requires further construction of upper-level services and applications.

[0004] As part of the Hadoop ecosystem, HBase is a column-based distributed database system that focuses on large-scale data management. HBase draws on the design concept of Google BigTable and provides the ability to read and write massive amounts of data in real time and randomly. The structure of the HBase table has the characteristics of sparsity and dynamic columns, which enables it to flexibly adapt to different business needs and linearly expand by adding nodes to meet the storage and query of large-scale data. However, compared with traditional relational databases, HBase has a higher threshold for use, complex operations, and higher requirements for the design of data models. Therefore, there are still many challenges in deeply integrating HBase with business needs.

[0005] In the era of big data, data has become the most valuable asset of an organization. Especially in the context of big data, data governance has gradually become a hot topic in research institutions and industries at home and abroad. Data governance is not only related to data quality and security compliance, but also affects the ability of data-driven service innovation. However, in terms of data management, current institutions generally have the phenomenon of "focusing on collection but not management, focusing on scale but not quality, and focusing on utilization but not security", which makes it difficult to give full play to the value of data. At the same time, with the rapid growth of data scale, how to improve data governance efficiency and ensure data security, privacy protection and compliance has also become an urgent problem to be solved in the design of data middle platform and data warehouse.

[0006] In recent years, with the rapid development of data middle-end technology, data warehouses have formed a relatively complete system in terms of structural design and functional implementation. Modern data warehouses achieve full-process management from data collection to analysis services through a layered architecture. However, existing data warehouses still have deficiencies in application flexibility and scalability, especially when supporting complex business needs, requiring a lot of manual intervention to adjust data models and expand service functions. For example, the model design of traditional data warehouses is usually relatively static. When business needs change, the adjustment process often takes a long time and is prone to errors.

[0007] In the context of modernization of governance systems and governance capabilities, how to use data technology to improve the level of urban governance has become a new research hotspot. Smart city solutions represented by the "City Brain" have proposed a new model for empowering urban development through data. As one of the core technologies for data analysis and decision support, the data warehouse is the infrastructure of the "City Brain". However, the functions of existing data warehouses are relatively limited and it is difficult to adapt to the needs of high-concurrency queries, dynamic data integration, and real-time response in the "City Brain" scenario. How to expand the functions of the data warehouse so that it can meet the complex needs of urban governance has become a technical problem that needs to be solved urgently. Summary of the invention

[0008] One purpose of the present invention is to propose a configurable data warehouse design method based on a data middle platform. The present invention introduces the configurable management idea of ​​the data middle platform, combines the efficient storage of HBase wide tables with the flexible operation of relational databases, and constructs a management system that automates the entire process from data access and cleaning to services. It not only improves the flexibility and ease of use of the data warehouse, but also enhances its adaptability in complex business scenarios, providing solid data support for smart governance scenarios such as the "city brain".

[0009] A configuration-based data warehouse design method based on a data middle platform according to an embodiment of the present invention includes the following steps:

[0010] S1. Deploy the data center and the underlying big data cluster. By deploying the data center components and the big data cluster on the server, the basic environment of the data center is built.

[0011] S2. Obtain data, formulate a data collection plan based on the data existence, collect data from various data sources, and store the collected data in the data warehouse ODS source layer;

[0012] S3, clean the data in the ODS source layer of the data warehouse and store it in the DWD standard layer of the data warehouse;

[0013] S4. Design a topic model library based on the data in the DWD standard layer of the data warehouse, write a data integration script, integrate the data into the topic model library according to the designed topic model library, and store it in the DWI integration layer of the data warehouse;

[0014] S5. Create an HBase wide table, design the row key and column family of the HBase wide table, build the HBase wide table in the HBase database of the data center, establish a mapping table in the relational database of the data center, write data insertion scripts and field mapping synchronization scripts, and synchronize data with HBase through the mapping table of the relational database;

[0015] S6. Design the data middle platform configuration module.

[0016] Optionally, the S1 specifically includes:

[0017] S11. Select a deployment environment and install basic software components for the data center according to the server's hardware configuration and operating system type. The basic software components include the container management tool Docker, the distributed resource scheduling framework Kubernetes, and the database management tool.

[0018] S12, installing and configuring a big data cluster, wherein the big data cluster includes a distributed storage system and a distributed computing framework, wherein the distributed storage system uses a distributed file system HDFS, and the distributed computing framework uses an open source computing framework Spark;

[0019] S13, initialize the data middle platform component, the data middle platform component includes a data access module, a metadata management module, a data governance module, a data service module and a configuration module, and the initialization process includes binding configuration of the logical functions and hardware resources of each module;

[0020] S14. Deploy network communication services and configure the communication protocols required by the distributed system, including the RPC protocol and the distributed message queue service Kafka;

[0021] S15. Configure system permissions and security policies, which include allocating access rights to different modules, enabling data encryption transmission protocol TLS, and setting user authentication and permission allocation policies.

[0022] Optionally, the S2 specifically includes:

[0023] S21, obtaining and analyzing data from data sources, and classifying and arranging data information according to the distribution of each data source, data storage type, and access interface;

[0024] S22, configure data access parameters, define connection parameters for each data source, and specify a transmission protocol for data access, wherein the connection parameters include a data source address, authentication information, a data table name, and a file path;

[0025] S23. Formulate a data collection plan, determine the scope of the collected data and the update method according to business requirements and data characteristics. The update method includes full extraction and incremental extraction. The incremental extraction implements data filtering based on the primary key field or timestamp field;

[0026] S24. Execute data collection tasks, use the data access module of the data center to collect data from various data sources, and store the collected data in the data warehouse ODS source layer.

[0027] Optionally, the S4 specifically includes:

[0028] S41. Evaluate the depth and breadth of the data in the DWD standard layer of the data warehouse, analyze the overall situation of the data according to business needs, and design a topic model library. The topic model library divides the data storage structure according to business themes to ensure that the topic library can cover all business aspects;

[0029] S42. According to the design result of the subject model library, define data integration rules, wherein the data integration rules include selection of data source tables, definition of associated fields, aggregation calculation rules, and mapping of target fields;

[0030] S43. Writing a data integration script according to the data integration rules, wherein the data integration script includes logic for extracting data, converting field formats, and executing table association and field mapping;

[0031] S44, execute the data integration task, extract data according to the data integration script, complete field conversion, association calculation and data loading, and store the integrated data in the data warehouse DWI integration layer.

[0032] Optionally, the S5 specifically includes:

[0033] S51. Design an HBase wide table structure, and determine the row key and column family of the HBase wide table according to business requirements. The row key is used as a unique identifier of the data, and the column family is divided according to business topics.

[0034] S52. Create an HBase wide table in the HBase database at the bottom layer of the data center, and configure the table name, column family name, and storage settings of the HBase wide table;

[0035] S53. Create a mapping table in the relational database of the data center, where the fields of the mapping table include the row key field of the wide table and the field name of each column family;

[0036] S54. Write a data insertion script, select a main table, filter out the fields required to build an HBase wide table, left-connect other business tables through the main table, and insert the data into the mapping table of the relational database;

[0037] S55. Design field mapping rules and write field mapping synchronization scripts. Use the field mapping synchronization scripts to complete data synchronization from the relational database to the HBase wide table. The steps of the field mapping synchronization scripts include reading data from the mapping table, generating commands for HBase Put operations according to the field mapping rules, and writing data into the HBase wide table.

[0038] Optionally, the S6 specifically includes:

[0039] S61. Design the front-end interface of the data middle platform configuration module. The front-end interface adopts a drag-and-drop design, corresponding to the column families and fields of the HBase wide table. The front-end elements are dynamically generated. When the HBase wide table increases or decreases the column family, the front-end interface is synchronously updated and displayed.

[0040] S62. Develop the background logic of the data middle platform configuration module, which includes data application, data sharing and data query. The data application records the user's application request for module data, the data sharing supports the permission configuration of data between modules, and the data query generates a query command and returns the query result.

[0041] S63. Design a change monitoring solution for HBase wide tables, push the data and metadata of HBase wide tables to Kafka, and implement monitoring of changes in HBase wide tables by writing Kafka producer programs. When table data changes, encapsulate it into Kafka messages for sending, and write consumer programs to read data from Kafka and write them to the underlying tables, return the data to the front end, and display the description information of the HBase wide table through the front end interface.

[0042] The beneficial effects of the present invention are:

[0043] By introducing the configuration management concept of the data center, the deficiencies of traditional data warehouses in terms of flexibility, scalability and management complexity are effectively solved. Through the configuration design, all aspects of the data warehouse, including data access, cleaning, integration and service provision, have been modularized and automated, significantly reducing the workload of development and maintenance, while improving the system's response speed and adaptability to complex business needs.

[0044] By utilizing the unified management capabilities of data resources in the data center and deploying Hadoop distributed storage and computing architecture, a high-performance data processing environment can be built. Combined with the flexible scalability and real-time storage capabilities of HBase wide tables, the data warehouse can not only efficiently store and manage large-scale data, but also achieve dynamic synchronization and query optimization of complex data through the mapping mechanism of wide table design and relational databases, thereby improving the query performance and data availability of the system. In addition, through the configuration module, the column families and fields of HBase wide tables are dynamically displayed, and users can complete the application, sharing, query and fusion of data without complex technical operations, which greatly simplifies the process of data operation.

[0045] Significant results have also been achieved in data governance. Through the configuration management of data access parameters, cleaning rules and integration logic, the quality consistency and structural standardization of data from source to target are ensured, solving the problem of "focusing on collection and neglecting management" in traditional data governance. At the same time, through the metadata monitoring and update module, real-time capture and front-end display of data structure changes are achieved, which enhances the operability of the system and the traceability of data, and ensures the security and compliance of data management.

[0046] In actual application scenarios, the present invention provides strong data support for the construction of the "city brain" of smart cities. The data warehouse constructed by this method can quickly integrate multi-source heterogeneous data, achieve high-concurrency queries and real-time data analysis, and meet the needs of urban governance for data agility and reliability. The flexibility of the configurable design enables the data warehouse to quickly adapt to the needs of different business scenarios and support multi-dimensional data analysis and decision-making. Overall, the present invention significantly improves the functionality and applicability of the data warehouse, and provides a new technical path for data-driven business innovation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 A flowchart of a configuration data warehouse design method based on a data middle platform proposed by the present invention;

[0049] Figure 2 This is a schematic diagram of the data middle platform configuration module of the configurable data warehouse design method based on the data middle platform proposed in the present invention. DETAILED DESCRIPTION

[0050] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0051] refer to Figure 1 , a configuration data warehouse design method based on data middle platform, including the following steps:

[0052] S1. Deploy the data center and the underlying big data cluster. By deploying the data center components and the big data cluster on the server, the basic environment of the data center is built.

[0053] S2. Obtain data, formulate a data collection plan based on the data existence, collect data from various data sources, and store the collected data in the data warehouse ODS source layer;

[0054] S3, clean the data in the ODS source layer of the data warehouse and store it in the DWD standard layer of the data warehouse;

[0055] S4. Design a topic model library based on the data in the DWD standard layer of the data warehouse, write a data integration script, integrate the data into the topic model library according to the designed topic model library, and store it in the DWI integration layer of the data warehouse;

[0056] S5. Create an HBase wide table, design the row key and column family of the HBase wide table, build the HBase wide table in the HBase database of the data center, establish a mapping table in the relational database of the data center, write data insertion scripts and field mapping synchronization scripts, and synchronize data with HBase through the mapping table of the relational database;

[0057] S6. Design the data middle platform configuration module.

[0058] In this implementation, S1 specifically includes:

[0059] S11. Select a deployment environment and install basic software components for the data center according to the server's hardware configuration and operating system type. The basic software components include the container management tool Docker, the distributed resource scheduling framework Kubernetes, and the database management tool.

[0060] S12, installing and configuring a big data cluster, wherein the big data cluster includes a distributed storage system and a distributed computing framework, wherein the distributed storage system uses a distributed file system HDFS, and the distributed computing framework uses an open source computing framework Spark;

[0061] S13, initialize the data middle platform component, the data middle platform component includes a data access module, a metadata management module, a data governance module, a data service module and a configuration module, and the initialization process includes binding configuration of the logical functions and hardware resources of each module;

[0062] S14. Deploy network communication services and configure the communication protocols required by the distributed system, including the RPC protocol and the distributed message queue service Kafka;

[0063] S15. Configure system permissions and security policies, which include allocating access rights to different modules, enabling data encryption transmission protocol TLS, and setting user authentication and permission allocation policies.

[0064] In this implementation, S2 specifically includes:

[0065] S21, obtaining and analyzing data from data sources, and classifying and arranging data information according to the distribution of each data source, data storage type, and access interface;

[0066] S22, configure data access parameters, define connection parameters for each data source, and specify a transmission protocol for data access, wherein the connection parameters include a data source address, authentication information, a data table name, and a file path;

[0067] S23. Formulate a data collection plan, determine the scope of the collected data and the update method according to business requirements and data characteristics. The update method includes full extraction and incremental extraction. The incremental extraction implements data filtering based on the primary key field or timestamp field;

[0068] S24. Execute data collection tasks, use the data access module of the data center to collect data from various data sources, and store the collected data in the data warehouse ODS source layer.

[0069] In this implementation, S4 specifically includes:

[0070] S41. Evaluate the depth and breadth of the data in the DWD standard layer of the data warehouse, analyze the overall situation of the data according to business needs, and design a topic model library. The topic model library divides the data storage structure according to business themes to ensure that the topic library can cover all business aspects;

[0071] S42. According to the design result of the subject model library, define data integration rules, wherein the data integration rules include selection of data source tables, definition of associated fields, aggregation calculation rules, and mapping of target fields;

[0072] S43. Writing a data integration script according to the data integration rules, wherein the data integration script includes logic for extracting data, converting field formats, and executing table association and field mapping;

[0073] S44, execute the data integration task, extract data according to the data integration script, complete field conversion, association calculation and data loading, and store the integrated data in the data warehouse DWI integration layer.

[0074] In this implementation manner, S5 specifically includes:

[0075] S51. Design an HBase wide table structure, and determine the row key and column family of the HBase wide table according to business requirements. The row key is used as a unique identifier of the data, and the column family is divided according to business topics.

[0076] S52. Create an HBase wide table in the HBase database at the bottom layer of the data center, and configure the table name, column family name, and storage settings of the HBase wide table;

[0077] S53. Create a mapping table in the relational database of the data center, where the fields of the mapping table include the row key field of the wide table and the field name of each column family;

[0078] S54. Write a data insertion script, select a main table, filter out the fields required to build an HBase wide table, left-connect other business tables through the main table, and insert the data into the mapping table of the relational database;

[0079] S55. Design field mapping rules and write field mapping synchronization scripts. Use the field mapping synchronization scripts to complete data synchronization from the relational database to the HBase wide table. The steps of the field mapping synchronization scripts include reading data from the mapping table, generating commands for HBase Put operations according to the field mapping rules, and writing data into the HBase wide table.

[0080] In this implementation manner, S6 specifically includes:

[0081] S61. Design the front-end interface of the data middle platform configuration module. The front-end interface adopts a drag-and-drop design, corresponding to the column families and fields of the HBase wide table. The front-end elements are dynamically generated. When the HBase wide table increases or decreases the column family, the front-end interface is synchronously updated and displayed.

[0082] S62. Develop the background logic of the data middle platform configuration module, which includes data application, data sharing and data query. The data application records the user's application request for module data, the data sharing supports the permission configuration of data between modules, and the data query generates a query command and returns the query result.

[0083] S63. Design a change monitoring solution for HBase wide tables, push the data and metadata of HBase wide tables to Kafka, and implement monitoring of changes in HBase wide tables by writing Kafka producer programs. When table data changes, encapsulate it into Kafka messages for sending, and write consumer programs to read data from Kafka and write them to the underlying tables, return the data to the front end, and display the description information of the HBase wide table through the front end interface.

[0084] Embodiment 1:

[0085] In a certain city's smart city "city brain" construction project, the goal is to improve the level of urban governance through a data-driven approach, achieve multi-department collaboration, efficient resource allocation and real-time decision support. The main challenge facing this project is how to integrate multi-source heterogeneous data to achieve efficient storage, management and utilization of massive urban data. The deficiencies in flexibility and scalability of traditional data warehouses make it difficult for data modeling, updating and service response speeds to meet the real-time needs of urban governance. To this end, the present invention provides a configurable data warehouse design method based on a data middle platform, which has achieved remarkable results in the complex scenarios of smart cities.

[0086] During the implementation of the project, the data center and the underlying big data cluster were deployed on high-performance servers in the municipal data center. The big data cluster consists of 100 node servers, each equipped with a 64-core CPU, 512GB memory and 50TB storage space. The underlying HDFS distributed file system is used for storage, and Spark is used as the computing framework, which significantly improves the large-scale data processing capabilities. The data center is modularly deployed through the containerization tool Docker, including data access module, metadata management module, data governance module, data service module and configuration module, realizing the full process management of the data warehouse from access to service.

[0087] Table 1 Performance comparison between traditional method and the method of the present invention

[0088]

[0089] In the data collection stage, the traditional method takes 120 hours to complete the collection of 300TB of data, with a collection efficiency of 2TB / hour. The present invention achieves a processing efficiency of 6TB / hour through the optimization of the data middle platform access module, shortening the total time to 48 hours, and improving the efficiency by 200%. In addition, the traditional method supports a peak user number of 500 in a high-concurrency collection scenario, while the present invention supports 1,500 concurrent users in the same environment. This shows that the present invention not only improves efficiency in the data collection process, but also greatly improves the stability and expansion capabilities of the system, laying a solid foundation for subsequent data cleaning and integration.

[0090] The performance of the present invention is particularly outstanding in the data cleaning stage. The cleaning efficiency of the traditional method is 1TB / hour, and the total time is 200 hours. The present invention improves the cleaning efficiency to 4.5TB / hour through configurable cleaning rules and automated processing scripts, and the total time is shortened to 60 hours, which improves the efficiency by 350%. At the same time, the error rate in the data cleaning process is significantly reduced. The error rate of the traditional method is 3.0%, while the present invention reduces the error rate to 0.8% through automated detection and repair mechanisms, a reduction of 73%. This improvement is due to the innovative application of the present invention in cleaning rule definition, distributed task decomposition and parallel processing.

[0091] The data processing efficiency in the data integration stage is also greatly improved. The integration efficiency of the traditional method is 1.5TB / hour, and the total time consumption is 150 hours. The present invention achieves an integration efficiency of 6TB / hour through the efficient design of HBase wide tables and the automated integration script of the data center, and the total time consumption is shortened to 50 hours, with an efficiency improvement of 400%. In addition, the error rate of the traditional method during the integration process is 4.0%, while the present invention reduces the error rate to 1.0% through dynamic field mapping rules and automatic verification mechanisms, a reduction of 75%. This efficient integration capability makes the present invention more stable and reliable in the core data processing of the data warehouse.

[0092] In the data query stage, the present invention shows extremely high performance advantages. The query efficiency of the traditional method is 1000 items / second, and each query takes about 5 seconds. The present invention uses the efficient storage of HBase wide tables and the configurable query module to increase the query efficiency to 5000 items / second, shorten the query time to 2 seconds, and increase the efficiency by 400%. This performance improvement can not only meet the needs of high-frequency query scenarios, but also significantly reduce the resource occupancy rate of the query. At the same time, in terms of the accuracy of the query results, the error rate of the traditional method is 2.0%, while that of the present invention is only 0.2%, showing the stability and reliability of the system.

[0093] The present invention improves efficiency and accuracy in the entire process of data collection, cleaning, integration and query through a configurable data middle-end management method. Experimental data show that the present invention supports more users in high-concurrency scenarios, while effectively reducing the error rate in the data processing process, and provides an efficient and reliable technical solution for large-scale data processing. This technological innovation provides support for smart cities, enterprise data governance, and other complex data application scenarios, verifying that the present invention has practical application value.

[0094] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A configuration data warehouse design method based on data middle platform, characterized in that: The steps include: S1. Deploy the data center and the underlying big data cluster. By deploying the data center components and the big data cluster on the server, the basic environment of the data center is built. S2. Obtain data, formulate a data collection plan based on the data existence, collect data from various data sources, and store the collected data in the data warehouse ODS source layer; S3, clean the data in the ODS source layer of the data warehouse and store it in the DWD standard layer of the data warehouse; S4. Design a topic model library based on the data in the DWD standard layer of the data warehouse, write a data integration script, integrate the data into the topic model library according to the designed topic model library, and store it in the DWI integration layer of the data warehouse; S5. Create an HBase wide table, design the row key and column family of the HBase wide table, build the HBase wide table in the HBase database of the data center, establish a mapping table in the relational database of the data center, write data insertion scripts and field mapping synchronization scripts, and synchronize data with HBase through the mapping table of the relational database; S6. Design the data middle platform configuration module.

2. A configuration-based data warehouse design method based on data middle platform according to claim 1, characterized in that: The S1 specifically includes: S11. Select a deployment environment and install basic software components for the data center according to the server's hardware configuration and operating system type. The basic software components include the container management tool Docker, the distributed resource scheduling framework Kubernetes, and the database management tool. S12, installing and configuring a big data cluster, wherein the big data cluster includes a distributed storage system and a distributed computing framework, wherein the distributed storage system uses a distributed file system HDFS, and the distributed computing framework uses an open source computing framework Spark; S13, initialize the data middle platform component, the data middle platform component includes a data access module, a metadata management module, a data governance module, a data service module and a configuration module, and the initialization process includes binding configuration of the logical functions and hardware resources of each module; S14. Deploy network communication services and configure the communication protocols required by the distributed system, including the RPC protocol and the distributed message queue service Kafka; S15. Configure system permissions and security policies, which include allocating access rights to different modules, enabling data encryption transmission protocol TLS, and setting user authentication and permission allocation policies.

3. A configuration-based data warehouse design method based on data middle platform according to claim 1, characterized in that: The S2 specifically includes: S21, obtaining and analyzing data from data sources, and classifying and arranging data information according to the distribution of each data source, data storage type, and access interface; S22, configure data access parameters, define connection parameters for each data source, and specify a transmission protocol for data access, wherein the connection parameters include a data source address, authentication information, a data table name, and a file path; S23. Formulate a data collection plan, determine the scope of the collected data and the update method according to business requirements and data characteristics. The update method includes full extraction and incremental extraction. The incremental extraction implements data filtering based on the primary key field or timestamp field; S24. Execute data collection tasks, use the data access module of the data center to collect data from various data sources, and store the collected data in the data warehouse ODS source layer.

4. A configuration-based data warehouse design method based on data middle platform according to claim 1, characterized in that: The S4 specifically includes: S41. Evaluate the depth and breadth of the data in the DWD standard layer of the data warehouse, analyze the overall situation of the data according to business needs, and design a topic model library. The topic model library divides the data storage structure according to business themes to ensure that the topic library can cover all business aspects; S42. According to the design result of the subject model library, define data integration rules, wherein the data integration rules include selection of data source tables, definition of associated fields, aggregation calculation rules, and mapping of target fields; S43. Writing a data integration script according to the data integration rules, wherein the data integration script includes logic for extracting data, converting field formats, and executing table association and field mapping; S44, execute the data integration task, extract data according to the data integration script, complete field conversion, association calculation and data loading, and store the integrated data in the data warehouse DWI integration layer.

5. A configuration-based data warehouse design method based on data middle platform according to claim 1, characterized in that: The S5 specifically includes: S51. Design an HBase wide table structure, and determine the row key and column family of the HBase wide table according to business requirements. The row key is used as a unique identifier of the data, and the column family is divided according to business topics. S52. Create an HBase wide table in the HBase database at the bottom layer of the data center, and configure the table name, column family name, and storage settings of the HBase wide table; S53. Create a mapping table in the relational database of the data center, where the fields of the mapping table include the row key field of the wide table and the field name of each column family; S54. Write a data insertion script, select a main table, filter out the fields required to build an HBase wide table, left-connect other business tables through the main table, and insert the data into the mapping table of the relational database; S55. Design field mapping rules and write field mapping synchronization scripts. Use the field mapping synchronization scripts to complete data synchronization from the relational database to the HBase wide table. The steps of the field mapping synchronization scripts include reading data from the mapping table, generating commands for HBase Put operations according to the field mapping rules, and writing data into the HBase wide table.

6. A configuration-based data warehouse design method based on data middle platform according to claim 1, characterized in that: The S6 specifically includes: S61. Design the front-end interface of the data middle platform configuration module. The front-end interface adopts a drag-and-drop design, corresponding to the column families and fields of the HBase wide table. The front-end elements are dynamically generated. When the HBase wide table increases or decreases the column family, the front-end interface is synchronously updated and displayed. S62. Design the background logic of the data middle platform configuration module, which includes data application, data sharing and data query. The data application records the user's application request for module data, the data sharing supports the permission configuration of data between modules, and the data query generates a query command and returns the query result. S63. Design a change monitoring solution for HBase wide tables, push the data and metadata of HBase wide tables to Kafka, and implement monitoring of changes in HBase wide tables by writing Kafka producer programs. When table data changes, encapsulate it into Kafka messages for sending, and write consumer programs to read data from Kafka and write them to the underlying tables, return the data to the front end, and display the description information of the HBase wide table through the front end interface.

Images