A data processing system and method based on a multi-layer architecture

Abstract

This invention discloses a multi-layered data processing system and method, relating to the field of big data processing and analysis technology. The system includes a data processing platform communicatively connected to a detailed data layer, a summary layer, an application domain data mart layer, a report data mart layer, a data engine layer, and a data service layer. Built on MaxCompute, the data processing platform receives, aggregates, processes, and analyzes raw data. By introducing an automated ETL processing component, this invention replaces traditional manual data extraction and processing workflows. The system can automatically complete data scheduling, cleaning, transformation, and loading operations, significantly improving data processing efficiency, reducing the frequency of manual intervention, enhancing system stability and automation capabilities, effectively reducing human error, and improving the accuracy and consistency of data processing, thus saving enterprises significant time and labor costs.

Description

Technical Field

[0001] This invention relates to the field of big data processing and analysis technology, specifically to a data processing system and method based on a multi-layer architecture. Background Technology

[0002] As enterprises continue to expand and their businesses diversify, data processing systems need to handle an increasing number of functional modules. Moreover, modern enterprise business processes are often very complex, involving collaborative work between multiple stages. Each stage generates a large amount of data, which needs to be shared and transferred between different stages. A multi-layered data processing architecture can better adapt to such complex business processes. By centrally managing and processing business rules through the business logic layer, the correct transmission and processing of data between different stages can be ensured.

[0003] In existing technologies, data processing systems are slow when dealing with massive amounts of data, failing to meet the needs of real-time analysis. Moreover, the data services provided by these systems are limited in form, unable to meet the diverse needs of different business scenarios, leading to delays in business decisions and impacting enterprise competitiveness. Therefore, the problem to be solved by this invention is how to combine a multi-layered architecture and a high-performance analysis engine to support high-concurrency, low-latency data analysis tasks, and how to divide data resources into three categories: report marts, dashboard marts, and API marts, providing diverse data service functions including report services, dashboard display services, and API interface services. To this end, a data processing system and method based on a multi-layered architecture are proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a data processing system and method based on a multi-layer architecture to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, a data processing system based on a multi-layer architecture includes a data processing platform, which is communicatively connected to a detailed data layer, a summary layer, an application domain mart layer, a report mart layer, a data engine layer, and a data service layer.

[0007] The data processing platform, built on MaxCompute, is divided into multiple logical processing layers for receiving, aggregating, processing, and analyzing raw data to form structured and standardized data assets. This enables unified data management and efficient processing, improves data quality, provides a reliable data foundation for upper-layer applications, and introduces an ETL automated processing component to replace traditional manual data extraction and processing workflows. This component automatically completes the scheduling, cleaning, transformation, and loading operations of raw data, significantly improving data processing efficiency, reducing the frequency of manual intervention, enhancing system stability and automation capabilities, reducing human error, and improving the accuracy and consistency of data processing.

[0008] The Detailed Data Layer (DWD) is used to receive and aggregate raw detail data, and perform atomicity, integrity and normalization processing on the data to ensure the originality and integrity of the data;

[0009] The summary layer (DWS) extracts key indicators and information items based on the detailed data of the detailed data layer to form a unified data standard;

[0010] The Application Domain Marketplace (ADM) layer is used to aggregate data from the aggregation layer by topic, forming a dedicated dataset for specific application scenarios.

[0011] The report mart layer is used to divide data resources into three categories: report mart, cockpit mart, and interface mart, and to calculate the mart matching coefficient to analyze the adaptability of data resources with different types of marts, so as to meet the data service needs under different business scenarios.

[0012] The data engine layer includes a high-performance analysis engine built on HOLOGRES and STARROCKS, which respectively connects to the data visualization and reporting tools of SMART BI and FINE REPORT, supports high-concurrency and low-latency data analysis tasks, and realizes unified access, processing and sharing of structured and semi-structured data.

[0013] The data service layer is used to provide data service functions including reporting services, cockpit display services, and API interface services, realizing end-to-end closed-loop management of data from collection, processing, modeling to service consumption.

[0014] A further improvement to the technical solution of the present invention is that the detailed data layer specifically includes:

[0015] The detailed data layer receives raw detailed data from different business systems through various data source interfaces, performs preliminary data aggregation, and brings together the scattered data into a centralized data warehouse.

[0016] After receiving and aggregating the data, the detailed data layer performs atomicity and integrity verification on the data;

[0017] After receiving, aggregating, and checking the atomicity and integrity of the data, the detailed data layer performs data standardization processing. This standardization process includes data format unification, data cleaning, and data coding standardization. The data is then classified and indexed according to its business attributes and time dimension, thereby transforming the raw data into structured and standardized data assets.

[0018] A further improvement to the technical solution of the present invention is that the summarizing layer specifically includes:

[0019] The aggregation layer extracts normalized data from the detailed data layer and performs aggregation and cleaning operations on the data. The aggregation operation includes performing aggregation calculations on the data, aggregating data by time, business type, and region, and generating data views. The cleaning operation includes removing duplicate data, correcting outliers, and filling in missing values ​​to ensure the accuracy and consistency of the data, reduce data redundancy, and optimize the data structure.

[0020] After data aggregation and cleaning, the aggregation layer performs key indicator extraction and standardization.

[0021] After completing data aggregation, key indicator extraction and standardization, the aggregation layer organizes the processed data into a structured data model and stores it in a designated data storage area. The data model is designed based on business needs and analysis objectives, and organizes the data according to themes or business dimensions.

[0022] A further improvement to the technical solution of this invention lies in that: the application domain marketplace layer specifically includes:

[0023] The application domain marketplace layer obtains data that has been aggregated and standardized from the aggregation layer, and performs thematic aggregation on the data according to different business themes. Then, the data is classified and organized according to business needs to form a dedicated dataset for specific application scenarios.

[0024] After completing the topic aggregation, the application domain marketplace layer further processes and organizes the data according to business needs, generating indicators and datasets that meet the needs of specific business scenarios.

[0025] After completing thematic aggregation and business requirement customization, the generated dedicated datasets for specific application scenarios will be organized in a structured manner and stored in the data warehouse.

[0026] A further improvement to the technical solution of this invention lies in that: the report collection mart layer specifically includes:

[0027] Customized datasets are obtained from the application domain marketplace layer. Data resources are divided into three categories: report marketplace, dashboard marketplace, and interface marketplace. Feature analysis is performed on the data resources to extract matching features for data classification, namely data complexity, data granularity, data sharing requirements, user interaction requirements, and data update requirements. Then, values ​​are assigned to each matching feature, and the baseline values ​​of each matching feature belonging to the three marketplace categories are determined.

[0028] Based on the matching characteristics of data resources and business needs, calculate the mart matching coefficient of each data resource, evaluate the adaptability of data resources to different types of marts, and then accurately classify each data resource into the corresponding mart type.

[0029] After completing the data classification and mart matching coefficient calculation, we provide diversified data services based on the classification and matching results of data resources. At the same time, we continuously optimize the form and content of data services based on user feedback and actual usage to ensure the efficiency of data services and user satisfaction.

[0030] A further improvement to the technical solution of this invention lies in that: the assignment of the matching feature and the baseline values ​​of the three types of marketplaces specifically include:

[0031] For data complexity, the number of fields is used for quantification. Data resources with fewer than 10 fields are assigned a value of 1, data resources with 10-50 fields are assigned a value of 2, and data resources with more than 50 fields are assigned a value of 3. For report marts, the baseline value of data complexity is set to 3, for cockpit marts, the baseline value of data complexity is set to 1, and for interface marts, the baseline value of data complexity is set to 2.

[0032] For data granularity, the degree of data aggregation is used for quantification. Detailed data is assigned a value of 1, lightly aggregated data is assigned a value of 2, and highly aggregated data is assigned a value of 3. For report marts, the baseline value of data granularity is set to 1. For cockpit marts, the baseline value of data granularity is set to 3. For interface marts, the baseline value of data granularity is set to 2.

[0033] For data sharing requirements, the sharing frequency is used for quantification. No sharing requirement is assigned a value of 1, low sharing requirement is assigned a value of 2, and high sharing requirement is assigned a value of 3. For report marts, the baseline value of data sharing requirement is set to 1. For cockpit marts, the baseline value of data sharing requirement is set to 1. For interface marts, the baseline value of data sharing requirement can be set to 3.

[0034] For user interaction needs, interaction frequency is used for quantification. Low interaction needs are assigned a value of 1, medium interaction needs are assigned a value of 2, and high interaction needs are assigned a value of 3. For the report mart, the baseline value of user interaction needs is set to 3. For the cockpit mart, the baseline value of user interaction needs is set to 2. For the interface mart, the baseline value of user interaction needs is set to 1.

[0035] For data update requirements, the real-time requirements of the update are quantified. Real-time update requirements are assigned a value of 3, daily update requirements are assigned a value of 2, and periodic updates are assigned a value of 1. For report marts, the baseline value of data update requirements is set to 1. For dashboard marts, the baseline value of data update requirements can be set to 3. For interface marts, the baseline value of data update requirements is set to 2.

[0036] A further improvement to the technical solution of this invention lies in the following: the calculation process of the marketplace matching coefficient is as follows:

[0037] Determine the quantified value of each data resource on five matching features, including data complexity, data granularity, data sharing requirements, user interaction requirements, and data update requirements. The quantified value is assigned according to preset rules, and the baseline value of each matching feature for the three types of marketplaces is determined. The baseline value is set according to the business needs and characteristics of the marketplace.

[0038] Based on the importance of matching features in mart matching, the weight of each matching feature is determined, where the weight of data complexity is 0.20, the weight of data granularity is 0.25, the weight of data sharing requirements is 0.10, the weight of user interaction requirements is 0.15, and the weight of data update requirements is 0.3.

[0039] For each data resource and each marketplace, calculate the deviation term for each matching feature, sum the deviation terms of all matching features, and take the reciprocal to obtain the marketplace matching coefficient.

[0040] For each data resource, calculate its matching coefficient with the three types of marketplaces, compare the three marketplace matching coefficients, and allocate the data resource to the marketplace with the highest matching coefficient.

[0041] A further improvement to the technical solution of the present invention is that the data engine layer specifically includes:

[0042] A high-performance analysis engine is built based on HOLOGRES and STARROCKS, and it interfaces with data visualization and reporting tools such as SMART BI and FINE REPORT. At the same time, it provides unified access to structured and semi-structured data.

[0043] After data is received, the computing power of HOLOGRES and STARROCKS is used to process the data. Optimized algorithms are used for different types of data to quickly complete data cleaning, aggregation and correlation operations. It supports high-concurrency and low-latency data analysis tasks and can respond to query requests in real time.

[0044] After data processing and analysis are completed, the results are shared via an interface, enabling SMART BI and FINEREPORT to obtain the necessary data and present users with intuitive visual reports and analysis results.

[0045] A further improvement to the technical solution of the present invention is that the data service layer specifically includes:

[0046] It integrates multiple data service functions, including reporting services, cockpit display services, and API interface services;

[0047] After the functional integration is completed, the data service layer provides corresponding data services based on user needs and business scenarios. The reporting service provides users with detailed data analysis through automatic generation and customization of reports. The dashboard display service displays key business indicators in real time through intuitive dashboards and charts. The API interface service provides data to other systems and applications through standardized interfaces.

[0048] The data service layer provides unified management of the relevant data services, ensuring their stability and reliability, and continuously optimizing the form and content of the data services based on user feedback and actual usage.

[0049] Secondly, a data processing method based on a multi-layer architecture, implemented using a multi-layer architecture data processing system, includes the following steps:

[0050] S1. Receive and aggregate raw detailed data from different systems through the detailed data layer, and perform atomicity, integrity and normalization processing;

[0051] S2. Based on the detailed data layer data, perform data aggregation and key indicator extraction to reduce redundancy and provide a concise data view for upper-layer applications.

[0052] S3. Utilize the application domain marketplace layer to perform thematic aggregation on the aggregated data, forming datasets tailored to specific application scenarios to meet the customized needs of different business scenarios.

[0053] S4. Perform feature analysis on the data resources of the application domain mart layer through the report mart layer, calculate the mart matching coefficient, and divide the data resources into three categories: report mart, cockpit mart and interface mart.

[0054] S5 utilizes a high-performance analytics engine to process data, supporting high-concurrency, low-latency analysis tasks, and provides reports, dashboard displays, and API interface services through data visualization tools.

[0055] Due to the adoption of the above technical solution, the technical progress achieved by this invention compared to the prior art is as follows:

[0056] 1. This invention provides a data processing system and method based on a multi-layer architecture. By introducing an ETL automated processing component to replace the traditional manual data extraction and processing process, the system can automatically complete data scheduling, cleaning, transformation and loading operations, which significantly improves data processing efficiency, reduces the frequency of manual intervention, enhances the stability and automation capabilities of the system, effectively reduces human error, improves the accuracy and consistency of data processing, and saves enterprises a lot of time and labor costs.

[0057] 2. This invention provides a data processing system and method based on a multi-layer architecture, which divides data resources into three categories: report mart, dashboard mart, and interface mart. The report mart layer provides diversified data service functions, including report services, dashboard display services, and API interface services, to meet the data service needs of different business scenarios. This allows users to choose the appropriate data service form according to their actual needs, improves the relevance and effectiveness of data services, and accelerates the business decision-making process.

[0058] 3. This invention provides a data processing system and method based on a multi-layer architecture, introducing a high-performance analysis engine built on HOLOGRES and STARROCKS, supporting high-concurrency and low-latency data analysis tasks, enabling unified access, processing and sharing of structured and semi-structured data, significantly improving data processing speed, meeting the needs of real-time analysis, enabling enterprises to obtain data insights faster, make timely business decisions, and improve enterprise competitiveness.

[0059] 4. This invention provides a data processing system and method based on a multi-layer architecture. By using a mart matching coefficient, data resources can be accurately divided into corresponding mart types according to the characteristics of data resources and business needs, ensuring that data resources can be classified and managed for different business scenarios, thereby improving the utilization rate and management efficiency of data resources. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0061] Figure 1 This is a schematic diagram of the system functional modules of the present invention;

[0062] Figure 2 This is a schematic diagram of the system functions of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Example 1, as Figure 1 , Figure 2 As shown, the present invention provides a data processing system based on a multi-layer architecture, including a data processing platform. The data processing platform is communicatively connected to a detailed data layer, a summary layer, an application domain mart layer, a report mart layer, a data engine layer, and a data service layer. Among them, the detailed data layer (DWD), the summary layer (DWS), and the application domain mart layer (ADM) are the data processing layers.

[0065] The data processing platform, built on MaxCompute, is divided into multiple logical processing layers. It receives, aggregates, processes, and analyzes raw data to form structured and standardized data assets, achieving unified data management and efficient processing, improving data quality, and providing a reliable data foundation for upper-layer applications. It introduces ETL automated processing components to replace traditional manual data extraction and processing workflows, automatically completing the scheduling, cleaning, transformation, and loading operations of raw data. This significantly improves data processing efficiency, reduces the frequency of manual intervention, enhances system stability and automation capabilities, reduces human error, and improves the accuracy and consistency of data processing. The ETL automated processing components specifically include:

[0066] After the ETL automated processing component starts, it schedules and extracts raw data. Based on a preset schedule or triggering conditions, it automatically generates a data extraction task list, extracts data from various data sources (databases, file systems, APIs, etc.), and records the timestamps and data volume information of the extracted data. After the data extraction is completed, the ETL component enters the data cleaning and transformation stage, automatically cleaning the extracted raw data, removing duplicate records, correcting erroneous data, filling in missing values, etc., and transforming the cleaned data according to predefined rules, including format unification, data type conversion, field mapping, etc. Through automated processing, human error is reduced, the accuracy and consistency of data processing are improved, and data quality is ensured. The ETL component loads the cleaned and transformed data into the target data warehouse, and after loading is completed, it generates detailed logs and reports, recording the data loading time, data volume, success or failure records. At the same time, it monitors the entire data processing process in real time, detects and handles abnormal situations, and ensures the stable operation of the system.

[0067] The Detailed Data Layer (DWD) is used to receive and aggregate raw detailed data, performing atomicity, integrity, and normalization processing to ensure the originality and integrity of the data. The DWD receives raw detailed data from different business systems (ERP, CRM, log systems, etc.) through various data source interfaces and performs initial data aggregation, bringing the scattered data into a centralized data warehouse. After receiving and aggregating the data, the DWD performs atomicity and integrity verification. Atomicity checks ensure that the smallest unit of data conforms to business rules, while integrity checks ensure the integrity and accuracy of data records. If data is missing or abnormal, it is marked as problematic data and repaired. After completing data reception, aggregation, and atomicity and integrity checks, the DWD performs data normalization processing. This normalization includes data format unification, data cleaning (removing duplicate data and correcting erroneous data), and data coding standardization. The data is also classified and indexed according to its business attributes and time dimension, transforming the raw data into structured and normalized data assets.

[0068] The Data Summarization Layer (DWS), based on the detailed data from the detailed data layer, extracts key indicators and information items to form a unified data standard. Through data summarization and key indicator extraction, it reduces data redundancy, improves data utilization, and provides a concise and clear data view for upper-layer applications. The DWS extracts standardized data from the detailed data layer and performs summarization and cleaning operations. Summarization operations include data aggregation calculations, summarizing data by time, business type, and region to generate a data view. Cleaning operations include removing duplicate data, correcting outliers, and filling in missing values ​​to ensure data accuracy and consistency, reduce data redundancy, and optimize the data structure. After data summarization and cleaning, the DWS... The process involves extracting and standardizing key performance indicators (KPIs). KPI extraction refers to identifying and extracting indicators that are crucial to business decisions based on business needs. KPIs are calculated from multiple data fields. Standardization ensures that the calculation methods and definitions of KPIs remain consistent throughout the system. By extracting and standardizing KPIs, a unified and clear data standard is provided to upper-layer applications, enabling business analysis and decision-making to be based on consistent data standards. After data aggregation, KPI extraction, and standardization are completed, the aggregation layer organizes the processed data into a structured data model and stores it in a designated data storage area. The data model is designed based on business needs and analysis objectives, organizing the data according to themes or business dimensions.

[0069] The Application Domain Mart (ADM) layer is used to perform thematic aggregation on the data from the aggregation layer, forming a dedicated dataset for specific application scenarios. Based on different business needs, it further processes and organizes the data, providing customized data support for different business scenarios, meeting diverse needs under different business scenarios, improving the relevance and effectiveness of data services, and accelerating the business decision-making process. The ADM layer obtains aggregated and standardized data from the aggregation layer, and performs thematic aggregation on the data according to different business themes. It then classifies and organizes the data according to business needs, forming a dedicated dataset for specific application scenarios. After completing thematic aggregation, the ADM layer further processes and organizes the data according to business needs, generating indicators and datasets that meet the needs of specific business scenarios. This not only improves the relevance of the data but also enhances its ability to support business decisions, enabling the data to more directly serve business objectives. After completing thematic aggregation and customized processing for business needs, the generated dedicated dataset for specific application scenarios is organized in a structured manner and stored in the data warehouse.

[0070] The report mart layer is used to divide data resources into three categories: report mart, dashboard mart, and API mart. It calculates the mart matching coefficient, analyzes the compatibility of data resources with different types of marts, meets the data service needs of different business scenarios, and provides comprehensive data services, including report services, dashboard display services, and API interface services, through diversified data service forms to meet the needs of different user groups.

[0071] The data engine layer includes a high-performance analysis engine built on HOLOGRES and STARROCKS, which respectively connects to SMART BI and FINE REPORT data visualization and reporting tools. It supports high-concurrency, low-latency data analysis tasks, realizes unified access, processing and sharing of structured and semi-structured data, significantly improves data processing speed, meets the needs of real-time analysis, and improves the efficiency and accuracy of data analysis.

[0072] The data service layer provides data service functions including reporting services, dashboard display services, and API interface services, realizing end-to-end closed-loop management of data from collection, processing, modeling to service consumption, significantly improving the utilization efficiency of data assets and the level of enterprise-level data governance.

[0073] Example 2, as Figure 1 , Figure 2 As shown, based on Embodiment 1, the present invention provides a technical solution: preferably, the report collection mart layer specifically includes:

[0074] Customized datasets are obtained from the application domain mart layer, and data resources are categorized into three types: report marts, dashboard marts, and API marts. Feature analysis is performed on these data resources to extract matching features for data classification, namely data complexity, data granularity, data sharing needs, user interaction needs, and data update needs. Values ​​are then assigned to each matching feature, and baseline values ​​for each feature belonging to the three mart types are determined. Specifically, the report mart provides detailed report data, the dashboard mart provides overview-level key performance indicator data, and the API mart provides API interface data for interaction with other systems. This ensures that data resources can be categorized and managed according to different business scenarios, based on the matching features of the data resources and the business requirements. The system calculates the matching coefficient of each data resource in the marketplace, assesses the compatibility of data resources with different types of marketplaces, and then accurately classifies each data resource into the corresponding marketplace type. After completing the data classification and marketplace matching coefficient calculation, it provides diversified data services based on the classification and matching results of the data resources. Among them, the report marketplace provides detailed report services to meet users' needs for historical data and detailed information; the dashboard marketplace provides real-time dashboard display services to quickly present key indicators and business overviews; and the interface marketplace provides API interface services to facilitate integration with other systems and data sharing. At the same time, based on user feedback and actual usage, the system continuously optimizes the form and content of data services to ensure the efficiency of data services and user satisfaction.

[0075] The assignment of matching features and the baseline values ​​for the three types of marketplaces specifically include:

[0076] For data complexity, the number of fields is used for quantification. Data resources with fewer than 10 fields are assigned a value of 1, data resources with 10-50 fields are assigned a value of 2, and data resources with more than 50 fields are assigned a value of 3. For report marts, the baseline value for data complexity is set to 3 (high complexity), for dashboard marts, the baseline value for data complexity is set to 1 (simple data table), and for interface marts, the baseline value for data complexity is set to 2 (medium complexity). For data granularity, the degree of data aggregation is used for quantification. Detailed data is assigned a value of 1, indicating light aggregation. Aggregated data (daily summary) is assigned a value of 2, highly aggregated data (monthly summary) is assigned a value of 3. For report marts, the baseline value for data granularity is set to 1 (detailed data); for cockpit marts, the baseline value for data granularity is set to 3 (highly aggregated data); for interface marts, the baseline value for data granularity is set to 2 (lightly aggregated data). For data sharing requirements, sharing frequency is used for quantification: no sharing requirement is assigned a value of 1, low sharing requirement (occasional sharing) is assigned a value of 2, and high sharing requirement (frequent sharing) is assigned a value of 3. For report marts, data sharing... The baseline value for data sharing requirements is set to 1 (no sharing requirement). For the Cockpit Mart, the baseline value for data sharing requirements is set to 1 (no sharing requirement). For the Interface Mart, the baseline value for data sharing requirements can be set to 3 (high sharing requirement). For user interaction requirements, interaction frequency is used for quantification. Low interaction requirements (occasional queries) are assigned a value of 1, medium interaction requirements (periodic queries) are assigned a value of 2, and high interaction requirements (frequent queries and filtering) are assigned a value of 3. For the Report Mart, the baseline value for user interaction requirements is set to 3 (high interaction requirement). For the Cockpit Mart, the baseline value for user interaction requirements is set to 2 (medium interaction requirement). For the Interface Mart, the baseline value for user interaction requirements is set to 1 (low interaction requirement). For data update requirements, real-time update requirements are used for quantification. Real-time update requirements are assigned a value of 3, daily update requirements are assigned a value of 2, and periodic updates are assigned a value of 1. For the Report Mart, the baseline value for data update requirements is set to 1 (periodic update). For the Cockpit Mart, the baseline value for data update requirements can be set to 3 (real-time update). For the Interface Mart, the baseline value for data update requirements is set to 2 (daily update).

[0077] Furthermore, the calculation process for the market matching coefficient is as follows:

[0078] The process involves determining quantified values ​​for each data resource across five matching features: data complexity, data granularity, data sharing requirements, user interaction requirements, and data update requirements. These quantified values ​​are assigned according to predefined rules. Baseline values ​​for each matching feature are then determined for the three types of marketplaces, set based on their business needs and characteristics. The weight of each matching feature is determined based on its importance in marketplace matching: data complexity has a weight of 0.20, data granularity 0.25, data sharing requirements 0.10, user interaction requirements 0.15, and data update requirements 0.3. For each data resource and each marketplace, the deviation term for each matching feature is calculated, and the sum of all deviation terms is taken as the reciprocal to obtain the marketplace matching coefficient. For each data resource, its marketplace matching coefficient with the three types of marketplaces is calculated. The three marketplace matching coefficients are compared, and the data resource is assigned to the marketplace with the highest matching coefficient.

[0079] The expression for the marketplace matching coefficient is:

[0080]

[0081] In the formula, M ij V represents the market matching coefficient, indicating the market matching coefficient between the i-th data resource and the j-th market. The closer the market matching coefficient is to 1, the higher the fit between the data resource and the market; the closer it is to 0, the lower the fit. ik Let B be the quantization value of the i-th data resource on the k-th matching feature, where k is the type of matching feature. jk W represents the baseline value of the j-th type of market on the k-th matching feature. k Let V be the weight of the k-th matching feature, and α be an exponential adjustment parameter used to control the influence of bias on the matching coefficient, where α = 0.5. ik With B jk The closer they get, the more M ij The closer V is to 1, the better. ik With B jk When the deviation is large, M ij Significantly reduced, exponential function This will amplify the impact of the deviation, making the matching coefficient more sensitive to larger deviations;

[0082] The data engine layer specifically includes:

[0083] A high-performance analysis engine is built on HOLOGRES and STARROCKS, and it interfaces with data visualization and reporting tools such as SMART BI and FINE REPORT. It also provides unified access to structured and semi-structured data. Through flexible data parsing and transformation mechanisms, it adapts data of different formats to the engine. After data access, it uses the computing power of HOLOGRES and STARROCKS to process the data. It adopts optimized algorithms for different types of data to quickly complete data cleaning, aggregation and correlation operations. It supports high-concurrency and low-latency data analysis tasks and can respond to query requests in real time. After data processing and analysis, the results are shared through interfaces, allowing SMART BI and FINE REPORT to obtain the required data and present users with intuitive visualization reports and analysis results.

[0084] The data service layer specifically includes:

[0085] This system integrates multiple data service functions, including reporting services, dashboard services, and API interface services. The reporting service provides detailed report data to meet users' needs for historical data and detailed information. The dashboard service provides real-time key indicators and business overviews to help management quickly grasp business dynamics. The API interface service provides data interfaces for integration with other systems, facilitating data sharing and system integration, ensuring the diversity and flexibility of data services to meet the needs of different user groups. After functional integration, the data service layer provides corresponding data services based on user needs and business scenarios. The reporting service provides users with detailed data analysis through automatic generation and customization of reports. The dashboard service displays key business indicators in real time through intuitive dashboards and charts. The API interface service provides data to other systems and applications through standardized interfaces. The data service layer provides unified management of the corresponding data services, ensuring service stability and reliability. Based on user feedback and actual usage, the form and content of data services are continuously optimized to improve the quality and user experience of data services, ensuring that data services better support enterprise business decisions and operations, thereby significantly improving the utilization efficiency of data assets and the level of enterprise-level data governance.

[0086] Example 3, as Figure 1 , Figure 2 As shown, based on embodiments 1-2, the present invention also provides a data processing method based on a multi-layer architecture, implemented using a multi-layer architecture data processing system, comprising the following steps:

[0087] S1. Receive and aggregate raw detailed data from different systems through the detailed data layer, and perform atomicity, integrity and normalization processing;

[0088] S2. Based on the detailed data layer data, perform data aggregation and key indicator extraction to reduce redundancy and provide a concise data view for upper-layer applications.

[0089] S3. Utilize the application domain marketplace layer to perform thematic aggregation on the aggregated data, forming datasets tailored to specific application scenarios to meet the customized needs of different business scenarios.

[0090] S4. Perform feature analysis on the data resources of the application domain mart layer through the report mart layer, calculate the mart matching coefficient, and divide the data resources into three categories: report mart, cockpit mart and interface mart.

[0091] S5 utilizes a high-performance analytics engine to process data, supporting high-concurrency, low-latency analysis tasks, and provides reports, dashboard displays, and API interface services through data visualization tools.

[0092] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing system based on a multi-layer architecture, comprising a data processing platform, characterized in that: The data processing platform has communication connections to a detailed data layer, a summary layer, an application domain mart layer, a report mart layer, a data engine layer, and a data service layer. The data processing platform, built on MaxCompute, is used to receive, aggregate, process, and analyze raw data. The detailed data layer is used to receive and aggregate the original detailed data, and to perform atomicity, integrity and normalization processing on the data; The summary layer extracts key indicators and information items based on the detailed data of the detailed data layer to form a unified data standard. The application domain marketplace layer is used to aggregate the data from the aggregation layer by topic, forming a dedicated dataset for specific application scenarios. The report mart layer is used to classify data resources into three categories: report marts, dashboard marts, and interface marts, and to calculate mart matching coefficients to analyze the compatibility of data resources with different types of marts. Specifically, this includes: Customized datasets are obtained from the application domain marketplace layer. Data resources are divided into three categories: report marketplace, dashboard marketplace, and interface marketplace. Feature analysis is performed on the data resources to extract matching features for data classification, namely data complexity, data granularity, data sharing requirements, user interaction requirements, and data update requirements. Then, values ​​are assigned to each matching feature, and the baseline values ​​of each matching feature belonging to the three marketplace categories are determined. Based on the matching characteristics of data resources and business needs, calculate the mart matching coefficient of each data resource, evaluate the adaptability of data resources to different types of marts, and then accurately classify each data resource into the corresponding mart type. After completing the data classification and mart matching coefficient calculation, we provide diversified data services based on the classification and matching results of data resources. At the same time, we continuously optimize the form and content of data services based on user feedback and actual usage. The calculation process for the market matching coefficient is as follows: The quantitative values ​​of each data resource on five matching features are determined. The matching features include data complexity, data granularity, data sharing requirements, user interaction requirements, and data update requirements. The quantitative values ​​are assigned according to preset rules, and the baseline values ​​of the three types of marketplaces on each matching feature are determined. Based on the importance of matching features in mart matching, the weight of each matching feature is determined, where the weight of data complexity is 0.20, the weight of data granularity is 0.25, the weight of data sharing requirements is 0.10, the weight of user interaction requirements is 0.15, and the weight of data update requirements is 0.

3. For each data resource and each marketplace, calculate the deviation term for each matching feature, sum the deviation terms of all matching features, and take the reciprocal to obtain the marketplace matching coefficient. For each data resource, calculate its matching coefficient with the three types of marketplaces, compare the three marketplace matching coefficients, and allocate the data resource to the marketplace with the highest matching coefficient. The data engine layer includes an analysis engine built on HOLOGRES and STARROCKS, which respectively connect to SMART BI and FINE REPORT's data visualization and reporting tools; The data service layer is used to provide data service functions including reporting services, cockpit display services, and API interface services.

2. The data processing system based on a multi-layer architecture according to claim 1, characterized in that: The detailed data layer specifically includes: The detailed data layer receives raw detailed data from different business systems through various data source interfaces, performs preliminary data aggregation, and brings together the scattered data into a centralized data warehouse. After receiving and aggregating the data, the detailed data layer performs atomicity and integrity verification on the data; After receiving, aggregating, and checking the atomicity and integrity of the data, the detailed data layer performs data standardization processing. This standardization process includes data format unification, data cleaning, and data coding standardization. The data is then classified and indexed according to its business attributes and time dimension, thereby transforming the raw data into structured and standardized data assets.

3. The data processing system based on a multi-layer architecture according to claim 1, characterized in that: The aggregation layer specifically includes: The aggregation layer extracts normalized data from the detailed data layer and performs aggregation and cleaning operations on the data. The aggregation operation includes performing aggregation calculations on the data, aggregating data by time, business type, and region, and generating data views. The cleaning operation includes removing duplicate data, correcting outliers, and filling in missing values. After data aggregation and cleaning, the aggregation layer performs key indicator extraction and standardization. After completing data aggregation, key indicator extraction and standardization, the aggregation layer organizes the processed data into a structured data model and stores it in the designated data storage area.

4. A data processing system based on a multi-layer architecture according to claim 1, characterized in that: The application domain marketplace layer specifically includes: The application domain marketplace layer obtains data that has been aggregated and standardized from the aggregation layer, and performs thematic aggregation on the data according to different business themes. Then, the data is classified and organized according to business needs to form a dedicated dataset for specific application scenarios. After completing the topic aggregation, the application domain marketplace layer further processes and organizes the data according to business needs, generating indicators and datasets that meet the needs of specific business scenarios. After completing thematic aggregation and business requirement customization, the generated dedicated datasets for specific application scenarios will be organized in a structured manner and stored in the data warehouse.

5. A data processing system based on a multi-layer architecture according to claim 1, characterized in that: The assignment of the matching features and the baseline values ​​of the three types of marketplaces specifically include: For data complexity, the number of fields is used for quantification. Data resources with fewer than 10 fields are assigned a value of 1, data resources with 10-50 fields are assigned a value of 2, and data resources with more than 50 fields are assigned a value of 3. For report marts, the baseline value of data complexity is set to 3, for cockpit marts, the baseline value of data complexity is set to 1, and for interface marts, the baseline value of data complexity is set to 2. For data granularity, the degree of data aggregation is used for quantification. Detailed data is assigned a value of 1, lightly aggregated data is assigned a value of 2, and highly aggregated data is assigned a value of 3. For report marts, the baseline value of data granularity is set to 1. For cockpit marts, the baseline value of data granularity is set to 3. For interface marts, the baseline value of data granularity is set to 2. For data sharing requirements, the sharing frequency is used for quantification. No sharing requirement is assigned a value of 1, low sharing requirement is assigned a value of 2, and high sharing requirement is assigned a value of 3. For report marts, the baseline value of data sharing requirement is set to 1. For cockpit marts, the baseline value of data sharing requirement is set to 1. For interface marts, the baseline value of data sharing requirement can be set to 3. For user interaction needs, interaction frequency is used for quantification. Low interaction needs are assigned a value of 1, medium interaction needs are assigned a value of 2, and high interaction needs are assigned a value of 3. For the report mart, the baseline value of user interaction needs is set to 3. For the cockpit mart, the baseline value of user interaction needs is set to 2. For the interface mart, the baseline value of user interaction needs is set to 1. For data update requirements, the real-time requirements of the update are quantified. Real-time update requirements are assigned a value of 3, daily update requirements are assigned a value of 2, and periodic updates are assigned a value of 1. For report marts, the baseline value of data update requirements is set to 1. For dashboard marts, the baseline value of data update requirements can be set to 3. For interface marts, the baseline value of data update requirements is set to 2.

6. A data processing system based on a multi-layer architecture according to claim 1, characterized in that: The data engine layer specifically includes: A high-performance analysis engine is built based on HOLOGRES and STARROCKS, and it interfaces with data visualization and reporting tools such as SMART BI and FINE REPORT. At the same time, it provides unified access to structured and semi-structured data. After the data is accessed, HOLOGRES and STARROCKS are used to process the data. Optimized algorithms are used for different types of data to complete data cleaning, aggregation and correlation operations. After data processing and analysis are completed, the results are shared via an interface, enabling SMART BI and FINEREPORT to obtain the necessary data and present users with visual reports and analysis results.

7. A data processing system based on a multi-layer architecture according to claim 6, characterized in that: The data service layer specifically includes: It integrates multiple data service functions, including reporting services, cockpit display services, and API interface services; After the functional integration is completed, the data service layer provides corresponding data services based on user needs and business scenarios. The reporting service provides data analysis for users through automatic generation and customization of reports. The dashboard display service displays key business indicators in real time through intuitive dashboards and charts. The API interface service provides data to other systems and applications through standardized interfaces. The data service layer provides unified management of the relevant data services and continuously optimizes the form and content of the data services based on user feedback and actual usage.

8. A data processing method based on a multi-layer architecture, implemented based on the multi-layer architecture data processing system according to any one of claims 1-7, characterized in that, Includes the following steps: S1. Receive and aggregate raw detailed data from different systems through the detailed data layer, and perform atomicity, integrity and normalization processing; S2. Based on the detailed data layer data, perform data aggregation and extract key indicators; S3. Utilize the application domain marketplace layer to perform thematic aggregation on the aggregated data, forming datasets oriented towards specific application scenarios; S4. Perform feature analysis on the data resources of the application domain mart layer through the report mart layer, calculate the mart matching coefficient, and divide the data resources into three categories: report mart, cockpit mart and interface mart. S5 utilizes an analytics engine to process data and provides reports, dashboard displays, and API interface services through data visualization tools.

Images