Business scenario-oriented data intelligent materialization construction and hierarchical mapping processing method

Abstract

The present application relates to the technical field of data processing, in particular to a data intelligent materialization construction and hierarchical mapping processing method for business scenarios. The method comprises the following steps: multi-source heterogeneous business data acquisition and standardized preprocessing; business scenario demand analysis and materialized dimension feature extraction; intelligent materialization construction and dynamic update based on hierarchical distinguishable constraints; materialized data hierarchical mapping and business hierarchical classification system alignment; hierarchical materialized data storage verification and service output. The present application reversely determines the materialized aggregation boundary through hierarchical distinguishable constraints, and matches hierarchical attribution and aligns the system structure, so as to realize accurate mapping and structure alignment of materialized data and the business hierarchical classification system. The present application adopts a mechanism of affected unit positioning and local aggregation boundary reconstruction driven by business hierarchical change, so as to accurately lock the change influence range and perform local update, thereby avoiding the calculation resource overhead of full reconstruction.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method for constructing and hierarchically mapping data intelligence for business scenarios. Background Technology

[0002] As enterprises continue to expand their multi-source heterogeneous business systems, the materialization and hierarchical application of business data have become an important part of data platform construction, providing efficient data support for business data analysis and data service-oriented calls.

[0003] In the field of data materialization construction and optimization, a variety of technical solutions have been disclosed. For example, Chinese patent CN202411135362.1 discloses a materialized view processing method and related apparatus, and a cloud time-series data storage system. This solution pulls data streams from source data shards contained in the source database, performs aggregation calculations on the data in each time interval according to a specified time interval, obtains corresponding aggregated data, and stores the aggregated data of each time interval in the materialized view data shards of the materialized view database. Different materialized view data shards are responsible for storing data of different dimension combinations. For example, Chinese patent CN201710801784.1 discloses a method and apparatus for selecting and optimizing materialized views. This scheme obtains information from a database and extracts grouped data with aggregation functions from it, generating a set of grouped data; decomposes the non-distributed aggregation functions of the grouped data to generate grouped data elements and obtain a set of grouped data elements; identifies grouped data elements with the same data dimension and aggregation operation expression as the same grouped data family, establishes a statistical binary tree model corresponding to each grouped data family, and then establishes a minimum linear programming model for optimizing materialized views to obtain an optimized set of materialized views. The materialized views are updated according to changes in the fact table related to the materialized views, enabling rapid response in the statistical analysis of massive data.

[0004] Despite the design advantages of the aforementioned technical solutions, they also suffer from the following technical shortcomings: First, the materialized aggregation boundaries are set solely based on the data's own attributes, without establishing a corresponding mapping relationship with the business hierarchy classification system, making it difficult to adapt to the needs of hierarchical business applications. Chinese patent CN202411135362.1 divides materialized view fragments based solely on a combination of time intervals and data dimensions, and Chinese patent CN201710801784.1 divides materialized view groups solely based on data dimensions and aggregation operation expressions. Neither of these incorporates the business hierarchy classification system into the constraints of materialization construction. The generated materialized data lacks structural correspondence with the business hierarchy architecture and cannot be directly aligned and adapted to the business hierarchy classification system. Second, materialized data updates are driven solely by changes in the underlying source data and lack the ability to respond to local updates in response to changes in the business hierarchy classification system. The materialized view update in Chinese patent CN202411135362.1 is triggered by changes in the data flow of source data shards, while the materialized view update in Chinese patent CN201710801784.1 is based on changes in the associated fact table. Neither of these methods can pinpoint the affected scope and perform local adjustments for version changes in the business-level classification system. When the business-level architecture is adjusted, a full reconstruction of the materialized data is required for adaptation, resulting in high computational resource overhead and low adaptation efficiency during the update process. Therefore, we propose a data intelligent materialization construction and hierarchical mapping processing method tailored to business scenarios. Summary of the Invention

[0005] This invention addresses the technical problems in existing technologies, such as the lack of integration between materialized aggregation boundaries and business hierarchical classification system settings, and the need for full reconstruction of materialized data when business hierarchical architecture changes. It provides a data intelligent materialization construction and hierarchical mapping processing method for business scenarios, which can realize hierarchical mapping and structural alignment between materialized data and business hierarchical classification system. At the same time, it supports local incremental updates when the business hierarchical classification system version changes, effectively improving the business adaptability and dynamic adjustment efficiency of data materialization construction.

[0006] To address the aforementioned technical problems, this invention provides a method for constructing and mapping data intelligence in business scenarios, comprising the following steps: S1. Multi-source heterogeneous business data collection and standardized preprocessing: Based on the heterogeneous original business data of multi-source business systems and the preset data standardization rules, a standardized business dataset is generated by using data extraction, format normalization and metadata association annotation methods. S2. Business Scenario Requirement Analysis and Materialized Dimension Feature Extraction: Based on standardized business datasets and pre-defined business scenario computation requirement configuration information, a set of materialized dimension features is generated by using scenario requirement decomposition, materialized dimension filtering, and feature label extraction methods. S3. Intelligent materialization construction and dynamic update based on hierarchical distinguishable constraints: Based on standardized business datasets, materialized dimension feature sets, and preset business hierarchical classification systems, the materialization aggregation boundary is determined in reverse by the distinguishability requirements of downstream hierarchical mapping, generating a set of materialized data units; in response to changes in the business hierarchical classification system, the affected materialized data units are located and local aggregation boundary reconstruction is performed; S4. Alignment of materialized data hierarchy mapping with business hierarchy classification system: Based on the set of materialized data units and the preset business hierarchy classification system, hierarchical materialized datasets are generated by using hierarchical attribution matching, association mapping and system architecture alignment. S5. Layered materialized data storage verification and service-oriented output: Based on the layered materialized dataset, using distributed storage deployment, data consistency verification and call interface encapsulation methods, a layered materialized data service that can be called by business systems is generated.

[0007] As a further improvement to this technical solution, the business scenario requirement analysis and materialized dimension feature extraction in S2 include the following steps: S2.1 Based on the preset business scenario computing requirement configuration information, extract the computing dimension, aggregation granularity and access attribute parameters of the corresponding business scenario to generate a scenario dimension requirement set; S2.2 Based on the metadata information of the standardized business dataset, and combined with the scenario dimension requirement set, perform dimension validity verification to filter and obtain a set of valid materialized dimensions that meet the business calculation requirements; S2.3 Perform dimension attribute labeling and feature encoding processing on the effective materialized dimension set to generate a materialized dimension feature set carrying dimension granularity identifiers.

[0008] As a further improvement to this technical solution, the intelligent materialization construction and dynamic update based on hierarchical distinguishable constraints in S3 is divided into two sequentially executed stages: initial construction and incremental update, including the following steps: S3.1. Based on the current version of the business hierarchy classification system, and based on the standardized business dataset and the set of materialized dimension features, determine the materialized aggregation boundary and generate the initial set of materialized data units. S3.2 In response to a version change in the business-level classification system, perform a partial update based on the currently effective set of materialized data units, locate the affected materialized data units and perform a partial aggregation boundary reconstruction, generate an updated set of materialized data units and replace the original currently effective set of materialized data units as the update benchmark for the next round of version changes.

[0009] As a further improvement to this technical solution, determining the physicochemical polymerization boundary in S3.1 includes the following steps: S3.11. Based on the materialized dimension feature set, generate the computational dimension projection of the corresponding business scenario for each piece of business data in the standardized business dataset; S3.12. Based on the preset business hierarchy classification system, generate hierarchical distinguishable identification information for each business data in the standardized business dataset, corresponding to the hierarchical version. S3.13. Using the simultaneous consistency of the computational dimension projection and the hierarchical distinguishable identification information as the equivalence aggregation condition, the standardized business dataset is divided into materialized equivalence classes, and the materialized aggregation boundary is determined; wherein, the aggregateable equivalence relationship between any two business data in the standardized business dataset is determined by the following formula: ; In the formula, For business data and The result of the aggregateability equivalence determination; For any two business data points in the standardized business dataset; Projecting the computational dimensions of business data. Business data Business data The corresponding computational dimension projection; To provide hierarchical distinguishable identification information for business data. Business data Business data The corresponding levels can distinguish the identification information; This is an indicator function; it takes the value 1 when the condition inside the parentheses is true and 0 when it is false.

[0010] As a further improvement to this technical solution, the materialized data unit includes a materialized data unit identifier, a mergeable aggregation state, hierarchical member association information, and source data association information, wherein: The materialized data unit identifier is a unique coded identifier for the corresponding materialized equivalence class; Statistical status data that can be aggregated from all business data within the same materialized equivalence class can be merged and aggregated. The hierarchical member association information is a set of hierarchical member identifiers corresponding to all business data within the same materialized equivalence class; The source data association information is the range mapping identifier of all business data within the same materialized equivalence class in the standardized business dataset.

[0011] As a further improvement to this technical solution, locating the affected materialized data unit in S3.2 includes the following steps: S3.21. Obtain the version change set of the business hierarchy classification system, wherein the version change set contains information on all hierarchical members that have undergone changes in this instance; S3.22. Based on the hierarchical member association information in each materialized data unit, a matching and filtering process is performed to obtain the set of affected materialized data units containing changed hierarchical members; wherein, the set of affected materialized data units is determined by the following formula: ; In the formula, For the set of materialized data units affected, For a single materialized data unit, This refers to the hierarchical member association information contained within a materialized data unit. A set of version changes for a business hierarchy classification system; This is a condition separator in set description, representing "satisfying the following conditions". For set intersection operators, It is an empty set.

[0012] As a further improvement to this technical solution, the aggregation boundary failure determination in S3.2 specifically includes: For each materialized data unit in the affected materialized data unit set, the corresponding local standardized business data is located based on the source data association information of the materialized data unit. The hierarchical distinguishable identification information is regenerated according to the changed business hierarchical classification system. The hierarchical distinguishable identification information corresponding to all business data in the materialized data unit is checked to see if they are consistent. If they are all consistent, the materialized data unit is retained. If there is inconsistency, the aggregation boundary of the materialized data unit is determined to be invalid.

[0013] As a further improvement to this technical solution, the local aggregation boundary reconstruction in S3.2 includes the following steps: S3.23. Based on the source data association information in the failed materialized data unit, locate the corresponding local standardized business data; S3.24. Based on the revised business hierarchy classification system, the local standardized business data is reclassified into materialized equivalence classes and reconstructed into mergeable aggregation states, generating new materialized data units to replace the original invalid materialized data units.

[0014] As a further improvement to this technical solution, the alignment of the materialized data hierarchy mapping with the business hierarchy classification system in S4 includes the following steps: S4.1 Based on the set of materialized data units and the preset business hierarchy classification system, match the hierarchy belonging node corresponding to each materialized data unit and generate the hierarchy belonging matching result; S4.2 Based on the hierarchical affiliation matching results, establish hierarchical association mapping relationships between each materialized data unit according to the hierarchical association rules of the business hierarchical classification system; S4.3. According to the hierarchical structure of the business classification system, perform structural arrangement and alignment processing on the materialized data units that have completed the association mapping to generate hierarchical materialized datasets.

[0015] As a further improvement to this technical solution, the hierarchical materialized data storage verification and service-oriented output in S5 includes the following steps: S5.1 Based on the hierarchical materialized dataset, a distributed storage deployment is executed according to the hierarchical structure of the business hierarchical classification system to complete the storage of hierarchical materialized data; S5.2 Perform a data consistency check on the stored hierarchical materialized data to verify the consistency between the stored data and the content of the hierarchical materialized dataset; S5.3. For the verified hierarchical materialized data, execute the call interface encapsulation to generate a hierarchical materialized data service that can be called by business systems.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention determines the materialized aggregation boundary in reverse through hierarchical distinguishable constraints, and is equipped with hierarchical attribution matching and system architecture alignment processing to achieve accurate mapping and structural alignment between materialized data and business hierarchical classification system, effectively improving the business adaptability of materialized data and providing reliable data support for hierarchical business data service output; 2. This invention adopts a business-level change-driven mechanism for locating affected units and reconstructing local aggregation boundaries, which can accurately pinpoint the scope of change impact and perform local updates, avoiding the computational resource overhead of full reconstruction, shortening the materialized data adaptation cycle after hierarchical system adjustment, and improving the running efficiency of dynamic scenarios. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall method steps of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1As shown, this embodiment provides a data intelligence materialization construction and hierarchical mapping processing method for business scenarios. It is applicable to scenarios where enterprises have multiple business systems, diverse business data sources, and adjustments to business hierarchical classifications are needed. It is implemented based on a data processing platform with data extraction, distributed storage, and computing capabilities. The basic idea of ​​this method is: when materializing and aggregating business data, the distinguishability of the business hierarchical classification system is used as one of the conditions for determining the aggregation boundary. This ensures that data within the same materialized data unit remains consistent at the business level, allowing for direct correspondence during subsequent hierarchical mapping without additional alignment processing. When the business hierarchical classification system changes, the aggregation boundary is redefined only for the affected units based on the hierarchical member information and source data range already recorded in the materialized data unit. Overall, after standardization, dimensional feature extraction, materialized aggregation under hierarchical constraints, and hierarchical mapping alignment, multi-source business data forms a hierarchical materialized dataset, which is then encapsulated as a data service that can be called by business systems. The specific steps are as follows: S1. Multi-source heterogeneous business data collection and standardized preprocessing: Based on the heterogeneous original business data of multi-source business systems and the preset data standardization rules, a standardized business dataset is generated by using data extraction, format normalization and metadata association annotation methods. During implementation, the data processing platform connects to various data sources, including relational databases, log file systems, message queues, and application programming interfaces (APIs), through pre-configured data source connectors to extract raw business data. The extracted data varies in terms of field naming, data type, encoding format, and value domain representation. For example, the same business field might be recorded as "user_id," "uid," or "user identifier" in different systems, while dates and times might be stored in different formats such as timestamps or formatted strings.

[0020] To address the aforementioned differences, the extracted data is formatted and normalized according to pre-defined data standardization rules. These rules include: a unified field naming mapping table, data type conversion rules (e.g., converting string values ​​to floating-point or integer types), unified time format specifications (e.g., converting to "yyyy-MM-dd HH:mm:ss" format), unified encoding rules (e.g., converting different character sets to UTF-8 encoding), and a unified marking method for null and outlier values. Format normalization is performed field-by-field. Fields not defined in the rules are preserved as is; fields defined in the rules but missing from the source data are filled with pre-defined missing value markers.

[0021] During or after format normalization, metadata association and annotation are performed on the data records. The annotation includes adding a data source system identifier, a data generation timestamp, a data version number, and a unique record identifier to each data record. The data source system identifier is used to track the data's origin in subsequent processing, and the data version number is used to distinguish version differences of the same data in different batches or at different times.

[0022] After the data extraction, format normalization, and metadata association and annotation processes described above, a standardized business dataset is obtained. This dataset maintains uniformity in field structure, naming conventions, data types, and encoding formats, and each record carries traceable metadata annotation information. The standardized business dataset is stored in the data platform's distributed file system or columnar storage structure for subsequent S2 steps.

[0023] S2. Business Scenario Requirement Analysis and Materialized Dimension Feature Extraction: Based on a standardized business dataset and pre-defined business scenario computational requirement configuration information, a set of materialized dimension features is generated using scenario requirement decomposition, materialized dimension filtering, and feature label extraction methods; including the following steps: S2.1 Based on the preset business scenario computing requirement configuration information, extract the computing dimension, aggregation granularity and access attribute parameters of the corresponding business scenario to generate a scenario dimension requirement set; Specifically, the pre-defined business scenario computational requirements are stored in the data processing platform in the form of structured configuration files or configuration tables. Each configuration record corresponds to a business scenario and includes at least a unique scenario identifier, a list of computational dimensions required for that scenario, the aggregation granularity requirements for each computational dimension, and access attribute parameters. The computational dimension refers to the name of the dimension field that needs to participate in statistical analysis under the business scenario, such as "Region," "Product Line," or "Department." The aggregation granularity is represented by a hierarchical identifier; for example, the aggregation granularity of the "Region" dimension can be configured as "Province," "City," or "District / County," each corresponding to a different statistical summary level. The access attribute parameters indicate the business scenario's requirements for data timeliness and query patterns, such as whether real-time queries are required.

[0024] During implementation, the configuration information of the computing requirements for each business scenario is read one by one. The scenario identifier for each business scenario, as well as the names of each computing dimension, aggregation granularity identifier, and access attribute parameter values ​​under that scenario, are extracted and organized line by line according to the structure of "scenario identifier - dimension name - aggregation granularity - access attribute" to form a set of scenario dimension requirements. The set of scenario dimension requirements is temporarily stored in memory or intermediate storage area in the form of a structured data table. Each record corresponds to one dimension requirement in a scenario and is called by S2.2 for line-by-line verification.

[0025] S2.2 Based on the metadata information of the standardized business dataset, and combined with the scenario dimension requirement set, perform dimension validity verification to filter and obtain a set of valid materialized dimensions that meet the business calculation requirements; Specifically, the metadata information of the standardized business dataset is automatically collected and maintained by the data platform after step S1, including the field name, data type, field length or precision, and business meaning description of each field in the dataset. The purpose of dimension validity verification is to confirm one by one whether each calculation dimension in the scenario dimension requirement set has a corresponding field in the standardized business dataset, whether the data type of the field can participate in the aggregation calculation, and whether the data content carried by the field can support the aggregation granularity specified in the requirement.

[0026] During implementation, each record in the scenario-dimensional requirement set shall be verified according to the following rules: Field Existence Validation: Using the dimension name as the search key, a matching query is performed in the metadata information of the standardized business dataset. If no field with a matching name or a corresponding field that can be mapped through a preset field mapping relationship is found in the metadata information, the dimension is determined to be non-existent in the current dataset, marked as invalid, and the validation conclusion of "field missing" is recorded.

[0027] Type compatibility check: For dimensions where the field existence check passes, read the data type of that field from the metadata information. If the field type is any of numeric, string, date / time, or enumeration, it is considered type compatible; if the field type is a binary large object or unstructured text, or other types that cannot directly participate in aggregation operations, it is considered type incompatible, marked as invalid, and the check conclusion "type incompatible" is recorded.

[0028] Granularity Support Verification: For dimensions that pass type compatibility verification, verify whether the actual data of the dimension field contains the hierarchical information required for aggregation granularity. Specifically, obtain a value range sample of the field in the standardized business dataset, perform an exact match using the preset hierarchical encoding mapping rules in the dimension hierarchy table, and check whether the value range sample contains the complete hierarchical encoding corresponding to the target aggregation granularity; or check whether the field is registered as a dimension with the corresponding hierarchy in the data platform's dimension hierarchy table. If the value range contains the corresponding hierarchical information or is already registered in the dimension hierarchy table, the granularity is deemed supportable; otherwise, it is marked as invalid and the verification conclusion "granularity not satisfied" is recorded.

[0029] Dimensional requirement records that pass all three of the above checks are marked as valid and added to the valid materialized dimension set. Each record in the valid materialized dimension set includes: scenario identifier, dimension name, source field name (i.e., the corresponding field name in the standardized business dataset), data type, aggregation granularity identifier that passed the check, and a check conclusion summary.

[0030] S2.3 Perform dimension attribute labeling and feature encoding processing on the effective materialized dimension set to generate a materialized dimension feature set carrying dimension granularity identifiers.

[0031] Specifically, after obtaining a valid set of materialized dimensions, each dimension is labeled with attributes and encoded with features so that S3 can quickly reference the complete information of the dimensions when determining the materialized aggregation boundary.

[0032] Dimension attribute annotation supplements each valid materialized dimension with its field path, data type, value range summary, and hierarchical structure description in the standardized business dataset. The field path and data type are extracted from the metadata of the standardized business dataset; the value range summary is obtained by deduplicating the values ​​of the field in the standardized business dataset, with the sampling upper limit set to a preset threshold based on the total data volume (typically 1000 values, which can be flexibly adjusted according to the actual data scale); the hierarchical structure description is obtained by querying a dimension hierarchy relationship table pre-maintained by the data platform. This table records the hierarchical composition of each dimension in the form of a tree structure or parent-child pairs, for example, the hierarchical structure description of the "Region" dimension is "Region—Province—City—District / County"; if the original granularity of the source data field is finer than the currently configured aggregation granularity, dimension roll-up is performed based on this hierarchical structure description to obtain dimension values ​​consistent with the configured granularity, which are used for subsequent calculation of dimension projection generation.

[0033] Feature encoding involves generating a unique feature code for each valid materialized dimension, identifying that dimension and its aggregation granularity. The encoding rule is as follows: the code consists of three segments connected by a separator. The first segment is the hash value or abbreviation of the scene identifier; the second segment is the sequential number of the dimension within the set of valid materialized dimensions; and the third segment is the hierarchical number of the aggregation granularity. For example, "SC01_D03_L2" represents the second-level aggregation granularity of the third dimension under scene SC01.

[0034] After dimension attribute annotation and feature encoding, each valid materialized dimension forms a complete record containing dimension name, source field path, data type, value range summary, hierarchical structure description, aggregation granularity identifier, access attribute parameters, and feature encoding. Access attribute parameters are passed down with the materialized dimension feature set to guide subsequent materialized data update strategies, storage deployment, and interface access control configuration. These records together constitute the materialized dimension feature set.

[0035] S3. Intelligent Materialization Construction and Dynamic Update Based on Hierarchical Distinguishing Constraints: Based on standardized business datasets, materialized dimension feature sets, and a pre-defined business hierarchical classification system, the materialization aggregation boundary is determined in reverse based on the distinguishability requirements of downstream hierarchical mapping, generating a set of materialized data units. In response to changes in the business hierarchical classification system, the affected materialized data units are located and local aggregation boundary reconstruction is performed. Specifically, this step uses the distinguishability requirements of the business hierarchical classification system as a constraint to determine the materialized aggregation boundary, ensuring that business data within the same materialized data unit remains consistent at the business hierarchy level and can be directly mapped in the hierarchical mapping of step S4. When the business hierarchical classification system changes due to organizational restructuring or business rule changes, the hierarchical member association information and source data association information recorded in the materialized data units are used to only re-verify and reconstruct the local aggregation boundary of the affected units, without performing a full reconstruction. S3 includes two stages: initial construction and incremental update, which are described below: S3.1. Based on the current version of the business hierarchy classification system, and using the standardized business dataset and materialized dimension feature set, determine the materialization aggregation boundary and generate an initial set of materialized data units. Specifically, in the initial construction phase, based on the currently effective version of the business hierarchy classification system, perform a complete materialization aggregation process on all business data in the standardized business dataset to generate an initial set of materialized data units. The specific steps are as follows: S3.11. Based on the materialized dimension feature set, generate the computational dimension projection of the corresponding business scenario for each piece of business data in the standardized business dataset; In S3.11, the materialized dimension feature set is generated by S2, where each entry corresponds to a materialized dimension that has passed the validity check, including dimension name, source field path, data type, aggregation granularity identifier, and feature code.

[0036] For each piece of business data in the standardized business dataset, the calculated dimension projection is generated as follows: Traverse each dimension entry in the materialized dimension feature set, extracting the corresponding field value from the current business data record based on the source field path of that dimension; perform granularity roll-up processing based on the aggregation granularity identifier corresponding to that dimension to obtain dimension values ​​consistent with the configured granularity; concatenate the target granularity values ​​of each dimension sequentially according to the order of dimensions in the materialized dimension feature set to form the calculated dimension projection of that business data. The calculated dimension projection is represented in string form, with each dimension value separated by a preset delimiter; if the dimension value itself contains the preset delimiter, the delimiter character in the value is subjected to preset escape encoding processing to ensure that the projection string can be uniquely parsed. For example, a business scenario includes two materialized dimensions: "Region" and "Product Line". In a piece of business data, the region value is "East China" and the product line value is "PL-03". The calculated dimension projection of this business data is represented as "East China|PL-03". For ease of reference in subsequent formulas, the calculated dimension projection generated in the above manner will be denoted as... .

[0037] It is worth noting that computational dimension projection is used to represent the features of a piece of business data on the computational dimension that is of interest in the business scenario as a comparable identifier, which is then used by S3.13 for equivalence determination.

[0038] S3.12. Based on the preset business hierarchy classification system, generate hierarchical distinguishable identification information for each business data in the standardized business dataset, corresponding to the hierarchical version. In S3.12, the default business hierarchy classification system adopts a single-hierarchical tree structure, stored in the data processing platform in the form of a tree structure or a hierarchical relationship table. This structure describes the hierarchical relationships of business organizational structures, product classification systems, or other business management dimensions. The current version information of the business hierarchy classification system includes the version number, the complete set of hierarchical nodes under this version, and the parent-child relationships between nodes. It also includes the mapping relationship between hierarchical classification-related fields and business data fields. This mapping relationship specifies which field value in the business data is used to determine the hierarchical affiliation.

[0039] For each piece of business data in the standardized business dataset, hierarchical distinguishable identification information is generated as follows: Based on the above mapping relationship, the field value used for hierarchical attribution determination is extracted from the business data record; based on this value, the corresponding hierarchical node is found in the business hierarchical classification system, and the path is traced upwards to the root node to obtain the complete path from the root node to the attribution node; the node identifiers of each hierarchical node on this path are arranged sequentially from root to leaf to form the hierarchical distinguishable identification information of this business data. The hierarchical distinguishable identification information is represented in the form of hierarchical path encoding, for example, " / ROOT / D01 / D0102 / D0102003", indicating that this business data belongs to the hierarchical unit corresponding to the first-level node D01, the second-level node D0102, and the third-level node D0102003 under the root node. For ease of reference in subsequent formulas, the hierarchical distinguishable identification information generated in the above manner is denoted as... .

[0040] The hierarchical distinguishable identifier is used to identify the precise location of each piece of business data within the business hierarchy classification system. Business data under the same hierarchical path belong to the same hierarchical member, while business data under different hierarchical paths belong to different hierarchical members. This identifier is used as a component of the equivalence aggregation condition in S3.13.

[0041] S3.13. Using the simultaneous consistency of the computational dimension projection and the hierarchical distinguishable identification information as the equivalence aggregation condition, the standardized business dataset is divided into materialized equivalence classes and the materialized aggregation boundary is determined. In S3.13, the calculation dimension projection is obtained for each piece of business data. Distinguishing information by level Then, based on the condition that "the calculation dimension projection is the same and the hierarchical distinguishable identification information is the same", all business data in the standardized business dataset are divided into equivalence classes.

[0042] Any two business data and Aggregatable equivalence relation Determine by the following formula: ; In the formula: For business data and The result of the aggregation equivalence determination is 1, which means that two business data can be aggregated to the same materialized equivalence class, and 0 means that they cannot be aggregated. and Business data and The calculated dimensional projection is generated by step S3.11; and Business data and The hierarchical distinguishable identification information is generated by step S3.12; This is an indicator function; it takes the value 1 when the condition inside the parentheses is true and 0 when it is false.

[0043] According to this formula, only when the calculated dimension projections of two business data are completely identical and the hierarchical distinguishable identification information is completely identical, A value of 1 indicates that the two data points belong to the same materialization equivalence class; if any condition is not met, If the value is 0, the two data points will not be classified into the same materialization equivalence class.

[0044] During implementation, the string concatenated by the calculated dimensional projection and the hierarchical distinguishable identifier information is used as the grouping key. Equivalence class partitioning is performed on all business data in the standardized business dataset using hash grouping, thus satisfying the following conditions: Business data is grouped into the same materialized equivalence class. After the partitioning, all business data within each materialized equivalence class takes the same value across all computational dimensions relevant to the business scenario, and belongs to the same level node in the business hierarchy classification system. The boundary between each materialized equivalence class is the aggregation boundary of this materialized aggregation.

[0045] Furthermore, for each materialized equivalence class, a corresponding materialized data unit is generated. The materialized data unit contains the following four elements: Materialized data unit identifier: A unique coded identifier assigned to the corresponding materialized equivalence class. It can be generated in the form of "scene identifier + serial number" or UUID, and is used to uniquely reference the materialized data unit in subsequent steps.

[0046] Mergeable Aggregate Status: Statistical status data generated after performing aggregation calculations on all business data within the same materialized equivalence class. The content of the aggregation calculation depends on the aggregation indicators defined in the business scenario calculation requirement configuration information. By default, the independent definitions in the business scenario calculation requirement configuration information are used, and the materialized dimension feature set synchronously carries this indicator information for parameter pass-through. If not defined separately in the configuration information, the aggregation indicators carried in the materialized dimension feature set are used. Aggregate indicators include basic statistics such as the count of data records, the sum of numerical indicator fields, and the maximum and minimum values. The statistics stored in the mergeable aggregate status are the basic statistics that can be merged. For example, the count and sum value are stored simultaneously to support the merging and recalculation of the mean. When two or more materialized data units need to be merged due to subsequent changes in the hierarchical system, their aggregation status can be recalculated by merging the basic statistics without having to backtrack the original data.

[0047] Hierarchical member association information: A set of hierarchical member identifiers corresponding to all business data within the same materialized equivalence class. Since all business data within the same materialized equivalence class has the same distinguishable hierarchical identifier information, the hierarchical member association information is the set of hierarchical node identifiers at the end of the hierarchical path corresponding to that equivalence class, for example, {“D0102003”}. To facilitate set operations in subsequent step S3.22, the hierarchical member association information is stored in set form.

[0048] Source data association information: This is a range mapping identifier for all business data within the same materialized equivalence class in the standardized business dataset. It records the storage location of each piece of business data constituting the materialized equivalence class in the standardized business dataset. Depending on the storage method of the standardized business dataset, the range mapping identifier can be represented as a row number range, a list of record primary keys, or an offset within a partition. For example, if an equivalence class involves 15 records in the standardized business dataset with primary keys ID00125 to ID00138 and ID00256, the source data association information can be recorded as "keys:[ID00125-ID00138,ID00256]". This information is used in subsequent incremental update phases to quickly locate local original data and supports the re-verification and reconstruction of aggregation boundaries.

[0049] After the above processing, the materialized data units corresponding to all materialized equivalence classes together constitute the initial set of materialized data units, which serves as the currently effective set of materialized data units for hierarchical mapping alignment in step S4, and as the update benchmark for the incremental update stage of S3.2.

[0050] S3.2 In response to a version change in the business-level classification system, a partial update is performed based on the currently effective set of materialized data units. This involves locating the affected materialized data units and reconstructing the local aggregation boundaries, generating an updated set of materialized data units that replaces the original set, serving as the update benchmark for the next version change. Specifically, the business-level classification system may undergo version changes in actual business operations due to organizational restructuring, product classification changes, or changes in management granularity. Version change detection can be achieved by comparing version number changes in the business-level classification system, monitoring change events in the configuration management system, or receiving change notifications. When a version change in the business-level classification system is detected, the incremental update process in S3.2 is triggered, performing a partial update based on the currently effective set of materialized data units. Wherein: Locating the affected materialized data unit involves the following steps: S3.21. Obtain the version change set of the business hierarchy classification system, wherein the version change set contains information on all hierarchical members that have undergone changes in this instance; In S3.21, version change sets are generated by the data platform when the business hierarchy classification system undergoes version changes. A version change set is a list or collection of hierarchical member identifiers, where each record corresponds to a hierarchical member that has changed in this version change. Types of hierarchical member changes include: adding hierarchical nodes, deleting hierarchical nodes, changing the parent node of a hierarchical node, and merging or splitting hierarchical nodes. The version change set records the hierarchical member identifiers affected by these changes, regardless of the change type. When a hierarchical node is added, deleted, has its parent node changed, merged, or split, the version change set simultaneously includes the directly changed hierarchical node and all its descendant hierarchical nodes, ensuring that all hierarchical members whose hierarchical affiliation paths are affected by this change are included. For example, if hierarchical node D0102 is split and the affiliation of hierarchical node D0203 is transferred in a certain change, then the version change set... It includes D0102 and all its descendant nodes, and D0203 and all its descendant nodes.

[0051] S3.22. Based on the hierarchical member association information in each materialized data unit, perform matching and filtering to obtain a set of affected materialized data units that include changed hierarchical members; In S3.22, after obtaining the version change set... Then, iterate through each materialized data unit in the currently active set of materialized data units. Extract its hierarchical member association information . For materialized data units The set of hierarchical member identifiers recorded in step S3.13. Check. and Does an intersection exist? The set of affected materialized data units. Determine by the following formula: ; In the formula: For the set of materialized data units affected; For a single materialized data unit in the currently active set of materialized data units; For materialized data units The hierarchical member association information contained therein is represented in the form of a set; A set of version changes for a business hierarchy classification system; This represents the intersection operation of sets. This indicates that there is an overlap between the hierarchical member association information and the version change set.

[0052] When the hierarchical member association information of a materialized data unit intersects with the version change set, the unit is marked as an affected unit and further aggregation boundary failure determination is required; if the intersection is empty, it indicates that the hierarchical affiliation of the materialized data unit has not been affected by this change, its aggregation boundary is still valid, and it is directly retained.

[0053] The aggregation boundary failure determination specifically includes: for each materialized data unit in the affected materialized data unit set, locating the corresponding local standardized business data based on the source data association information of the materialized data unit, regenerating the hierarchical distinguishable identification information according to the changed business hierarchy classification system, and verifying whether the hierarchical distinguishable identification information corresponding to all business data in the materialized data unit is consistent; if all are consistent, the materialized data unit is retained; if there is inconsistency, the aggregation boundary of the materialized data unit is determined to be failed.

[0054] Specifically, for the affected set of materialized data units For each materialized data unit in the data, the following determination is performed: First, based on the source data association information of the materialized data unit, the corresponding local standardized business data is located in the standardized business dataset. For example, if the source data association information record is "keys:[ID00125-ID00138,ID00256]", then the corresponding business data record is extracted from the standardized business dataset based on the record's primary key, serving as the local original data corresponding to this unit.

[0055] Secondly, based on the revised business hierarchy classification system, the same processing method as in step S3.12 is used to regenerate hierarchical distinguishable identification information for each record in the extracted local standardized business data.

[0056] Then, verify whether the regenerated hierarchical distinguishable identifiers are consistent, that is, traverse the hierarchical distinguishable identifiers of each record in the local data and compare whether they are all the same.

[0057] If all are consistent, it indicates that under the changed business hierarchy classification system, all business data within this materialized data unit still belong to the same level node, the original aggregation boundary remains valid, the materialized data unit is retained, and its hierarchical member association information is updated to the new hierarchical node identifier.

[0058] If inconsistencies exist, it indicates that under the changed business hierarchy classification system, the business data within the materialized data unit is scattered to different hierarchical nodes, and the original aggregation boundary has been broken. Therefore, the aggregation boundary of the materialized data unit is deemed invalid, and a local aggregation boundary reconstruction is performed.

[0059] Local aggregation boundary reconstruction includes the following steps: S3.23. Based on the source data association information in the failed materialized data unit, locate the corresponding local standardized business data; specifically, for the materialized data unit determined to be failed, locate the corresponding local standardized business data in the standardized business dataset according to its source data association information, that is, extract the local business data of the corresponding record from the standardized business dataset as the original input data for local reconstruction.

[0060] S3.24. Based on the revised business hierarchy classification system, the local standardized business data is reclassified into materialized equivalence classes and reconstructed into mergeable aggregation states, generating new materialized data units to replace the original invalid materialized data units.

[0061] In S3.24, for the locally standardized business data located in S3.23, based on the revised business hierarchy classification system, the same processing flow as S3.11 to S3.13 in S3.1 is adopted, and the following operations are performed again: A calculated dimension projection is generated for each record in the local data based on the materialized dimension feature set; new hierarchical distinguishable identifier information is generated for each record based on the revised business hierarchy classification system; and materialized equivalence classes are re-divided based on the condition that the calculated dimension projection and the hierarchical distinguishable identifier information are simultaneously consistent. During local reconstruction, the materialized dimension feature set remains unchanged, and the generation method of the calculated dimension projection is consistent with S3.11, and does not change due to the local data range.

[0062] For each materialized equivalence class obtained by re-division, a new materialized data unit is generated according to the four-element structure described in S3.1, including assigning a new materialized data unit identifier, reconstructing a mergeable aggregation state based on business data within the equivalence class, and recording new hierarchical member association information and source data association information.

[0063] The newly generated materialized data units replace the original invalid materialized data units, while unaffected materialized data units are retained. For all affected materialized data units, deduplication is performed using the result of concatenating the calculated dimension projection and the hierarchical distinguishable identifier information as the unique key. If multiple units have completely identical key values, cross-unit merging calculations are performed based on their respective mergeable aggregation states to generate a unified materialized data unit to replace the original multiple duplicate units. After the partial reconstruction is completed, an updated set of materialized data units is obtained, replacing the original currently effective set of materialized data units. This set serves as the update benchmark for the next round of version changes and is also used by S4 for subsequent hierarchical mapping alignment.

[0064] It should be noted that, through the coordination of the two phases of S3.1 initial construction and S3.2 incremental update, this step uses the simultaneous consistency of the calculated dimension projection and the hierarchical distinguishable identifier information as the equivalence aggregation condition to divide materialized equivalence classes. This implements the distinguishability requirement of the business hierarchical classification system into the process of determining the materialized aggregation boundary, ensuring that the materialized data unit establishes a correspondence with the business hierarchical node at the time of generation. When the business hierarchical classification system undergoes a version change, the intersection matching is performed based on the pre-set hierarchical member association information in the materialized data unit and the version change set to locate the affected unit. After the aggregation boundary failure is determined, only the failed unit is subjected to local equivalence class repartitioning and aggregation state reconstruction. The updated set of materialized data units replaces the original set as the update benchmark for the next round of changes. The above processing method maintains the consistency between the materialization result and the hierarchical system while limiting the impact of hierarchical changes to a local scope, avoiding the rematerialization of the entire data, and providing a stable data foundation for the hierarchical mapping alignment in the subsequent S4 step.

[0065] S4. Alignment of Materialized Data Hierarchical Mapping with Business Hierarchical Classification System: Based on the set of materialized data units and the preset business hierarchical classification system, hierarchical materialized datasets are generated using hierarchical attribution matching, relational mapping, and system architecture alignment methods; including the following steps: S4.1 Based on the set of materialized data units and the preset business hierarchy classification system, match the hierarchy belonging node corresponding to each materialized data unit and generate the hierarchy belonging matching result; Specifically, the materialized data unit set is generated by S3. Each materialized data unit carries hierarchical member association information, which records the hierarchical node identifier to which the business data within the unit belongs in the business hierarchical classification system. The preset business hierarchical classification system is stored in the form of a tree structure or a hierarchical relationship table, including the node identifier, node name, parent node identifier, and depth position of each node in the hierarchical structure.

[0066] During implementation, each materialized data unit in the set of materialized data units is traversed, and the hierarchical node identifier in its hierarchical member association information is extracted. Using this node identifier as the search key, the corresponding hierarchical node is found in the business hierarchical classification system. The matching method is as follows: the hierarchical node identifier is matched against the node identifiers of each node in the business hierarchical classification system to locate a unique hierarchical node. If a match is successful, the affiliation relationship between the materialized data unit and the corresponding hierarchical node is recorded, generating a hierarchical affiliation matching record, including the materialized data unit identifier, hierarchical node identifier, node name, and the node's depth position in the hierarchical structure. If some hierarchical node identifiers become invalid due to changes in the business hierarchical classification system version, resulting in no match, the materialized data unit is marked as pending, the reason for the non-match is recorded, and it is entered into step S4.2 along with the materialized data units that have completed the matching process, and will be processed separately when establishing hierarchical association mapping relationships later.

[0067] After the matching is completed, the hierarchical affiliation matching results are obtained and stored in the form of a structured data table. Each record corresponds to the hierarchical affiliation information of a materialized data unit, which is used in step S4.2 to establish hierarchical association mapping relationship.

[0068] S4.2. Based on the hierarchical attribution matching results, and according to the hierarchical association rules of the business hierarchical classification system, establish the hierarchical association mapping relationship between each materialized data unit. Specifically, in the business hierarchical classification system, nodes are hierarchically associated through parent-child relationships, meaning each non-root node has a unique parent node, and each node can have zero or more child nodes. This parent-child relationship is explicitly recorded in the data structure of the business hierarchical classification system. This step uses this as the hierarchical association rule to establish the hierarchical association mapping relationship between materialized data units.

[0069] In implementation, firstly, based on the hierarchical affiliation matching results generated in step S4.1, each materialized data unit is mapped to its corresponding hierarchical node using the hierarchical node identifier as the association key, thus determining the list of materialized data units attached to each hierarchical node. Secondly, the nodes of each level of the business hierarchical classification system are traversed from bottom to top. Based on the mergeable aggregation state of the materialized data units corresponding to all child nodes under the same parent node, a merging calculation is performed to generate the parent materialized data units corresponding to each non-terminal level layer by layer. The parent units are synchronously associated with the identifier and depth information of the corresponding hierarchical nodes. Then, according to the parent-child node relationship of each node in the business hierarchical classification system, a parent-child mapping is established for the matched materialized data units: for any materialized data unit, based on the parent node identifier of its corresponding hierarchical node, the materialized data units belonging to that parent node are found and used as the parent unit of the current unit; at the same time, based on the identifiers of each child node of the current hierarchical node, the materialized data units belonging to each child node are found and used as the child units of the current unit. The parent-child mapping relationship is recorded in the form of "parent unit identifier - child unit identifier" pairs.

[0070] For multiple materialized data units belonging to the same level node, they are parallel at that level and together constitute the materialized data set under that level node. For example, if a level node has three materialized data units with different computational dimension projections but the same level affiliation, they will be organized in a parallel relationship in the subsequent structural arrangement.

[0071] For materialized data units marked as pending in step S4.1, since their hierarchical node identifiers have expired, they cannot be located as valid nodes in the business hierarchical classification system. When establishing hierarchical association mapping relationships, they are attached to the preset unmatched nodes or recorded separately. They do not participate in the normal hierarchical parent-child summary calculation, but retain the complete unit data and unmatched mark so as not to block the subsequent structural layout and alignment processing in step S4.3. Their unmatched mark is also retained in the processing results for subsequent manual verification or automatic repair.

[0072] Once the hierarchical association mapping is established, a tree-like association structure is formed between each materialized data unit, which is isomorphic to the business hierarchical classification system. Each matched materialized data unit has a definite hierarchical position and parent-child relationship in this structure.

[0073] S4.3. According to the hierarchical structure of the business classification system, perform structural arrangement and alignment processing on the materialized data units that have completed the association mapping to generate hierarchical materialized datasets.

[0074] Specifically, based on the hierarchical association mapping relationship established in S4.2, the materialized data units are structurally arranged according to the hierarchical structure of the business hierarchical classification system. The specific method is as follows: starting from the root node of the business hierarchical classification system, materialized data units are arranged layer by layer from top to bottom in ascending order of hierarchical depth. Within the same level, materialized data units belonging to the same parent node are arranged sequentially according to a preset sorting rule. The sorting rule can be based on the lexicographical order of the materialized data unit identifier or the order of generation.

[0075] After the structure is arranged, in one optional implementation, the aggregated state data of each level of materialized data unit is aligned and verified, specifically including the following two items: Granularity Consistency Verification: This checks whether the aggregated state data of each materialized data unit under each level node is consistent with the aggregation granularity requirements of that level node. The aggregation granularity requirements are defined in the materialized dimension feature set generated in step S2, representing the required level of summarization fineness for that level dimension. If the aggregation granularity of a materialized data unit is inconsistent with the requirements of its level node, the granularity difference information is recorded, and the unit is marked as having an inconsistent granularity state.

[0076] Parent-child aggregation relationship verification: This checks whether the aggregation status data of materialized data units between parent and child levels satisfy the aggregation relationship. Specifically, based on the parent materialized data units generated in stage S4.2, for a pair of materialized data units with a parent-child relationship, the mergeable aggregation status of the parent unit should cover the sum of the aggregation results of all its child units in terms of summable statistics such as counts and summations. During verification, the counts and summations of all child materialized data units under the same parent node are accumulated separately and compared with the corresponding statistics of the parent materialized data unit. If the two are inconsistent and the deviation exceeds the preset allowable range (typically set to 0, i.e., requiring complete consistency, or set to non-zero tolerance according to data accuracy requirements), the aggregation deviation information is recorded, and the relevant units are marked as aggregation deviation states.

[0077] For inconsistencies in the granularity of the markers or information on the summary deviations during the verification process, these are recorded and output along with the hierarchical materialized dataset as part of the alignment verification results, for use in subsequent data quality management or manual verification.

[0078] After structural arrangement and alignment, a hierarchical materialized dataset is obtained. This dataset is organized layer by layer according to the hierarchical structure of the business hierarchy classification system. Each level contains materialized data units belonging to each node of that level, and there are clear parent-child mapping relationships and hierarchical affiliations between the materialized data units. The hierarchical materialized dataset is stored in the form of a tree data structure or a hierarchically nested data structure for subsequent distributed storage deployment and service output in the S5 steps.

[0079] S5. Layered Materialized Data Storage Validation and Service-Oriented Output: Based on a layered materialized dataset, using distributed storage deployment, data consistency verification, and API encapsulation, a layered materialized data service that can be invoked by business systems is generated. This includes the following steps: S5.1. Based on the hierarchical materialized dataset, distributed storage deployment is executed according to the hierarchical structure of the business classification system to complete the storage of hierarchical materialized data. Specifically, the hierarchical materialized dataset is generated by S4 and organized in the form of a tree data structure or a hierarchical nested data structure, containing the materialized data units of each level node and their parent-child mapping relationships. The distributed storage deployment is executed on the data platform storage layer composed of multiple storage nodes.

[0080] During implementation, data from each level of the hierarchical materialized dataset is distributed and stored across storage nodes according to the hierarchical structure of the business classification system. The storage distribution strategy is as follows: data at the same level is preferentially stored on nearby storage nodes within the same or the same availability zone / network topology to reduce network transmission overhead during cross-level queries; at the same time, for levels with large data volumes, further sharding can be performed by parent node, storing materialized data units belonging to the same parent node within the same shard to support localized data retrieval during parent node summary queries.

[0081] When storing each materialized data unit, its materialized data unit identifier, mergeable aggregation status, hierarchical member association information, and source data association information are written as a storage record. For materialized data units at the leaf level, source data association information is written synchronously. For materialized data units generated by parent aggregations, source data association information does not need to be stored. The storage format can adopt a columnar storage structure, organizing the statistics in the aggregation status by column, which facilitates subsequent query and aggregation operations by indicator dimension. The primary key of the stored record uses the materialized data unit identifier, and an auxiliary index based on the hierarchical node identifier is established to support fast location of hierarchical node retrieval.

[0082] After storage is completed, the storage location mapping information of each materialized data unit is recorded, including the storage node identifier and storage fragment identifier, for locating data during the subsequent S5.2 step of data consistency verification.

[0083] S5.2 Perform a data consistency check on the stored hierarchical materialized data to verify the consistency between the stored data and the hierarchical materialized dataset. Specifically, the data consistency check is used to ensure that the data written to the storage system is completely consistent with the hierarchical materialized dataset generated in step S4 in terms of content, and to eliminate possible write errors or data loss during the storage process.

[0084] During implementation, verification is performed as follows: Each materialized data unit in the hierarchical materialized dataset is traversed, and the corresponding storage record is read from the storage system using an auxiliary index based on its materialized data unit identifier. The field values ​​in the storage record are compared one by one with the corresponding field values ​​in the hierarchical materialized dataset. The comparison includes the materialized data unit identifier, the values ​​of each statistic in the mergeable aggregation state, hierarchical member association information, and source data association information. For leaf-level materialized data units, the source data association information is verified synchronously. For materialized data units generated by parent aggregations, there is no need to verify the source data association information.

[0085] Furthermore, the comparison results are handled in the following ways: If all fields match, the materialized data unit storage verification is considered successful.

[0086] If inconsistencies are found in the field comparison, the materialized data unit is deemed to have failed storage verification. The names, expected values, and actual values ​​of the inconsistent fields are recorded, and a rewrite operation is triggered for that unit, overwriting the corresponding record in the storage system with the content from the hierarchical materialized dataset. After rewriting, the comparison verification is performed again until the verification passes or the preset retry limit is reached (typically set to 3 times). If the verification still fails after reaching the retry limit, the unit is marked as being in a storage abnormal state, the abnormal information is recorded, and an alarm is output.

[0087] After all materialized data units have been verified, the verification pass rate is calculated. If all units pass the verification, the stored data is consistent with the hierarchical materialized dataset, and the process proceeds to S5.3 to encapsulate the call interface. If there are any units with storage anomalies, the anomaly information is output along with the verification results and handled by operations and maintenance.

[0088] S5.3. For the verified hierarchical materialized data, an API call is encapsulated to generate a hierarchical materialized data service that can be called by business systems. Specifically, after the data consistency verification in S5.2 passes, the hierarchical materialized data in the storage system is encapsulated into an API call and exposed as a service for business systems to call.

[0089] Interface encapsulation includes the following types of interface definitions and implementations: The query interface by hierarchical node accepts the hierarchical node identifier as an input parameter, locates all materialized data units under that hierarchical node through auxiliary indexes, and returns the corresponding aggregated status data. It supports specifying return fields and can return partial statistics such as counts and sums as needed.

[0090] The hierarchical path query interface receives the hierarchical path code as input parameters, parses the node identifiers at each level in the path, locates and returns the set of materialized data units under the end node of the path level by level.

[0091] The parent-child hierarchical summary query interface receives the parent node identifier as input and queries all child-level materialized data units belonging to that parent node. It can directly return pre-stored parent aggregation state data, or perform secondary summary calculations based on the mergeable aggregation states of the child units, returning the parent summary result. This interface leverages the characteristic of distributed storage where parent and child node data reside on the same shard to achieve localized summary calculations, reducing data transmission across nodes.

[0092] Batch Export Interface: Receives a list of hierarchical node identifiers and field filtering conditions as input parameters, reads the corresponding materialized data units in batches, and returns a structured dataset for batch analysis or report generation by downstream business systems.

[0093] Each interface is implemented as a RESTful API or RPC service, and interface requests and responses use structured data formats such as JSON or Protocol Buffers. The interface implementation integrates permission verification and access frequency control to ensure the security and stability of the data service.

[0094] Once the interface is encapsulated, a hierarchical materialized data service is generated that can be called by business systems. By calling the above interface, business systems can obtain pre-aggregated materialized data aligned with the business hierarchical classification system according to hierarchical nodes or hierarchical paths, without needing to perform data association and aggregation calculations during querying.

[0095] Those skilled in the art will understand that the process of implementing all or part of the steps of the above embodiments can be carried out by hardware or by a program instructing the relevant hardware.

[0096] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for constructing and hierarchically mapping data intelligence for business scenarios, characterized in that, Includes the following steps: S1. Multi-source heterogeneous business data collection and standardized preprocessing: Based on the heterogeneous original business data of multi-source business systems and the preset data standardization rules, a standardized business dataset is generated by using data extraction, format normalization and metadata association annotation methods. S2. Business Scenario Requirement Analysis and Materialized Dimension Feature Extraction: Based on standardized business datasets and pre-defined business scenario computation requirement configuration information, a set of materialized dimension features is generated by using scenario requirement decomposition, materialized dimension filtering, and feature label extraction methods. S3. Intelligent materialization construction and dynamic update based on hierarchical distinguishable constraints: Based on standardized business datasets, materialized dimension feature sets, and preset business hierarchical classification systems, the materialization aggregation boundary is determined in reverse by the distinguishability requirements of downstream hierarchical mapping, generating a set of materialized data units; in response to changes in the business hierarchical classification system, the affected materialized data units are located and local aggregation boundary reconstruction is performed; S4. Alignment of materialized data hierarchy mapping with business hierarchy classification system: Based on the set of materialized data units and the preset business hierarchy classification system, hierarchical materialized datasets are generated by using hierarchical attribution matching, association mapping and system architecture alignment. S5. Layered materialized data storage verification and service-oriented output: Based on the layered materialized dataset, using distributed storage deployment, data consistency verification and call interface encapsulation methods, a layered materialized data service that can be called by business systems is generated.

2. The data intelligence materialization construction and hierarchical mapping processing method for business scenarios according to claim 1, characterized in that, The business scenario requirement analysis and materialized dimension feature extraction in S2 include the following steps: S2.1 Based on the preset business scenario computing requirement configuration information, extract the computing dimension, aggregation granularity and access attribute parameters of the corresponding business scenario to generate a scenario dimension requirement set; S2.2 Based on the metadata information of the standardized business dataset, and combined with the scenario dimension requirement set, perform dimension validity verification to filter and obtain a set of valid materialized dimensions that meet the business calculation requirements; S2.3 Perform dimension attribute labeling and feature encoding processing on the effective materialized dimension set to generate a materialized dimension feature set carrying dimension granularity identifiers.

3. The data intelligence materialization construction and hierarchical mapping processing method for business scenarios according to claim 1, characterized in that, The intelligent materialization construction and dynamic update based on hierarchical distinguishable constraints in S3 is divided into two sequentially executed stages: initial construction and incremental update, including the following steps: S3.

1. Based on the current version of the business hierarchy classification system, and based on the standardized business dataset and the set of materialized dimension features, determine the materialized aggregation boundary and generate the initial set of materialized data units. S3.2 In response to a version change in the business-level classification system, perform a partial update based on the currently effective set of materialized data units, locate the affected materialized data units and perform a partial aggregation boundary reconstruction, generate an updated set of materialized data units and replace the original currently effective set of materialized data units as the update benchmark for the next round of version changes.

4. The data intelligence materialization construction and hierarchical mapping processing method for business scenarios according to claim 3, characterized in that, Determining the physical-chemical polymerization boundary in S3.1 includes the following steps: S3.

11. Based on the materialized dimension feature set, generate the computational dimension projection of the corresponding business scenario for each piece of business data in the standardized business dataset; S3.

12. Based on the preset business hierarchy classification system, generate hierarchical distinguishable identification information for each business data in the standardized business dataset, corresponding to the hierarchical version. S3.

13. Using the simultaneous consistency of the computational dimension projection and the hierarchical distinguishable identification information as the equivalence aggregation condition, the standardized business dataset is divided into materialized equivalence classes, and the materialized aggregation boundary is determined; wherein, the aggregateable equivalence relationship between any two business data in the standardized business dataset is determined by the following formula: ； In the formula, For business data and The result of the aggregateability equivalence determination; For any two business data points in the standardized business dataset; Projecting the computational dimensions of business data. Business data Business data The corresponding computational dimension projection; To provide hierarchical distinguishable identification information for business data. Business data Business data The corresponding levels can distinguish the identification information; This is an indicator function; it takes the value 1 when the condition inside the parentheses is true and 0 when it is false.

5. The data intelligence materialization construction and hierarchical mapping processing method for business scenarios according to claim 3, characterized in that, The materialized data unit includes a materialized data unit identifier, a mergeable aggregation status, hierarchical member association information, and source data association information, wherein: The materialized data unit identifier is a unique coded identifier for the corresponding materialized equivalence class; Statistical status data that can be aggregated from all business data within the same materialized equivalence class can be merged and aggregated. The hierarchical member association information is a set of hierarchical member identifiers corresponding to all business data within the same materialized equivalence class; The source data association information is the range mapping identifier of all business data within the same materialized equivalence class in the standardized business dataset.

6. The data intelligence materialization construction and hierarchical mapping processing method for business scenarios according to claim 3, characterized in that, Locating the affected materialized data unit in S3.2 includes the following steps: S3.

21. Obtain the version change set of the business hierarchy classification system, wherein the version change set contains information on all hierarchical members that have undergone changes in this instance; S3.

22. Based on the hierarchical member association information in each materialized data unit, a matching and filtering process is performed to obtain the set of affected materialized data units containing changed hierarchical members; wherein, the set of affected materialized data units is determined by the following formula: ； In the formula, For the set of materialized data units affected, For a single materialized data unit, This refers to the hierarchical member association information contained within a materialized data unit. A set of version changes for a business hierarchy classification system; This is a condition separator in set description, representing "satisfying the following conditions". For set intersection operators, It is an empty set.

7. The data intelligence materialization construction and hierarchical mapping processing method for business scenarios according to claim 6, characterized in that, The aggregation boundary failure determination in S3.2 specifically includes: For each materialized data unit in the affected materialized data unit set, the corresponding local standardized business data is located based on the source data association information of the materialized data unit. The hierarchical distinguishable identification information is regenerated according to the changed business hierarchical classification system. The hierarchical distinguishable identification information corresponding to all business data in the materialized data unit is checked to see if they are consistent. If they are all consistent, the materialized data unit is retained. If there is inconsistency, the aggregation boundary of the materialized data unit is determined to be invalid.

8. The data intelligence materialization construction and hierarchical mapping processing method for business scenarios according to claim 7, characterized in that, The local aggregation boundary reconstruction in S3.2 includes the following steps: S3.

23. Based on the source data association information in the failed materialized data unit, locate the corresponding local standardized business data; S3.

24. Based on the revised business hierarchy classification system, the local standardized business data is reclassified into materialized equivalence classes and reconstructed into mergeable aggregation states, generating new materialized data units to replace the original invalid materialized data units.

9. The data intelligence materialization construction and hierarchical mapping processing method for business scenarios according to claim 1, characterized in that, The alignment of the materialized data hierarchy mapping with the business hierarchy classification system in S4 includes the following steps: S4.1 Based on the set of materialized data units and the preset business hierarchy classification system, match the hierarchy belonging node corresponding to each materialized data unit and generate the hierarchy belonging matching result; S4.2 Based on the hierarchical affiliation matching results, establish hierarchical association mapping relationships between each materialized data unit according to the hierarchical association rules of the business hierarchical classification system; S4.

3. According to the hierarchical structure of the business classification system, perform structural arrangement and alignment processing on the materialized data units that have completed the association mapping to generate hierarchical materialized datasets.

10. The data intelligence materialization construction and hierarchical mapping processing method for business scenarios according to claim 1, characterized in that, The hierarchical materialized data storage verification and service-oriented output in S5 includes the following steps: S5.1 Based on the hierarchical materialized dataset, a distributed storage deployment is executed according to the hierarchical structure of the business hierarchical classification system to complete the storage of hierarchical materialized data; S5.2 Perform a data consistency check on the stored hierarchical materialized data to verify the consistency between the stored data and the content of the hierarchical materialized dataset; S5.

3. For the verified hierarchical materialized data, execute the call interface encapsulation to generate a hierarchical materialized data service that can be called by business systems.

Images