Trusted data space-based label flow batch calculation and arrangement and assembly method and device

Abstract

The invention provides a label flow batch calculation and arrangement assembly method and device based on a trusted data space. Based on a trusted data space base, business index logic is abstracted into a reusable operator through a logic plan and physical execution decoupling mechanism, and optimal allocation of calculation tasks is realized in combination with a flow batch uniform resource scheduler; and meanwhile, an incremental state management engine is introduced to ensure the data consistency in a stream batch mixed scene, the label query service of millisecond-level response is realized, compared with a traditional scheme, the resource consumption is effectively reduced, and real-time label production and second-level updating under a large data volume are supported.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method and apparatus for tag stream batch calculation, arrangement and assembly based on a trusted data space. Background Technology

[0002] Data tags, as a core element of modern data asset management, directly impact the real-time nature and accuracy of enterprise business insights through their efficient generation and application capabilities. In traditional data processing architectures, storage and computing models based on single machines or simple clusters are limited by hardware resources and algorithm design, generally suffering from low data throughput, high computational latency, and fixed indicator dimensions, making it difficult to handle the mixed processing needs of multi-source heterogeneous data. Especially when processing large-scale data, existing technologies rely on ETL processes for data unification, requiring the entire raw data to be loaded into memory before performing format conversion and logical operations. This leads to a surge in serialization overhead, competition for computing resources, and data skew, resulting in excessively low resource utilization and severely restricting the timeliness of tag generation.

[0003] In addition, the traditional batch-stream separation architecture requires the maintenance of two independent computing engines, which not only doubles the development and maintenance costs, but also causes discrepancies in the metric standards of offline tags and real-time tags due to the lack of a data consistency guarantee mechanism.

[0004] To address the aforementioned pain points, there is an urgent need to build a tag computing paradigm that integrates batch and stream processing. This paradigm should enable collaborative computing of offline batch data and real-time streaming data through dynamic resource scheduling, incremental computing models, and a unified execution engine, while also supporting the flexible combination and on-demand orchestration of multi-dimensional tags. Summary of the Invention

[0005] The embodiments of the present invention provide a method and apparatus for batch calculation and orchestration of tag streams based on trusted data space, which can realize the collaborative calculation of offline batch data and real-time streaming data, and support the flexible combination and on-demand orchestration of multi-dimensional tags.

[0006] According to one aspect of the present invention, a method for tag stream batch computation and orchestration assembly based on trusted data space is provided, comprising:

[0007] Based on multidimensional data modeling and dynamic rule engine, a full-domain business data asset library is constructed; key business entities are extracted through dimensional modeling to build dimensional models, and unique business attribute fields are selected as dimensional primary keys and quantitative analysis indicators are selected as measurement items; a declarative configuration interface is used to define multi-level operation logic of basic operations, conditional operations and compound operations, and version management of rule parameter templates is achieved through JSON Schema;

[0008] Based on the structured configuration parsing and syntax tree reconstruction mechanism, a KPI rule metadata table containing dimension identifiers, metric identifiers, compound expressions, and execution strategies is constructed; operation types, field mapping sets, and condition expression trees are encapsulated through the Form intermediate object, and standard ANSI SQL is generated through the Velocity template engine; a multi-stage syntax processor is built using ANTLR v4 to adapt to the target database dialect, and the optimal execution plan is generated through logical optimization, physical optimization, and pre-compilation verification.

[0009] A collaborative processing architecture for stream and batch processing is constructed. Data standardization and transformation are achieved through the metadata registry center of the multi-source heterogeneous data access layer. An engine is built based on the runtime context to dynamically load dimension mapping tables and feature encoding libraries, allocate computing resources, and pre-configure fault tolerance and retry mechanisms. A hierarchical processing engine is adopted to execute streaming real-time stateful aggregation and batch incremental computing respectively. Labels with lineage are generated through dynamic rule parsing, and after standardized quality control, they are output using a hierarchical storage strategy.

[0010] Optionally, the basic operations achieve multi-dimensional data summarization through sliding window aggregation, accumulation operations, deduplication operations, and COUNT_DISTINCT statistics; the conditional operations support combined filtering rules of time interval BETWEEN, numerical comparison, regular expression REGEXP_MATCH, and enumerated value matching; the composite operations support multi-level nested expression definitions, including nested combinations of arithmetic and logical operations.

[0011] Optionally, the operation type of the Form intermediate object is mapped to CRUD semantics of SELECT, INSERT, UPDATE, and DELETE. The field mapping set uses a hash structure to store the one-to-one correspondence between the source field and the target field. The condition expression tree organizes the AND / OR logical operators in a binary tree structure.

[0012] When generating standard ANSI SQL, the corresponding basic template is matched according to the operation type, the field mapping set is injected to generate the field list of the SELECT clause, the condition expression tree is parsed to generate the WHERE clause of nested Boolean expressions, and when the execution strategy is DIFF_SYNC, the FOR UPDATE clause is automatically appended to ensure transaction isolation.

[0013] The multi-stage syntax processor includes a lexical analysis layer that splits SQL into a token stream, a syntax construction layer that generates hierarchical AST nodes, and a dialect conversion layer that implements pagination logic rewriting, function mapping replacement, and transaction semantic adaptation through the Visitor pattern.

[0014] Optionally, the multi-source heterogeneous data access layer supports parallel access of business system libraries, data tables, and real-time streaming data. The metadata registry center defines data pattern mapping rules to convert the raw data into a standardized model table containing tag key-value pairs, time window identifiers, and confidence coefficients.

[0015] The runtime context building engine analyzes task complexity through resource estimation algorithms, dynamically allocates CPU and memory resources, and the injected algorithm parameter templates support multiple feature modeling algorithms such as linear regression and decision trees. The batch fault-tolerant retry mechanism sets retry thresholds and exponential backoff strategies.

[0016] The hierarchical processing engine's streaming channels use a stateful computing model to achieve second-level data aggregation, while the batch channels use an incremental computing model to process only newly added and changed data.

[0017] Optionally, the method further includes:

[0018] Record the entire process of tag generation, including the original data source identifier, dimension mapping relationship, calculation rule version, intermediate result set identifier, and final tag ID;

[0019] By associating the input and output data of each processing node with metadata logs, a visual lineage map is formed, which supports tracing back from the final label to the original data and all intermediate processing steps;

[0020] Bloodline data and tag data are stored synchronously to retain historical bloodline records and meet the data traceability requirements in compliance audits.

[0021] Optionally, the method further includes:

[0022] The tag results are validated using JSON Schema to ensure that the field types, value ranges, and formats conform to business specifications.

[0023] The confidence coefficient is calculated by weighting data integrity, consistency, and timeliness. The weighting coefficients can be customized based on business scenarios.

[0024] When the confidence level of a tag is lower than a preset threshold, it is automatically marked as an abnormal tag and a verification process is triggered, including automatic verification such as data re-retrieval and rule re-execution, as well as manual verification that provides an interface for manual review.

[0025] Optionally, the method further includes:

[0026] Real-time tags are written to the hash structure of the Redis cluster, and an expiration time is set to balance performance and storage costs; batch calculation results are persisted to a distributed storage system, supporting adaptation to multiple storage engines such as HDFS and BlackHouse;

[0027] Push notifications are triggered via HTTP callback interfaces based on tag change types. The push content includes tag ID, associated business entity identifier, change time, and confidence level.

[0028] It receives feedback data from business systems to optimize metadata mapping rules and computing resource allocation algorithms, forming a closed-loop iteration of data processing.

[0029] According to another aspect of the present invention, a tag stream batch calculation and orchestration assembly apparatus based on a trusted data space is provided, characterized in that it comprises:

[0030] The tag arrangement and assembly unit is used to build a full-domain business data asset library based on multidimensional data modeling and dynamic rule engine; it extracts key business entities through dimensional modeling to build dimensional models, selects unique business attribute fields as dimensional primary keys and quantitative analysis indicators as measurement items; it uses a declarative configuration interface to define multi-level operation logic of basic operations, conditional operations and compound operations, and uses JSON Schema to realize version management of rule parameter templates;

[0031] The rule transformation unit is used to build a KPI rule metadata table containing dimension identifiers, metric identifiers, compound expressions, and execution strategies based on the structured configuration parsing and syntax tree reconstruction mechanism; it encapsulates operation types, field mapping sets, and condition expression trees through the Form intermediate object, and generates standard ANSI SQL through the Velocity template engine; it uses ANTLR v4 to build a multi-stage syntax processor to adapt to the target database dialect, and generates the optimal execution plan through logical optimization, physical optimization, and pre-compilation verification;

[0032] The streaming batch computing unit is used to build a streaming-batch collaborative processing architecture. It achieves data standardization and transformation through the metadata registry center of the multi-source heterogeneous data access layer. Based on the runtime context, the engine dynamically loads the dimension mapping table and feature encoding library, allocates computing resources, and pre-configures a fault-tolerant retry mechanism. The hierarchical processing engine executes streaming real-time stateful aggregation and batch incremental computing respectively. It generates labels with lineage through dynamic rule parsing, and outputs them using a hierarchical storage strategy after standardized quality control.

[0033] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0034] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the tag stream batch calculation and orchestration assembly method based on trusted data space according to any embodiment of the present invention.

[0035] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the tag stream batch calculation and orchestration assembly method based on a trusted data space as described in any embodiment of the present invention.

[0036] The solution of this invention is based on a trusted data space foundation. Through a decoupling mechanism between logical planning and physical execution, business indicator logic is abstracted into reusable operators. Combined with a unified resource scheduler for streaming and batch processing, the optimal allocation of computing tasks is achieved. At the same time, an incremental state management engine is introduced to ensure data consistency in mixed streaming and batch processing scenarios, and to achieve millisecond-level response tag query services. Compared with traditional solutions, this effectively reduces resource consumption and supports real-time tag production and second-level updates under large data volumes.

[0037] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of a tag stream batch calculation and orchestration assembly method based on a trusted data space provided in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the structure of a tag stream batch calculation and orchestration assembly device based on a trusted data space provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the tag stream batch calculation and arrangement assembly method based on trusted data space according to the embodiments of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0043] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] like Figure 1 As shown, this embodiment of the invention provides a method for batch calculation and orchestration of tag streams based on a trusted data space. This method may include the following steps:

[0045] S110. Based on multidimensional data modeling and dynamic rule engine, construct a full-domain business data asset library; extract key business entities through dimensional modeling to construct dimensional models, select unique business attribute fields as dimensional primary keys and quantitative analysis indicators as measurement items; adopt a declarative configuration interface to define multi-level operation logic of basic operations, conditional operations and compound operations, and implement version management of rule parameter templates through JSON Schema.

[0046] Multidimensional data modeling takes a business perspective, organizing data according to multiple dimensions (such as user, product, time, and geographic location) rather than the traditional table structure perspective, supporting multidimensional cross-analysis. The dynamic rule engine is a core component that can be flexibly configured and updated in real time for calculating rules, adapting to the tag calculation needs of different business scenarios without modifying the underlying code. JSONSchema is a specification for defining JSON data structures, clearly defining the field types, value ranges, and required fields of rule parameters, ensuring the standardization of rule templates.

[0047] Multidimensional data modeling assigns dimensional labels to data from different sources, giving dispersed data a unified classification standard. A dynamic rule engine pre-embeds basic rules for data filtering, format conversion, and consistency verification, ensuring that the data entering the database meets subsequent calculation requirements. This solves the problems of traditional data being scattered, inconsistent in format, and difficult to trace, achieving unified data storage, unified tracking, and end-to-end traceability.

[0048] Business entities are the core analytical objects in an enterprise's business; dimensional models are structured data frameworks built around business entities, such as user dimensional models containing basic user information, behavioral information, and transaction information; dimensional primary keys are fields that uniquely identify business entities, ensuring that data is not duplicated or confused; measures are quantifiable and analyzable indicator data, and are the core material for tag calculation. By structuring data according to business logic, different dimensions and measures can be flexibly combined to generate tags.

[0049] A declarative configuration interface is used to define multi-level operation logic, including basic operations, conditional operations, and compound operations. Version management of rule parameter templates is achieved through JSON Schema. The declarative configuration interface eliminates the need for users to write code; rules are configured visually through drag-and-drop, selection, and fill-in-the-blank methods. The three-level operation logic includes basic operations, conditional operations, and compound operations. JSON Schema version management defines the standard structure of rule templates; a version number is generated for each rule modification, supporting version rollback and comparison to avoid rule confusion.

[0050] The purpose of this feature is to solve the problems of low efficiency and cumbersome modification of traditional hard-coded rules. Business personnel can configure rules independently, update them in real time without interrupting the data processing flow, and greatly improve the flexibility and development efficiency of tag generation.

[0051] In this embodiment of the invention, basic operations achieve multi-dimensional data summarization through sliding window aggregation, cumulative operations, deduplication operations, and COUNT_DISTINCT statistics; conditional operations support combined filtering rules of time interval BETWEEN, numerical comparison, regular expression REGEXP_MATCH, and enumeration value matching; composite operations support multi-level nested expression definitions, including nested combinations of arithmetic and logical operations.

[0052] In the three-level operation logic, basic operations achieve multi-dimensional data summary statistics through sliding window aggregation, accumulation, deduplication, and COUNT_DISTINCT statistics; conditional operations can combine time intervals BETWEEN, numerical comparisons (>, <, =, ≠, etc.), regular expressions REGEXP_MATCH, and enumerated value matching rules to accurately filter and select data; compound operations allow the definition of multi-level nested expressions, supporting nested combinations of arithmetic operations (+, -, ×, ÷, etc.) and logical operations (AND, OR, NOT, etc.). Together, these three functions cover label calculation needs from simple statistics to complex logic, adapting to data processing requirements in multiple scenarios.

[0053] S120: Based on the structured configuration parsing and syntax tree reconstruction mechanism, a KPI rule metadata table containing dimension identifiers, metric identifiers, compound expressions, and execution strategies is constructed; operation types, field mapping sets, and condition expression trees are encapsulated through the Form intermediate object, and standard ANSI SQL is generated through the Velocity template engine; a multi-stage syntax processor is built using ANTLR v4 to adapt to the target database dialect, and the optimal execution plan is generated through logical optimization, physical optimization, and pre-compilation verification.

[0054] In this embodiment of the invention, the operation type of the Form intermediate object is mapped to the CRUD semantics of SELECT, INSERT, UPDATE, and DELETE. The field mapping set uses a hash structure to store the one-to-one correspondence between the source field and the target field. The condition expression tree organizes the AND / OR logical operators in a binary tree structure.

[0055] When generating standard ANSI SQL, the corresponding basic template is matched according to the operation type, the field mapping set is injected to generate the field list of the SELECT clause, the condition expression tree is parsed to generate the WHERE clause of nested Boolean expressions, and when the execution strategy is DIFF_SYNC, the FOR UPDATE clause is automatically appended to ensure transaction isolation.

[0056] The multi-stage syntax processor includes a lexical analysis layer that splits SQL into token streams, a syntax construction layer that generates hierarchical AST nodes, a dialect conversion layer that implements pagination logic rewriting, function mapping replacement, and transaction semantic adaptation through the Visitor pattern.

[0057] The rule conversion module solves the problems of traditional business rules being difficult to execute across databases, low SQL generation efficiency, and poor execution performance by standardizing rule storage, encapsulating intermediate layer logic, generating standard SQL, cross-database adaptation, and optimizing execution plans. It achieves automated and standardized conversion of business rules into executable statements in the database.

[0058] Structured storage of business rules consolidates rules scattered across different scenarios into a standardized KPI rule metadata table. The table includes the following fields:

[0059] Dimension identifier: The analysis dimension corresponding to the rule, such as "user dimension" or "product dimension", and the primary key of the associated dimension model;

[0060] Measurement indicators: Quantitative metrics involved in the rules, such as "access frequency" and "housing provident fund payment amount", which are related to the measurement items of the dimension model;

[0061] Compound expressions: The specific calculation logic, such as "SUM access frequency" and "payment amount > 5000", is the core of the rule;

[0062] Execution strategy: The way rules are executed, such as "DIFF_SYNC incremental synchronization" or "FULL_SYNC full synchronization", which determines the transaction processing logic.

[0063] Structured configuration parsing breaks down unstructured rules configured by business personnel into the aforementioned standardized fields and syntax tree reconstruction mechanism, pre-deconstructing compound expressions into machine-recognizable logic trees.

[0064] The intermediate layer logic is encapsulated using a Form intermediate object and the Velocity template engine to generate standard ANSI SQL. The Form intermediate object acts as a bridge between rule logic and SQL statements, encapsulating the fields of the KPI rule metadata table into a logical structure that can be directly processed by the machine. It consists of three parts:

[0065] Operation type: Maps the CRUD semantics of SQL (SELECT query / INSERT insert / UPDATE update / DELETE delete, such as using SELECT to count labels);

[0066] Field mapping set: Uses a hash structure to store the correspondence between source data fields and target label fields;

[0067] Conditional expression tree: Organizes filtering rules into a binary tree structure, which facilitates subsequent parsing into the SQL WHERE clause.

[0068] The Velocity template engine automatically populates the logical structure of the Form intermediate object into a preset SQL template, generating standard ANSI SQL (the universal standard syntax of SQL). For example: selecting "SELECT template" by OperationType; filling the field mapping relationships of FieldMap into the SELECT clause; parsing the binary tree logic of ConditionTree into nested Boolean expressions in the WHERE clause; and automatically appending a FOR UPDATE clause if the execution strategy is DIFF_SYNC, ensuring transaction isolation. This avoids the tediousness and errors of manually writing SQL, achieving automated and standardized SQL generation without relying on a specific database.

[0069] The ANTLR v4 syntax processor and multi-stage optimization were designed to address the issues of standard SQL's inability to directly adapt to all databases and its low execution efficiency, ultimately generating execution plans that can run efficiently on the target database. The core of the ANTLR v4 multi-stage syntax processor is the conversion of standard ANSI SQL into the target database's specific dialect SQL. The lexical analysis layer breaks down standard SQL into its smallest syntactic units, such as recognizing reserved words like "SELECT" and "WHERE," field names, and values. The syntax construction layer assembles the token stream into a hierarchical abstract syntax tree according to SQL syntax rules. The dialect conversion layer traverses the abstract syntax tree using a visitor pattern and performs adaptation conversions based on the characteristics of the target database: rewriting pagination logic, such as standard pagination versus MySQL's LIMIT and Oracle's ROWNUM; replacing function mappings, such as SUBSTR() versus MySQL's SUBSTRING() and SQL Server's LEFT(); adapting transaction semantics, such as appending NOWAIT or SKIPLOCKED to generate dialect SQL based on the database isolation level, optimizing execution efficiency and avoiding errors; merging adjacent filter conditions and deleting redundant field projections; injecting execution hints based on the target database characteristics to improve query speed; and performing syntax checks through the JDBC PreparedStatement interface, combined with EXPLAIN PLAN analysis of the execution path. If issues such as full table scans or missing indexes are found, warning logs are generated to ensure optimal execution plans.

[0070] S130. Construct a streaming-batch collaborative processing architecture, and realize data standardization transformation through the metadata registration center of the multi-source heterogeneous data access layer; build an engine based on runtime context to dynamically load dimension mapping tables and feature encoding libraries, allocate computing resources and pre-configure fault tolerance and retry mechanisms; adopt a hierarchical processing engine to execute streaming real-time stateful aggregation and batch incremental computing respectively, generate labels with lineage through dynamic rule parsing, and output them using a hierarchical storage strategy after standardized quality control.

[0071] First, data pattern mapping rules are defined through the metadata registry center of the multi-source heterogeneous data access layer, uniformly converting raw data from different sources and formats into a standardized model table containing tag key-value pairs, time window identifiers, and confidence coefficients. Then, the runtime context engine dynamically loads the dimension mapping table and feature encoding library, intelligently allocates computing resources through resource estimation algorithms, and pre-configures fault-tolerant retry mechanisms to complete the initialization of the computing environment. Subsequently, a hierarchical processing engine is adopted, with the streaming channel performing real-time stateful aggregation and the batch channel using an incremental computing model to process real-time and offline data respectively. Through dynamic rule parsing, business logic is compiled into executable instructions, generating tags that record the entire chain lineage. Finally, after standardized quality control through JSON Schema structured verification and encryption strategies, real-time tags are written to a low-latency storage engine and batch results are persisted to a distributed database according to a hierarchical storage strategy, realizing collaborative and efficient processing and high-quality output of streaming and batch data.

[0072] In this embodiment of the invention, the multi-source heterogeneous data access layer supports parallel access of business system libraries, data tables, and real-time streaming data. The metadata registry center defines data pattern mapping rules to convert the raw data into a standardized model table containing tag key-value pairs, time window identifiers, and confidence coefficients.

[0073] The runtime context building engine analyzes task complexity through resource estimation algorithms, dynamically allocates CPU and memory resources, and the injected algorithm parameter templates support multiple feature modeling algorithms such as linear regression and decision trees. The batch fault-tolerant retry mechanism sets retry thresholds and exponential backoff strategies.

[0074] The hierarchical processing engine's streaming channels use a stateful computing model to achieve second-level data aggregation, while the batch channels use an incremental computing model to process only newly added and changed data.

[0075] The multi-source heterogeneous data access layer supports parallel access to different types of data sources, such as business system libraries, data tables, and real-time streaming data, avoiding single access bottlenecks. On the other hand, it predefines data pattern mapping rules through a metadata registry center, uniformly converting raw data with different formats and semantics into standardized model tables containing tag key-value pairs, time window identifiers, and confidence coefficients, ensuring that the calculated data is from the same source and has a unified format.

[0076] The runtime context building engine, acting as the dynamic configuration center of the computing environment, first analyzes the magnitude and complexity of the current data processing task through resource estimation algorithms, and intelligently allocates hardware resources such as CPU and memory to avoid resource idleness or insufficiency. Then, it injects parameter templates that support various feature modeling algorithms such as linear regression and decision trees to adapt to the computing needs of different labels. At the same time, it has a pre-built batch fault-tolerant retry mechanism, which prevents task interruption due to temporary anomalies by setting retry thresholds and exponential backoff strategies, thus ensuring processing stability.

[0077] The hierarchical processing engine adopts a dual-channel design to precisely adapt to different data scenarios: the streaming channel uses a stateful computing model to continuously receive real-time streaming data and perform second-level aggregation calculations to meet the real-time response requirements of the business; the batch channel uses an incremental computing model, which does not require repeated processing of all historical data, but only performs calculations on newly added or changed data, which greatly reduces computing overhead and improves batch processing efficiency. The two work together to achieve the goal of fast real-time data response and high-efficiency batch data processing.

[0078] In embodiments of the present invention, the method may further include:

[0079] Record the entire process of tag generation, including the original data source identifier, dimension mapping relationship, calculation rule version, intermediate result set identifier, and final tag ID;

[0080] By associating the input and output data of each processing node with metadata logs, a visual lineage map is formed, which supports tracing back from the final label to the original data and all intermediate processing steps;

[0081] Bloodline data and tag data are stored synchronously to retain historical bloodline records and meet the data traceability requirements in compliance audits.

[0082] Recording the entire tag generation process involves capturing uninterrupted information along the complete tag generation path from source to result. The five key information categories each serve a specific purpose: Original data source identifier accurately marks the initial source of the tag data, avoiding ambiguity regarding data origin; Dimension mapping relationship records the association rules between source data fields and standard model fields, and between business entities and dimensions, ensuring the semantic traceability of fields; Calculation rule version binds to the version of the computational logic used in tag generation, clarifying the logical basis for tag calculation; Intermediate result set identifier assigns a unique identifier to each aggregation, filtering, and transformation process during batch processing, tracking the intermediate processing status of tag generation; and the final tag ID assigns a globally unique identifier to the generated structured tag, achieving precise binding between the tag and the business entity.

[0083] By linking the input and output data of each processing node through metadata logs, the essence is to use metadata logs as a link connecting the links. It records the input data identifier, processing logic, and output data identifier of each processing node in real time, thus connecting the scattered processing links into a complete link. The resulting visualized lineage graph presents the generation path of raw data, intermediate results, and final tags in intuitive forms such as tree structures and networks, allowing both technical and business personnel to quickly understand the origin of the tags. The core objective is to support reverse tracing from the final tag back to the raw data and all intermediate processing links. Users only need to enter the final tag ID to progressively query its corresponding intermediate result set, calculation rule version, and dimension mapping relationship, ultimately locating the original data source and access time. This completely solves the black-box problem of traditional tag generation links, achieving full-process reversible traceability.

[0084] Phylogenetic data and tag data are stored synchronously, emphasizing their consistency throughout their lifecycle. Phylogenetic data is generated and written to the storage medium simultaneously with tag data, ensuring that tags and corresponding pedigree information are not disconnected. Retaining historical pedigree records, based on synchronous storage, preserves the pedigree changes corresponding to each tag update, rule iteration, and data recalculation, supporting historical status retrospection. These designs ultimately address compliance audit requirements by providing a complete data evidence chain that is traceable in origin, process, and destination, accurately responding to the stringent data traceability requirements in compliance audits, ensuring that the generation and application of tag data comply with industry regulatory standards.

[0085] In embodiments of the present invention, the method may further include:

[0086] The tag results are validated using JSON Schema to ensure that the field types, value ranges, and formats conform to business specifications.

[0087] The confidence coefficient is calculated by weighting data integrity, consistency, and timeliness. The weighting coefficients can be customized based on business scenarios.

[0088] When the confidence level of a tag is lower than a preset threshold, it is automatically marked as an abnormal tag and a verification process is triggered, including automatic verification such as data re-retrieval and rule re-execution, as well as manual verification that provides an interface for manual review.

[0089] The core of using JSON Schema to perform structural validation on the tag results lies in leveraging the standardized definition capabilities of JSON Schema to establish a unified structural specification for the tag results. Each field in the tag must conform to preset types, value ranges, and format standards. Automated validation identifies and eliminates issues such as incorrect field types, out-of-range values, and inconsistent formatting, ensuring that the generated tags are highly consistent in structure and fully meet the data format requirements of the business system. This provides a reliable foundation of data for subsequent data analysis and business applications.

[0090] The confidence coefficient is a core indicator for evaluating the reliability of tagged data. Its calculation logic revolves around three key dimensions: data integrity, consistency, and timeliness. Data integrity focuses on whether the original data required for tag generation is missing; data consistency focuses on whether there are logical contradictions in the data; and data timeliness focuses on whether the original data is the latest and most valid data. By assigning different weights to these three dimensions and performing a weighted calculation, the confidence coefficient of the tag can be obtained. Furthermore, the weights can be flexibly customized according to business scenarios. For example, in compliance audit scenarios in the financial sector, the consistency weight can be increased, while in real-time recommendation scenarios, the timeliness weight can be increased, ensuring that the confidence coefficient assessment aligns with the specific business's core requirements for data quality.

[0091] When a tag's confidence level falls below a preset threshold, it is automatically marked as an anomalous tag, triggering a dual verification process to ensure data quality: First, an automatic verification mechanism is activated, automatically re-fetching data and re-executing rules to address potential causes of low confidence levels, attempting to correct the anomaly through automated means. Second, a manual review interface is provided, allowing business or technical personnel to manually intervene when automatic verification fails to resolve the anomaly. They can then view the full-link lineage information generated by the tag and details of the low-confidence dimensions, manually determine the cause of the anomaly, and complete the correction. Through automatic and manual dual verification, the impact of anomalous tags on subsequent business applications is minimized.

[0092] In embodiments of the present invention, the method may further include:

[0093] Real-time tags are written to the hash structure of the Redis cluster, and an expiration time is set to balance performance and storage costs; batch calculation results are persisted to a distributed storage system, supporting adaptation to multiple storage engines such as HDFS and BlackHouse;

[0094] Push notifications are triggered via HTTP callback interfaces based on tag change types. The push content includes tag ID, associated business entity identifier, change time, and confidence level.

[0095] It receives feedback data from business systems to optimize metadata mapping rules and computing resource allocation algorithms, forming a closed-loop iteration of data processing.

[0096] Writing real-time tags to the Hash structure of the Redis cluster leverages Redis's low-latency in-memory database capabilities to meet real-time business needs. The Hash structure efficiently stores key-value pairs, supports millisecond-level query responses, and satisfies scenarios with extremely high response speed requirements, such as real-time recommendations and real-time risk control. Setting expiration times is a key design element in balancing performance and storage costs: real-time tags are highly time-sensitive and do not require long-term retention. The automatic expiration cleanup mechanism prevents invalid historical tags from consuming Redis memory resources, ensuring that tag data is up-to-date when used by the business while reducing the hardware cost of in-memory storage—a perfect fit for the design principles of low-latency storage and resource optimization.

[0097] Batch calculation results are persisted to a distributed storage system and support multiple storage engines such as HDFS and ClickHouse. This is a dedicated storage solution for the characteristics of batch data: batch calculation results are usually large in volume and accessed less frequently than real-time tags, but need to be retained for long-term use in scenarios such as offline analysis and compliance auditing. Distributed storage systems have the advantages of high capacity and high reliability, which can meet the persistence requirements of massive amounts of data. Support for multiple storage engines improves the flexibility of the solution. Different enterprises have different existing storage architectures, and the solution can be integrated without reconstructing the underlying storage, reducing the implementation cost. At the same time, it adapts to the query requirements of different business scenarios, echoing the design goal of multimodal output.

[0098] Triggering push notifications based on tag change type via HTTP callback interfaces essentially builds an efficient communication channel between the tagging system and business systems. HTTP callback interfaces offer strong cross-system compatibility and reliable transmission, adapting to the access needs of most business systems. Triggering push notifications by tag change type avoids network redundancy caused by indiscriminate data transmission, allowing business systems to receive only targeted change information. Furthermore, the push notification content, including the tag ID, associated business entity identifier, change time, and confidence coefficient, provides business systems with complete decision-making support, enabling them to quickly respond to tag changes and realize the real-time value of tag data, thus meeting the core requirement of pushing tag change events to business systems.

[0099] Receiving feedback data from business systems and using it to optimize metadata mapping rules and computing resource allocation algorithms is a key step in building a closed-loop iterative data processing system. The feedback data from business systems encompasses various types of practical information; this feedback from real-world application scenarios can accurately pinpoint existing system problems. Using this feedback data to optimize metadata mapping rules and computing resource allocation algorithms allows the system to continuously improve itself. This closed-loop model avoids the decline in adaptability caused by unchanging rules and algorithms, ensuring that the solution consistently meets business needs.

[0100] like Figure 2As shown, this embodiment of the invention provides a tag stream batch calculation and orchestration assembly device based on a trusted data space. The device includes:

[0101] The tag arrangement and assembly unit 210 is used to build a full-domain business data asset library based on multidimensional data modeling and dynamic rule engine; it extracts key business entities through dimensional modeling to build dimensional models, selects business unique attribute fields as dimensional primary keys and quantitative analysis indicators as measurement items; it uses a declarative configuration interface to define multi-level operation logic of basic operations, conditional operations and compound operations, and uses JSON Schema to realize version management of rule parameter templates;

[0102] The rule transformation unit 220 is used to construct a KPI rule metadata table containing dimension identifiers, metric identifiers, compound expressions, and execution strategies based on a structured configuration parsing and syntax tree reconstruction mechanism; it encapsulates operation types, field mapping sets, and condition expression trees through a Form intermediate object, and generates standard ANSI SQL through the Velocity template engine; it uses ANTLR v4 to build a multi-stage syntax processor to adapt to the target database dialect, and generates the optimal execution plan through logical optimization, physical optimization, and pre-compilation verification;

[0103] The streaming batch computing unit 230 is used to build a streaming-batch collaborative processing architecture. It realizes data standardization transformation through the metadata registry center of the multi-source heterogeneous data access layer. Based on the runtime context, the engine dynamically loads the dimension mapping table and feature encoding library, allocates computing resources and pre-sets a fault-tolerant retry mechanism. The hierarchical processing engine executes streaming real-time stateful aggregation and batch incremental computing respectively. It generates labels with lineage through dynamic rule parsing, and outputs them using a hierarchical storage strategy after standardized quality control.

[0104] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the tag stream batch calculation and orchestration assembly apparatus based on trusted data space. In other embodiments of the present invention, the tag stream batch calculation and orchestration assembly apparatus based on trusted data space may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0105] The information interaction and execution process between the various units in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.

[0106] Figure 3A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0107] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0108] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0109] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as tag-based batch computation and orchestration assembly methods based on trusted data spaces.

[0110] In some embodiments, the tag stream batch calculation and orchestration assembly method based on trusted data space can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the tag stream batch calculation and orchestration assembly method based on trusted data space described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the tag stream batch calculation and orchestration assembly method based on trusted data space by any other suitable means (e.g., by means of firmware).

[0111] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0113] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0115] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0116] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0117] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0118] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for batch calculation and orchestration of tag streams based on trusted data space, characterized in that, include: Based on multidimensional data modeling and dynamic rule engine, construct a full-domain business data asset library; Dimensional models are constructed by extracting key business entities through dimensional modeling, selecting unique business attribute fields as dimensional primary keys and quantitative analysis indicators as measurement items; a declarative configuration interface is used to define multi-level operation logic for basic operations, conditional operations, and compound operations, and version management of rule parameter templates is achieved through JSON Schema. Based on the structured configuration parsing and syntax tree reconstruction mechanism, a KPI rule metadata table containing dimension identifiers, metric identifiers, compound expressions, and execution strategies is constructed; operation types, field mapping sets, and condition expression trees are encapsulated through the Form intermediate object, and standard ANSI SQL is generated through the Velocity template engine; a multi-stage syntax processor is built using ANTLR v4 to adapt to the target database dialect, and the optimal execution plan is generated through logical optimization, physical optimization, and pre-compilation verification. A collaborative processing architecture for stream and batch processing is constructed. Data standardization and transformation are achieved through the metadata registry center of the multi-source heterogeneous data access layer. An engine is built based on the runtime context to dynamically load dimension mapping tables and feature encoding libraries, allocate computing resources, and pre-configure fault tolerance and retry mechanisms. A hierarchical processing engine is adopted to execute streaming real-time stateful aggregation and batch incremental computing respectively. Labels with lineage are generated through dynamic rule parsing, and after standardized quality control, they are output using a hierarchical storage strategy.

2. The method according to claim 1, characterized in that, The basic operations achieve multi-dimensional data summarization through sliding window aggregation, cumulative operations, deduplication operations, and COUNT_DISTINCT statistics; the conditional operations support combined filtering rules of time interval BETWEEN, numerical comparison, regular expression REGEXP_MATCH, and enumerated value matching; the composite operations support multi-level nested expression definitions, including nested combinations of arithmetic and logical operations.

3. The method according to claim 1, characterized in that, The operation type of the Form intermediate object is mapped to CRUD semantics of SELECT, INSERT, UPDATE, and DELETE. The field mapping set uses a hash structure to store the one-to-one correspondence between the source field and the target field. The condition expression tree organizes the AND / OR logical operators in a binary tree structure. When generating standard ANSI SQL, the corresponding basic template is matched according to the operation type, the field mapping set is injected to generate the field list of the SELECT clause, the condition expression tree is parsed to generate the WHERE clause of nested Boolean expressions, and when the execution strategy is DIFF_SYNC, the FOR UPDATE clause is automatically appended to ensure transaction isolation. The multi-stage syntax processor includes a lexical analysis layer that splits SQL into a token stream, a syntax construction layer that generates hierarchical AST nodes, and a dialect conversion layer that implements pagination logic rewriting, function mapping replacement, and transaction semantic adaptation through the Visitor pattern.

4. The method according to claim 1, characterized in that, The multi-source heterogeneous data access layer supports parallel access of business system libraries, data tables, and real-time streaming data. The metadata registry center defines data pattern mapping rules to convert raw data into a standardized model table containing tag key-value pairs, time window identifiers, and confidence coefficients. The runtime context building engine analyzes task complexity through resource estimation algorithms, dynamically allocates CPU and memory resources, and the injected algorithm parameter templates support multiple feature modeling algorithms such as linear regression and decision trees. The batch fault-tolerant retry mechanism sets retry thresholds and exponential backoff strategies. The hierarchical processing engine's streaming channels use a stateful computing model to achieve second-level data aggregation, while the batch channels use an incremental computing model to process only newly added and changed data.

5. The method according to claim 1, characterized in that, The method further includes: Record the entire process of tag generation, including the original data source identifier, dimension mapping relationship, calculation rule version, intermediate result set identifier, and final tag ID; By associating the input and output data of each processing node with metadata logs, a visual lineage map is formed, which supports tracing back from the final label to the original data and all intermediate processing steps; Bloodline data and tag data are stored synchronously to retain historical bloodline records and meet the data traceability requirements in compliance audits.

6. The method according to claim 1, characterized in that, The method further includes: The tag results are validated using JSON Schema to ensure that the field types, value ranges, and formats conform to business specifications. The confidence coefficient is calculated by weighting data integrity, consistency, and timeliness. The weighting coefficients can be customized based on business scenarios. When the confidence level of a tag is lower than a preset threshold, it is automatically marked as an abnormal tag and a verification process is triggered, including automatic verification such as data re-retrieval and rule re-execution, as well as manual verification that provides an interface for manual review.

7. The method according to claim 1, characterized in that, The method further includes: Real-time tags are written to the hash structure of the Redis cluster, and an expiration time is set to balance performance and storage costs; batch calculation results are persisted to a distributed storage system, supporting adaptation to multiple storage engines such as HDFS and BlackHouse; Push notifications are triggered via HTTP callback interfaces based on tag change types. The push content includes tag ID, associated business entity identifier, change time, and confidence level. It receives feedback data from business systems to optimize metadata mapping rules and computing resource allocation algorithms, forming a closed-loop iteration of data processing.

8. A tag stream batch calculation and orchestration assembly device based on trusted data space, characterized in that, include: The tag arrangement and assembly unit is used to build a full-domain business data asset library based on multi-dimensional data modeling and dynamic rule engine; Dimensional models are constructed by extracting key business entities through dimensional modeling, selecting unique business attribute fields as dimensional primary keys and quantitative analysis indicators as measurement items; a declarative configuration interface is used to define multi-level operation logic for basic operations, conditional operations, and compound operations, and version management of rule parameter templates is achieved through JSON Schema. The rule transformation unit is used to build a KPI rule metadata table containing dimension identifiers, metric identifiers, compound expressions, and execution strategies based on the structured configuration parsing and syntax tree reconstruction mechanism; it encapsulates operation types, field mapping sets, and condition expression trees through the Form intermediate object, and generates standard ANSI SQL through the Velocity template engine; it uses ANTLR v4 to build a multi-stage syntax processor to adapt to the target database dialect, and generates the optimal execution plan through logical optimization, physical optimization, and pre-compilation verification; The streaming batch computing unit is used to build a streaming-batch collaborative processing architecture. It achieves data standardization and transformation through the metadata registry center of the multi-source heterogeneous data access layer. Based on the runtime context, the engine dynamically loads the dimension mapping table and feature encoding library, allocates computing resources, and pre-configures a fault-tolerant retry mechanism. The hierarchical processing engine executes streaming real-time stateful aggregation and batch incremental computing respectively. It generates labels with lineage through dynamic rule parsing, and outputs them using a hierarchical storage strategy after standardized quality control.

9. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the tag stream batch calculation and orchestration assembly method based on any one of claims 1-7.

10. A computer-readable medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the tag stream batch calculation and orchestration assembly method based on any one of claims 1-7.

Images