Method for automatic implantation and execution of security obligations in data processing pipelines

Abstract

The application discloses a method for automatically implanting and executing security obligations in a data processing pipeline. In the obligation analysis stage of the data processing pipeline, an obligation set query is initiated to a security semantic knowledge base for a field list processed by the pipeline, and an obligation execution plan is automatically generated according to a query result, the plan including rejection field exclusion, desensitization field grouping, anonymization field grouping and obligation execution order. In the pipeline execution process, the following are executed in sequence according to the obligation execution plan: excluding rejection fields from the output, performing desensitization processing on desensitization fields according to an obligation parameter, performing k-anonymity processing on a quasi-identifier field (k value defined by a knowledge base obligation parameter), and embedding a batch watermark at the data set level. After the obligation execution is completed, an audit log is atomically written within the same transaction boundary, and the audit log and an access layer audit log adopt a unified format and are continuously connected across stages through an SM3 hash chain. The application supports implanting an existing pipeline framework in the form of a Flink operator, a Spark operator or a DataX plug-in, and also supports running in an independent engine mode. The application also provides a delay compensation execution mode, allowing historical inventory data to be batched for obligation compensation processing before being shared externally. The application solves the systematic problems of manual configuration of security obligations in a data processing pipeline, pipeline state and access state audit chain rupture, inconsistency of obligation execution in batch and stream scenarios, and lack of security processing for unstructured data.

Description

Technical Field

[0001] This invention relates to the field of data security and data engineering, specifically to a method and system for automatically embedding and executing data security obligations in ETL batch processing pipelines, real-time stream processing pipelines, and data integration pipelines, which falls under the category of data lifecycle security management technology. Background Technology

[0002] Data processing pipelines (ETL / ELT pipelines, stream processing pipelines, data integration pipelines) are the core channels through which data flows from source systems to data warehouses, data lakes, and various data products in modern data systems. A large number of data cleaning, transformation, aggregation, and format conversion operations occur within these pipelines, and these operations are also critical moments when data security obligations must be fulfilled.

[0003] However, existing technical solutions have the following systemic limitations at the data processing pipeline level: First, existing ETL tools (DataX, Kettle, Informatica, etc.) lack data security semantic awareness capabilities and cannot automatically determine what security processing should be performed based on the business semantics of fields. Security operations such as de-identification and anonymization must be manually configured by data engineers, and the configuration quality depends entirely on individual skills, and cannot be automatically updated when fields are added or business changes occur; Second, the security processing within the pipeline is completely disconnected from the security control at the access layer. The data processing records generated during pipeline execution use the ETL tool's proprietary log format, which is incompatible with the audit logs at the data access layer, resulting in a break in the compliance proof chain at the pipeline stage. Third, batch processing security and real-time stream processing security require different technical frameworks and lack a unified obligation execution abstraction layer, resulting in inconsistent security policies in batch processing and stream processing scenarios. Fourth, existing data pipeline tools have almost no security processing capabilities for unstructured data such as DICOM medical images, and the protection of patient privacy during the transfer of medical images relies entirely on manual intervention. Summary of the Invention

[0004] This invention provides a method for automatically embedding and executing security obligations in a data processing pipeline. By automatically querying a semantic knowledge base at pipeline nodes to obtain an obligation set, generating an obligation execution plan, executing the obligation processing according to the plan, and atomically writing to a unified format audit log, it solves the fundamental problems of manual configuration of security obligations, broken audit chains, and inability to process unstructured data in data processing pipelines.

[0005] This invention also provides a delayed compensation execution mode, which allows historical data to complete obligation compensation through batch static obligation execution before being shared externally, thus realizing the design principle that "data security obligations do not necessarily have to be executed immediately when the data is generated, but must be completed before the data is shared".

[0006] The beneficial effects of this invention are: (1) Automatic derivation of obligations, eliminating manual configuration: The safety obligations in the pipeline are deduced by semantic knowledge base and automatically adapt when fields are added or obligation rules are changed, without the need for manual intervention; (2) Unified batch and stream execution: The same set of execution abstraction layers supports both batch processing (Spark operators) and stream processing (Flink operators) scenarios, and the consistency of security policies is uniformly guaranteed by the knowledge base; (3) Continuous audit chain across stages: The pipeline state audit log and the access state audit log adopt a unified format and are continuously connected through the SM3 hash chain. The compliance proof chain covers the complete path of data from pipeline flow to external service; (4) Native support for unstructured data: DICOM de-identification and document de-identification are built-in obligation executors, driven by obligation parameters in the semantic knowledge base, and managed in a unified manner with the obligation execution of structured data; (5) Delay compensation mechanism: Historical data can be batch compensated for before sharing, reducing the threshold for project initiation and protecting customers' existing data assets. Attached Figure Description

[0007] Figure 1 This is a diagram illustrating the overall architecture of the data processing pipeline safety obligation automatic implantation and execution method of the present invention. Figure 1 This paper demonstrates the five-step overall process of this invention. Step 1 (Pipeline Initialization & Obligation Query): The obligation parser performs a batch query of the security semantic knowledge base to obtain the set of obligations and execution parameters for all fields in the pipeline. The query results are cached locally. Step 2 (Obligation Execution Plan Generation): The obligation execution plan generator groups fields according to obligation type, determines the execution order (exclusion → desensitization → anonymization → unstructured obligation execution → watermarking), and generates a pipeline-level obligation execution plan. Step 3 (Pipeline Execution + Obligation Implantation): The obligation executor pool executes according to the plan, including excluding rejected fields from the output, performing masking processing on desensitized fields, performing k-anonymization using ARX, performing DICOM de-identification or document desensitization on unstructured data, and embedding a dataset-level watermark. Step 4 (Obligation Execution Failure Handling): Upon execution failure, data output is immediately stopped, the reason for failure is recorded, and an alarm is triggered, awaiting manual intervention. Step 5 (Atomic Generation of Audit Logs): A unified format of obligation execution audit logs is generated using atomic operations. The SM3 hash chain is continuous across nodes and layers, and a Kafka event stream is published for real-time consumption by the SOC.

[0008] Figure 2 Generate a flowchart for the execution plan of obligations. Figure 2 This demonstrates the logic behind generating the obligation execution plan. The left side shows the input: the query results for the obligation set, including example fields such as patient_name (MaskingObligation), diagnosis (Masking+kAnon), genetic_data (DenyAccessObligation), ct_image.dcm (DicomDeidentObligation), and visit_date (AuditLogObligation). The middle section shows the grouping process: fields are divided into exclusion groups (genetic_data), desensitization groups (patient_name, etc.), anonymization groups (diagnosis, etc.), and unstructured groups (ct_image.dcm, etc.) according to the obligation type. The right side shows the output: the sorted obligation execution plan, containing six execution steps: Step 1: Exclude denied access fields; Step 2: Desensitization; Step 3: k-anonymization; Step 4: Unstructured obligation execution; Step 5: Watermark embedding (on demand); Step 6: Audit log generation. The design of placing the denied fields first ensures that sensitive data does not enter subsequent processing stages.

[0009] Figure 3 This diagram illustrates two pipeline integration modes (framework plug-in mode A and independent engine mode B): Figure 3 This demonstrates an architecture (Pattern A) where the obligation executor is integrated into mainstream data processing frameworks as a plugin. The top layer comprises three frameworks: Apache Flink stream processing pipeline (integrated as Flink obligation execution operators / UDFs), Apache Spark batch processing pipeline (integrated as Spark obligation execution operators / UDFs), and DataX data synchronization tool (integrated as a DataXWriter plugin). These three frameworks share the same common obligation execution layer, which includes: an obligation parser (querying the P1 knowledge base and managing cache), an identity masking executor (ARX + rule-based identity masking), a DICOM processor (DCM4CHE for de-identification), a document processor (PDFBox + iText7), and an audit log generator (unified format + SM3 hash chain). This common obligation execution layer connects downwards to the security semantic knowledge base (P1, the unified obligation definition source) and the obligation execution audit library (PostgreSQL, with continuous SM3 hash chains across layers). Audit events from the audit library are published to the SOC semantic security operations center for real-time consumption via Kafka. Figure 3This section showcases the architecture of an independent obligation execution engine (Pattern B), suitable for scenarios where the customer environment lacks a unified data processing pipeline infrastructure. The left side features a data source connector, supporting five types of data sources: relational databases, data warehouses, data lakes, object storage, and PACS images. The center section houses the core engine, comprising five modules: a scheduler (batch / real-time scheduling, concurrency control, and breakpoint recovery), an obligation parser (querying the P1 knowledge base and caching obligation parameters), an executor pool (de-identification, anonymization, DICOM processing, document processing, and watermark embedding), a lineage tracker (data lineage recording and obligation transmission chain management), and an audit logger (unified format and SM3 hash chain maintenance). The right side features a target storage connector, supporting five types of target storage: data warehouse DWD layer, research databases, external shared packages, data lake archives, and PACS de-identified libraries.

[0010] Figure 4 This is a flowchart of the delay compensation execution mechanism. Figure 4 This document demonstrates the seven-step process of the delayed obligation compensation mechanism. Triggering Conditions: Data sharing needs trigger the process, and the target dataset contains historical data where obligations were not fully executed. Prerequisite Checks (Three): ① Data still exists in the storage system; ② Field semantic mapping in P1 is accurate (SSC has completed modeling); ③ Data flow genealogy records exist (SOG has recorded the flow path). Prerequisite Satisfaction Judgment: If all three prerequisites are met, the process continues; if any is not met, manual intervention is initiated. Generate Delay Compensation Plan: A compensation execution plan is generated based on the current obligation set in P1, using the same mechanism as the normal pipeline obligation execution plan generation. Execute Delay Obligation Compensation: DSOP performs batch obligation processing on historical data (de-identification, anonymization, DICOM de-identification, etc.). Generate Obligation Compensation Proof: A compensation audit log is generated and appended to the hash chain, generating an SM2-signed obligation compensation proof. Allow Data Sharing Operations: After confirming that the obligation has been fully executed, the data sharing process continues. The bottom label highlights the core design principle: Data security obligations do not necessarily need to be executed immediately upon data generation, but must be completed before the data is shared. Detailed Implementation Example 1: Batch Export Pipeline for Hospital Research Datasets

[0011] Taking the ETL pipeline for regularly synchronizing data from a tertiary hospital's HIS / LIS system to a research database as an example, the specific implementation of the present invention is described.

[0012] Pipeline Scenario: Every day at 2:00 AM, the DataX scheduling task reads the previous day's medical records from the HIS database, cleans the data, and writes it to the research database. Fields involved include patient_id (PersonalData·L3), diagnosis (DiagnosisRecord·L3), lab_results (LabResultValue·L3), genetic_markers (GeneticInformation·L4), and visit_date (BusinessData·L1).

[0013] Step 1—Obligation Set Query: When the task starts, the DataX Obligation Plugin queries the P1 Security Semantic Knowledge Base in batches for the obligation sets of the above fields. The knowledge base returns: patient_id→{MaskingObligation,kAnonymityObligation(k≥5)}; diagnosis→{MaskingObligation,kAnonymityObligation(k≥5)}; genetic_markers→{DenyShareObligation}; visit_date→{AuditLogObligation}.

[0014] Step 2—Execution Plan: The system generates an execution plan: Exclusion field set = {genetic_markers}; Desensitized fields = {patient_id, diagnosis, lab_results}; k-anonymization field = {patient_id, diagnosis} (quasi-identifier combination), k = 5; Dataset watermark = batch identifier B20240315 + task ID.

[0015] Step 3—Duty Execution: The DataX duty plugin executes the following during the Writer phase: the genetic_markers column is removed from the output records; patient_id and diagnosis are masked for anonymization; the ARX library performs k=5 anonymization on the {patient_id, diagnosis} combination; and a batch watermark is embedded in the output CSV file.

[0016] Step 4—Audit Log: After the obligation is completed, an audit log is written within the same transaction of the DataX task. The log records the fields: obligation execution list, k value = 5, watermark identifier, and execution time. The hash_prev field of the log entry points to the SM3 hash of the last access state audit log in P2, forming a continuous hash chain across stages.

[0017] Step 5—Verify Output: After receiving the data, the research database can verify the following through the attached checklist: the genetic_markers field is indeed missing (exclude execution); patient_id has been de-identified (verify de-identification mode); k-anonymization k value ≥ 5 (verify k-value field); watermark identifier corresponds to batch record. Example 2: Real-time CDC Data Stream Processing Pipeline

[0018] Taking the example of data changes in a bank's core system being captured by Debezium CDC, processed by Apache Flink, and written to a data lake, this paper describes the implementation method for a stream processing scenario.

[0019] Pipeline scenario: Debezium captures INSERT / UPDATE events from the bank's transaction table, which flow into the Flink streaming processing job via the Kafka message queue. Flink then writes the processed data into the Hudi table in the data lake.

[0020] Implementation: An obligation execution operator is embedded in the ProcessFunction of the Flink job. Upon startup, the operator performs batch queries of the P1 knowledge base (cached within the same Flink TaskManager to reduce network overhead). When processing each incoming record, the operator performs streaming anonymization on the fields according to the obligation execution plan (without waiting for batch accumulation). For fields requiring k-anonymization, the operator maintains a sliding time window (default 1 hour), performing batch k-anonymization after accumulating sufficient records within the window, satisfying the constraint k≥5. Audit logs are published to the audit log topic as Kafka messages, consumed in real-time by the SOC, and continuously connected to the access-state audit logs via an SM3 hash chain. Example 3: Delay Compensation Execution Scenario

[0021] Taking the example of a hospital needing to share three years of accumulated clinical data with research institutions, this paper describes the implementation method of delayed compensation.

[0022] Background: Historical data was not subject to systematic security obligations when written to the data lake (only basic encrypted storage was performed). Now, it needs to be shared externally, and compliance requirements stipulate that externally shared data must meet the current obligations rules defined in the P1 knowledge base.

[0023] Step 1—Delay Compensation Trigger: After the data sharing application is submitted, the system detects that the target dataset lacks an obligation execution record (there is no corresponding obligation execution entry in the audit log), and automatically triggers the delay compensation execution task.

[0024] Step 2—Compensation Obligation Query: Using the field list of the historical dataset as input, query the obligation rules of the current P1 knowledge base (note: use the currently valid rules, not the historical rules) and generate a compensation obligation execution plan.

[0025] Step 3—Batch Compensation Execution: Perform batch processing on historical data in the data lake. The process is the same as in Example 1, but the audit log is marked with execution_type=COMPENSATORY, recording the triggering reason and time of the compensation execution.

[0026] Step 4—Compensation Completion Confirmation: After the compensation is completed, the system generates a compensation execution report, which includes an SM2 signature, as proof of compliance that the obligations have been fulfilled before external sharing. Innovation points compared to existing technologies

[0027] Pipeline Obligation Manual script configuration carries a high risk of omissions. Unified semantic knowledge base driving automated execution Cross-level audit continuity The audit of the pipeline layer and the service layer is disconnected. Unified format + SM3 hash chain cross-layer continuum unstructured data Lack of systematic processing capabilities DICOM tag removal + document desensitization integration Historical data compliance No compensation enforcement mechanism Compensation for Delayed Obligations: Three Prerequisites for Verification Integration method Existing pipelines need to be modified Non-intrusive integration of framework plugins

Claims

1. A method for automatically embedding and executing security obligations in a data processing pipeline, characterized in that, Includes the following steps: (1) In the obligation parsing stage of the data processing pipeline, an obligation set query is initiated to the security semantic knowledge base for the list of data fields to be processed in the pipeline. The security semantic knowledge base returns the security obligation set and obligation execution parameters corresponding to the semantic class to which each field belongs. (2) Based on the set of obligations obtained from the query, an obligation execution plan is automatically generated. The obligation execution plan includes grouping fields by obligation type, determining the set of rejection fields, and determining the order of obligation execution. (3) During the execution of the data processing pipeline, perform mandatory processing on the data according to the mandatory execution plan, including: excluding rejected fields from the output dataset; performing desensitization processing on fields that need to be desensitized according to the mandatory parameters; performing k-anonymization or differential privacy processing on fields that need to be anonymized; and embedding a digital watermark carrying a batch identifier at the dataset level. (4) After the obligation is completed, the obligation execution audit log is atomically written within the same transaction boundary. The audit log and the data access layer audit log adopt the same format and are connected continuously through a hash chain. (5) Output the processed dataset only after the audit log is successfully written in step (4).

2. The method according to claim 1, characterized in that, In step (1), the query results of the obligation set are cached locally. The cached key value is the semantic class combination identifier of the field. When the semantic class definition or obligation rule in the security semantic knowledge base changes, the relevant cached key value automatically becomes invalid, forcibly triggering a re-query to ensure that the obligation execution parameters are consistent with the knowledge base.

3. The method according to claim 1, characterized in that, In step (2), the order of execution of obligations is determined by the following priority: rejection field exclusion has the highest priority, followed by field-level desensitization, then k-anonymization, and finally dataset-level watermark embedding. This order ensures that high-risk fields have been removed from the data before any processing step, preventing accidental leakage of high-risk data during processing.

4. The method according to claim 1, characterized in that, The obligation processing in step (3) is embedded into the data processing pipeline framework in the form of a plug-in, supporting the following two embedding modes: Mode A: Embed custom operators into streaming and batch processing frameworks such as Apache Flink or Apache Spark. The processing logic is treated as a processing operator in the pipeline, sharing the pipeline execution engine with other data transformation operators, without changing the existing pipeline architecture. Mode B: Integrate data integration tools such as DataX as a Writer plugin, embed the execution logic of the task during the data writing stage, and make it transparent to the data reading and transformation stages.

5. The method according to claim 1, characterized in that, In step (3), the k anonymization process calls the ARX data anonymization library. The lower limit of the k value is defined by the obligation parameter of the corresponding semantic class in the security semantic knowledge base. The data processing pipeline automatically reads the k value parameter from the obligation set before execution, without the need for manual configuration.

6. The method according to claim 1, characterized in that, In step (4), the hash chain of the audit log achieves continuity in the following way: each audit log entry contains the SM3 hash digest of the preceding entry as a link field, so that any tampering with any historical log entry will cause the hash verification of all subsequent entries to fail, thereby realizing a unified and tamper-proof audit chain for pipeline-state audit logs and access-state audit logs.

7. The method according to claim 1, characterized in that, The dataset output in step (5) includes an additional obligation execution list. The obligation execution list contains the obligation execution type, execution parameters, execution result status of each field, as well as the k-anonymization value, watermark identifier, and SM2 digital signature of the entire dataset. The recipient can independently verify the secure processing status of the dataset through the obligation execution list.

8. The method according to claim 1, characterized in that, It also includes a delayed compensation execution mode: when data has been stored in the data lake warehouse but has not been fully processed in the past, before the data is shared externally, batch static obligation compensation execution is triggered for the existing data. The compensation execution is based on the currently valid obligation rules in the security semantic knowledge base, and steps (1) to (5) are re-executed on the existing data, and marked as compensation execution type in the audit log.

9. The method according to claim 8, characterized in that, The triggering conditions for the delayed compensation execution include: there are unexecuted obligation records in the existing data; the existing data is requested to be shared externally; the obligation rules of the relevant fields in the security semantic knowledge base have changed, causing the obligation execution status of the existing data to no longer meet the current rule requirements. The compensation execution process can be started when any of the three triggering conditions are met.

10. The method according to claim 1, characterized in that, The obligation processing for unstructured data in step (3) includes: for DICOM format medical image data, performing tag de-identification processing according to the DICOM PS 3.15 standard, with the de-identification level specified by the DicomDeidentObligation obligation parameter corresponding to the MedicalImageData semantic class in the security semantic knowledge base; for PDF format document data, performing document content de-identification processing through text extraction and document reconstruction, with the de-identified field type defined by the obligation parameter.

11. A data processing pipeline security obligation enforcement system, characterized in that, include: The obligation parser is configured to receive a list of pipeline fields, query the security semantic knowledge base to obtain the obligation set, generate an obligation execution plan, and maintain a local cache of the obligation set. The execution scheduler is configured to schedule each executor according to the task execution plan, manage both batch and streaming execution modes, and handle retries and circuit breakers for execution failures. The obligation executor pool includes field desensitization executors, k-anonymization executors, digital watermarking executors, and unstructured data processing executors. Each executor reads the corresponding obligation parameters from the obligation execution plan and executes independently.

Images