Intelligent data development treatment system and method based on large model

Abstract

The invention relates to the technical field of artificial intelligence and data engineering, in particular to an intelligent data development treatment system and method based on a large model, the system is composed of a data access layer, a large model treatment engine, an intelligent feature factory, a compliance control layer and an application interface layer, and all the modules cooperate to achieve intelligent transformation of the whole data development process; the method has the beneficial effects that the potential mode of the multi-source heterogeneous data can be automatically identified, for example, transaction records, user behavior data and system monitoring information are distinguished from mixed log files, and manual rule pre-definition is not needed. And dynamically adjusting the protection intensity according to the data use scene.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and data engineering technology, specifically to an intelligent data development and governance system and method based on a large model. Background Technology

[0002] Traditional data development techniques (cited patent CN109902123A, "Data Cleaning System and Method Based on Rule Engine") employ predefined rules for data quality checks and standardization. They rely on human experience to construct feature engineering processes. Their ability to process unstructured data (such as text and images) is limited.

[0003] It demonstrates strong capabilities in natural language understanding and pattern recognition. It has been tested in data annotation (e.g., document CN113435572A, "A Data Annotation System Based on Deep Learning"). However, a technical framework deeply integrated with the entire data development process has not yet been established.

[0004] Existing technological shortcomings:

[0005] (1) Low level of intelligence in data processing: Rule engines struggle to handle complex data patterns (such as mixed medical image and text data). Feature engineering relies on expert experience and has a long iteration cycle (averaging 5-7 days per project).

[0006] (2) Lagging compliance risk control: Existing anonymization techniques (such as the K-anonymity algorithm) cannot dynamically adapt to the data context (cited case: a bank's customer information leak in 2022). It is difficult to balance privacy protection and data utility (see the paper "Privacy-Utility Tradeoff in Deep Learning").

[0007] (3) Fragmented technology stack: Large model capabilities are not deeply embedded in the ETL (Extract-Transform-Load) process. Data governance tools and AI development platforms lack organic synergy (such as integration barriers between Informatica and TensorFlow). Summary of the Invention

[0008] The purpose of this invention is to provide an intelligent data development and governance system and method based on a large model, so as to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution: an intelligent data development and governance system based on a large model, comprising:

[0010] The data access layer is used to achieve unified access and intelligent routing of multi-source heterogeneous data, supports structured, semi-structured and unstructured data types, and integrates real-time stream processing mechanism and distributed caching technology.

[0011] The large model governance engine includes a metadata intelligent generation system and a data quality enhancement module. It achieves field semantic parsing, dynamic data lineage tracking, and context-aware anomaly detection and missing value filling through a multimodal large model.

[0012] The intelligent feature factory provides a fully automated feature engineering pipeline and interactive feature optimization functions, supporting feature generation based on Prompt templates, SHAP value analysis and filtering, and feature iteration driven by natural language feedback.

[0013] The compliance control layer integrates a dynamic hierarchical de-identification system and a real-time compliance audit engine, which adaptively adjusts the de-identification strategy according to the data usage scenario, and has a built-in machine-readable rule base for regulatory clauses and natural language rule conversion capabilities.

[0014] The application interface layer provides a unified service gateway and a visual interactive interface, supporting multi-protocol service calls and natural language-driven data lineage penetration query and feature analysis task execution.

[0015] Preferably, the intelligent data identification and routing function of the data access layer is implemented through the following steps: receiving raw data samples and automatically identifying data structure types, including tables, text logs, or medical images, using a multimodal large model; generating data pattern labels and routing structured data to the ETL pipeline and semi-structured / unstructured data to the NLP processing module or image analysis module according to preset rules; integrating the Kafka real-time stream processing framework and Redis distributed caching technology to achieve millisecond-level latency IoT sensor data access and hot data caching optimization for high-frequency access scenarios.

[0016] Preferably, the metadata intelligent generation system of the large model governance engine achieves field semantic parsing through the following technologies: receiving field names and example values ​​as input; building a Prompt template, requiring the generation of metadata descriptions containing Chinese names, data types, business rules, and sensitivity levels; calling the large model to generate structured output; and using the Neo4j graph database to store data lineage, supporting real-time traceability and querying of the entire processing process from raw input to final features.

[0017] Preferably, the dynamic hierarchical data masking system of the compliance control layer performs data masking according to the following adaptive strategies: in the development environment, GAN is used to generate synthetic data to achieve strong masking; in the testing environment, masking technology that retains distribution characteristics is used to achieve partial masking; in the production environment, conditional masking is achieved based on the RBAC permission model, and sensitive fields are dynamically decrypted only for authorized users; and a machine-readable rule base integrating GDPR / HIPAA regulatory clauses is integrated to automatically convert text regulations into SQL query conditions through a large model.

[0018] Preferably, the interactive feature optimization function of the intelligent feature factory is implemented through the following natural language feedback mechanism: receiving business requirement text input by the user; parsing the semantics and matching it with a preset feature template library to generate a candidate feature list; combining business rules and feature importance assessment results, automatically performing data extraction, difference calculation and feature encoding, and finally generating interpretable feature indicators; providing a three-dimensional data lineage map visualization interface, supporting penetrating queries of the processing history, related indicators and compliance audit records of any field.

[0019] A method for developing and governing an intelligent data system based on a large model includes the following steps:

[0020] The data access layer enables unified access to heterogeneous data from multiple sources, supporting real-time stream processing and distributed caching of structured, semi-structured, and unstructured data.

[0021] The large model governance engine is used to intelligently generate metadata and enhance data quality of the accessed data, including field semantic parsing, dynamic data lineage tracking, context-aware anomaly detection, and missing value imputation based on data distribution.

[0022] A fully automated feature engineering pipeline is built through an intelligent feature factory, which combines interactive natural language feedback mechanisms to generate and optimize features.

[0023] Based on the compliance control layer, dynamic hierarchical desensitization and real-time compliance auditing are achieved. The desensitization strategy is adaptively adjusted according to the data usage scenario and the regulatory clauses are automatically converted into enforceable rules.

[0024] The application interface layer provides a unified service gateway and a visual interactive interface, supporting multi-protocol service calls and natural language-driven data lineage penetration query and feature analysis task execution.

[0025] Preferably, the intelligent data identification and routing steps of the data access layer include: receiving raw data samples, automatically identifying data structure types using a multimodal large model, and generating data pattern labels; routing structured data to the ETL pipeline, semi-structured data to the log parsing module, and unstructured data to the NLP processing module or image analysis module based on the labels; achieving millisecond-level latency access to real-time data streams through Kafka, and optimizing the efficiency of hot data storage in high-frequency access scenarios using Redis distributed caching technology.

[0026] Preferably, the metadata intelligent generation steps of the large model governance engine include: receiving field names and example values ​​as input, constructing a Prompt template to generate metadata descriptions containing Chinese names, data types, business rules, and sensitivity levels; calling the large model to parse field semantics; and using the Neo4j graph database to store data lineage, recording the entire data processing process from raw input to final features, supporting real-time traceability queries and correlation indicator analysis.

[0027] Preferably, the interactive feature optimization steps of the intelligent feature factory include: receiving natural language requirements input by the user; parsing the semantics and matching them with a preset feature template library to generate a candidate feature list; filtering candidate features by combining SHAP value analysis and business rules, and storing and versioning the filtered features using an adaptive encoding strategy; and displaying the feature generation process through a visual interactive interface, supporting users to dynamically adjust and iteratively optimize the feature library.

[0028] Preferably, the dynamic hierarchical data masking and real-time compliance auditing steps of the compliance control layer include: adaptively selecting masking strategies based on data usage scenarios: using GAN-generated synthetic data for strong masking in the development environment, using masking to retain distribution characteristics for partial masking in the testing environment, and implementing conditional masking based on the RBAC permission model in the production environment; building a machine-readable rule base for GDPR / HIPAA regulations, automatically converting textual regulations into SQL query conditions through a large model; triggering compliance audits in real time during data access, combining dynamic masking strategies and rule engines to verify the legality of data use, generating compliance audit logs and supporting penetrating queries.

[0029] Compared with the prior art, the beneficial effects of the present invention are:

[0030] This invention proposes an intelligent data development and governance system and method based on a large-scale model. This system can automatically identify potential patterns in multi-source heterogeneous data, such as distinguishing transaction records, user behavior data, and system monitoring information from mixed log files, without requiring manually predefined rules. The protection strength is dynamically adjusted according to the data usage scenario. For example, statistically accurate synthetic data is generated during the development and testing phase, while conditional decryption services are provided to authorized users in the production environment, satisfying compliance requirements while ensuring data utility. Through continuous analysis of data operation logs using the large-scale model, abnormal behavior patterns can be identified in real time (such as batch exporting sensitive fields outside of working hours), significantly reducing the risk of data leakage compared to traditional post-audit methods.

[0031] The system automatically transforms best practices in data processing (such as industry-specific feature combination strategies and compliance constraints) into reusable knowledge templates, forming a continuously evolving intelligent asset library. A 3D lineage graph displays the data processing chain, helping users intuitively understand the feature derivation process; attention mechanism visualization technology reveals the key basis of large models in data governance decisions. It combines the ability to generate large models with the stability of traditional data engineering. The system supports dynamic scheduling of heterogeneous computing resources, automatically using GPUs to accelerate model inference when processing unstructured data, and switching to CPU-optimized clusters when performing batch processing of structured data, maximizing resource utilization. Attached Figure Description

[0032] Figure 1 This is a system block diagram of the present invention;

[0033] Figure 2 This is a flowchart of the system interaction of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example 1, please refer to Figures 1 to 2 This invention provides a technical solution: an intelligent data development and governance system based on a large model, comprising five main parts: a data access layer, a large model governance engine, an intelligent feature factory, a compliance control layer, and an application interface layer. These modules work together to achieve intelligent transformation of the entire data development process. (See architecture diagram below.) Figure 1 System architecture and functional module diagram. The core functions of each module are explained in detail below:

[0036] (1) Data Access Layer

[0037] Unified access to multi-source heterogeneous data:

[0038] Supported data types: structured data (such as SQL databases), semi-structured data (JSON / XML logs), and unstructured data (text, images, audio).

[0039] Real-time stream processing: Access real-time data streams (such as IoT sensor data) via Kafka, supporting millisecond-level latency processing.

[0040] Data caching mechanism: Use a distributed in-memory database (such as Redis) to cache hot data and improve efficiency in high-frequency access scenarios.

[0041] Intelligent data identification and routing:

[0042] Data pattern self-identification technology based on large models:

[0043] Input: Raw data sample (e.g., the first 100 rows of a CSV file).

[0044] Processing: Utilize multimodal large models to automatically determine the data structure type (such as tables, text logs, medical images).

[0045] Output: Generate data pattern labels and route them to the corresponding processing pipelines (e.g., structured data enters the ETL pipeline, and unstructured data enters the NLP processing module).

[0046] (2) Large Model Governance Engine

[0047] 1) Metadata Intelligent Generation System

[0048] Field semantic parsing:

[0049] #Example: Metadata generation process for database column name "txn_amt"

[0050] input_text = "Field name: txn_amt, Example value: 1500.00"

[0051] prompt="Generate metadata description for this field, including its Chinese name, value range, and business meaning."

[0052] metadata=llm.generate(prompt,input_text)

[0053] #Output:

[0054] #Name: Transaction Amount

[0055] #Type: Numeric

[0056] #Business Rules: Positive floating-point number, unit: RMB yuan

[0057] #Sensitivity Level: PII-2 (Requires desensitization)

[0058] Data lineage dynamic tracking:

[0059] Record the entire data processing process from raw input to final features.

[0060] A graph database (Neo4j) is used to store bloodline relationships, supporting real-time tracing and querying.

[0061] 2) Data Quality Enhancement Module

[0062] Outlier detection:

[0063] Context-aware detection based on large models, combined with business scenario judgment (e.g., "blood pressure value 300 / 180" in medical data may be effective in ICU scenarios).

[0064] Missing value imputation:

[0065] Large models are used to generate filler values ​​that conform to the data distribution.

[0066] (3) Intelligent Feature Factory

[0067] 1) Fully automated feature engineering production line

[0068] stage Technical Implementation Feature generation Candidate features are generated based on the Prompt template. Feature Filtering Combining SHAP value analysis with business rule filtering Feature encoding Adaptive coding strategy Feature storage Feature library version management

[0069] 2) Interactive Feature Optimization

[0070] Natural language feedback mechanism:

[0071] User input: "I need more characteristics that reflect users' short-term behavior."

[0072] System response: Generate new features such as "number of logins in the last 7 days" and "variance of the interval between the last 3 transactions".

[0073] (4) Compliance control layer

[0074] 1) Dynamic hierarchical desensitization system

[0075] Scenario-adaptive desensitization strategy:

[0076] Data use cases Desensitization level Technical Implementation Development Environment Strong desensitization GAN generates synthetic data Test environment Partial desensitization Mask that preserves distribution characteristics Production Environment Conditional desensitization Dynamic decryption based on RBAC

[0077] 2) Real-time compliance audit

[0078] Audit rules engine:

[0079] It includes a machine-readable rule base containing regulations such as GDPR / HIPAA (e.g., "Personal data storage shall not exceed 6 months").

[0080] Large-scale model-driven natural language rule transformation: automatically converting textual regulations into SQL query conditions.

[0081] (5) Application Interface Layer

[0082] 1) Unified Service Gateway

[0083] Multi-protocol support: REST API, gRPC, GraphQL

[0084] Example of a functional endpoint:

[0085] #Feature Generation Service

[0086] POST / v1 / feature / generate

[0087] Body:{"data_sample":"...","task_type":"risk control model"}

[0088] #Compliance Inspection Service

[0089] POST / v1 / compliance / check

[0090] Body:{"data_schema":"...","regulation":"GDPR Article 17"}

[0091] 2) Visual interactive interface

[0092] Natural Language Console:

[0093] User input: "Compare sales data from last month and generate regional difference analysis features".

[0094] The system automatically performs the following steps: data extraction → difference calculation → generation of features such as the "regional sales fluctuation index".

[0095] 3D data lineage map: Supports penetrating queries of processing history and related indicators for any field.

[0096] 2. Implementation of key technologies

[0097] (1) Data governance in a federated learning environment

[0098] Secure data collaboration process:

[0099] The local data of the participants is anonymized and then encrypted. The central coordinating node aggregates the encrypted features of all parties and embeds them. After the global model is updated, the data is distributed to all participants.

[0100] (2) Generative Data Desensitization Algorithm

[0101] An improved architecture based on Wasserstein GAN:

[0102] 1) Generator network design:

[0103] class Generator(nn.Module):

[0104] def__init__(self):

[0105] super().__init__()

[0106] self.mlp = nn.Sequential(

[0107] nn.Linear(128,256),

[0108] nn.LeakyReLU(0.2),

[0109] nn.LayerNorm(256),

[0110] nn.Linear(256,512),

[0111] nn.Tanh() # Constrains the output range )

[0113] def forward(self, z):

[0114] return self.mlp(z)

[0115] 2) Training objective function:

[0116] \min_G\max_D\mathbb{E}[D(x)]-\mathbb{E}[D(G(z))]+\lambda\cdot

[0117] \text{Wasserstein_loss}

[0118] 3. System Interaction Flow Example

[0119] Taking a financial anti-fraud scenario as an example, the end-to-end processing flow is shown in the flowchart. Figure 2 System interaction flowchart:

[0120] Step 1: Data Access and Recognition

[0121] Input data includes transaction records (structured) and customer service call recordings (unstructured).

[0122] The system automatically identifies:

[0123] Transaction table → routed to structured processing pipeline.

[0124] Audio files are converted to text using the ASR module and then processed in the NLP analysis workflow.

[0125] Step 2: Automated Data Governance

[0126] The "user_id" field in the transaction table was marked as PII-1 level sensitive data, triggering real-time data masking.

[0127] The bank card number in the call text is identified and masked (e.g., "last four digits **1234").

[0128] Step 3: Intelligent Feature Engineering

[0129] Example of generated features:

[0130] "Number of accounts associated with the same IP address" (calculated based on a graph).

[0131] "Urgency rating in call text" (based on sentiment analysis model).

[0132] Step 4: Compliance Verification and Deployment

[0133] Automatically check whether the features comply with the "Technical Specifications for the Protection of Personal Financial Information".

[0134] The validated feature package is published to the model training platform.

[0135] Example 2, based on Example 1, proposes a method for an intelligent data development and governance system based on a large model as described in claim 5, comprising the following steps:

[0136] The data access layer enables unified access to multi-source heterogeneous data, supporting real-time stream processing and distributed caching of structured, semi-structured, and unstructured data. The intelligent data identification and routing steps of the data access layer include: receiving raw data samples, automatically identifying data structure types using a multimodal large model, and generating data pattern labels; routing structured data to the ETL pipeline, semi-structured data to the log parsing module, and unstructured data to the NLP processing module or image analysis module based on the labels; achieving millisecond-level latency access to real-time data streams through Kafka, and optimizing hot data storage efficiency in high-frequency access scenarios using Redis distributed caching technology.

[0137] The large model governance engine is used to intelligently generate metadata and enhance data quality of the incoming data, including field semantic parsing, dynamic data lineage tracking, context-aware anomaly detection, and missing value imputation based on data distribution. The intelligent metadata generation steps of the large model governance engine include: receiving field names and sample values ​​as input, constructing a Prompt template to generate metadata descriptions containing Chinese names, data types, business rules, and sensitivity levels; calling the large model to parse field semantics; and using the Neo4j graph database to store data lineage relationships, recording the entire processing process of data from raw input to final features, supporting real-time traceability queries and correlation indicator analysis.

[0138] A fully automated feature engineering pipeline is built through an intelligent feature factory, which generates and optimizes features using an interactive natural language feedback mechanism. The interactive feature optimization steps of the intelligent feature factory include: receiving natural language requirements input by the user; parsing the semantics and matching them with a preset feature template library to generate a candidate feature list; filtering candidate features by combining SHAP value analysis and business rules, and storing and versioning the selected features using an adaptive encoding strategy; and displaying the feature generation process through a visual interactive interface, supporting users to dynamically adjust and iteratively optimize the feature library.

[0139] The compliance control layer enables dynamic hierarchical data masking and real-time compliance auditing. It adaptively adjusts masking strategies based on data usage scenarios and automatically converts regulatory clauses into executable rules. The dynamic hierarchical data masking and real-time compliance auditing steps of the compliance control layer include: adaptively selecting masking strategies based on data usage scenarios: strong masking using GAN-generated synthetic data in the development environment; partial masking using masks that retain distribution characteristics in the testing environment; and conditional masking based on the RBAC permission model in the production environment. It incorporates a machine-readable rule base for regulations such as GDPR / HIPAA, automatically converting textual regulations into SQL query conditions through a large model. Real-time compliance auditing is triggered during data access, combining dynamic masking strategies and a rule engine to verify the legality of data use, generating compliance audit logs, and supporting penetrating queries.

[0140] The application interface layer provides a unified service gateway and a visual interactive interface, supporting multi-protocol service calls and natural language-driven data lineage penetration query and feature analysis task execution.

[0141] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An intelligent data development and governance system based on a large model, characterized in that: include: The data access layer is used to achieve unified access and intelligent routing of multi-source heterogeneous data, supports structured, semi-structured and unstructured data types, and integrates real-time stream processing mechanism and distributed caching technology. The large model governance engine includes a metadata intelligent generation system and a data quality enhancement module. It achieves field semantic parsing, dynamic data lineage tracking, and context-aware anomaly detection and missing value filling through a multimodal large model. The intelligent feature factory provides a fully automated feature engineering pipeline and interactive feature optimization functions, supporting feature generation based on Prompt templates, SHAP value analysis and filtering, and feature iteration driven by natural language feedback. The compliance control layer integrates a dynamic hierarchical de-identification system and a real-time compliance audit engine, which adaptively adjusts the de-identification strategy according to the data usage scenario, and has a built-in machine-readable rule base for regulatory clauses and natural language rule conversion capabilities. The application interface layer provides a unified service gateway and a visual interactive interface, supporting multi-protocol service calls and natural language-driven data lineage penetration query and feature analysis task execution.

2. The intelligent data development and governance system based on a large model according to claim 1, characterized in that: The intelligent data identification and routing function of the data access layer is implemented through the following steps: receiving raw data samples and automatically identifying data structure types, including tables, text logs, or medical images, using a multimodal large model; generating data pattern labels and routing structured data to the ETL pipeline and semi-structured / unstructured data to the NLP processing module or image analysis module according to preset rules; integrating the Kafka real-time stream processing framework and Redis distributed caching technology to achieve millisecond-level latency IoT sensor data access and hot data caching optimization for high-frequency access scenarios.

3. The intelligent data development and governance system based on a large model according to claim 2, characterized in that: The metadata intelligent generation system of the large model governance engine achieves field semantic parsing through the following technologies: receiving field names and example values ​​as input; building a Prompt template, requiring the generation of metadata descriptions containing Chinese names, data types, business rules, and sensitivity levels; calling the large model to generate structured output; and using the Neo4j graph database to store data lineage, supporting real-time traceability and querying of the entire processing process from raw input to final features.

4. The intelligent data development and governance system based on a large model according to claim 3, characterized in that: The compliance control layer's dynamic hierarchical data masking system performs data masking based on adaptive strategies for the following scenarios: the development environment uses GAN to generate synthetic data to achieve strong data masking; The test environment uses a masking technique that preserves distribution characteristics to achieve partial desensitization; the production environment uses the RBAC permission model to achieve conditional desensitization, dynamically decrypting sensitive fields only for authorized users; and it integrates a machine-readable rule base of GDPR / HIPAA regulations, automatically converting text regulations into SQL query conditions through a large model.

5. The intelligent data development and governance system based on a large model according to claim 4, characterized in that: The interactive feature optimization function of the intelligent feature factory is implemented through the following natural language feedback mechanism: receiving business requirement text input by the user; parsing the semantics and matching it with a preset feature template library to generate a candidate feature list; By combining business rules and feature importance assessment results, the system automatically performs data extraction, difference calculation, and feature coding to ultimately generate interpretable feature indicators. It also provides a 3D data lineage map visualization interface, supporting drill-down queries of the processing history, related indicators, and compliance audit records of any field.

6. A method for an intelligent data development and governance system based on a large model according to claim 5, characterized in that: Includes the following steps: The data access layer enables unified access to heterogeneous data from multiple sources, supporting real-time stream processing and distributed caching of structured, semi-structured, and unstructured data. The large model governance engine is used to intelligently generate metadata and enhance data quality of the accessed data, including field semantic parsing, dynamic data lineage tracking, context-aware anomaly detection, and missing value imputation based on data distribution. A fully automated feature engineering pipeline is built through an intelligent feature factory, which combines interactive natural language feedback mechanisms to generate and optimize features. Based on the compliance control layer, dynamic hierarchical desensitization and real-time compliance auditing are achieved. The desensitization strategy is adaptively adjusted according to the data usage scenario and the regulatory clauses are automatically converted into enforceable rules. The application interface layer provides a unified service gateway and a visual interactive interface, supporting multi-protocol service calls and natural language-driven data lineage penetration query and feature analysis task execution.

7. A method according to claim 6, characterized in that: The intelligent data identification and routing steps in the data access layer include: receiving raw data samples, automatically identifying data structure types using a multimodal large model, and generating data pattern labels; routing structured data to the ETL pipeline, semi-structured data to the log parsing module, and unstructured data to the NLP processing module or image analysis module based on the labels; achieving millisecond-level latency access to real-time data streams through Kafka, and optimizing the efficiency of hot data storage in high-frequency access scenarios using Redis distributed caching technology.

8. A method according to claim 7, characterized in that: The metadata intelligent generation steps of the large model governance engine include: receiving field names and example values ​​as input, building a Prompt template to generate metadata descriptions containing Chinese names, data types, business rules, and sensitivity levels; calling the large model to parse field semantics; and using the Neo4j graph database to store data lineage, recording the entire data processing process from raw input to final features, supporting real-time traceability queries and correlation indicator analysis.

9. A method according to claim 8, characterized in that: The interactive feature optimization steps of the intelligent feature factory include: receiving natural language requirements input by the user; parsing the semantics and matching them with a preset feature template library to generate a candidate feature list; filtering candidate features by combining SHAP value analysis and business rules, and storing and versioning the filtered features using an adaptive encoding strategy; and displaying the feature generation process through a visual interactive interface, supporting users to dynamically adjust and iteratively optimize the feature library.

10. A method according to claim 9, characterized in that: The dynamic hierarchical data masking and real-time compliance auditing steps of the compliance control layer include: adaptively selecting masking strategies based on data usage scenarios: strong masking using GAN-generated synthetic data in the development environment, partial masking using masks that retain distribution characteristics in the testing environment, and conditional masking based on the RBAC permission model in the production environment; a built-in machine-readable rule base of GDPR / HIPAA regulations, which automatically converts text regulations into SQL query conditions through a large model; real-time compliance auditing is triggered when data is accessed, combining dynamic masking strategies and rule engines to verify the legality of data use, generating compliance audit logs and supporting penetrating queries.

Images