Method for intelligently forming data warehouse by using AI
By introducing AI agents and entropy-priority loading mechanisms into the data warehouse ETL process, the problem of low processing efficiency in traditional data warehouses is solved, achieving efficient and accurate data processing and security compliance, and supporting real-time analysis of large-scale data.
Patent Information
- Application Number
- CN202511663487.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-13
- Publication Date
- 2026-02-03
AI Technical Summary
Traditional data warehouse technology relies on manual rules for data cleaning, which cannot adapt to high-frequency data updates. This results in end-to-end processing time becoming a key bottleneck for business agility, and also poses risks of data silos and loss of access control.
AI agents are used to extract, transform, and load data during the ETL process. By combining a multimodal information filtering agent cluster, a knowledge graph-driven data governance network, and an entropy-first intelligent loading mechanism, efficient and accurate data processing is achieved.
It improves data accuracy by 40%, invalid record identification rate by ≥95%, reduces processing time by 50%, reduces manual intervention costs by 70%, supports petabyte-level data response in seconds, and meets data security compliance requirements.
Smart Images
Figure CN121456064A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and more specifically to a method for forming a data warehouse using AI intelligence. Background Technology
[0002] As the cornerstone of enterprise decision support systems (DSS), data warehouses are subject-oriented, integrated, and non-volatile collections of data specifically designed to serve the strategic analysis and business insight needs of middle and senior managers. Their core value lies in transforming business information scattered across multiple heterogeneous data sources such as ERP, CRM, and log systems into quantifiable and traceable decision-making data through structured integration.
[0003] Current data warehouse technology has formed a traditional architecture centered on ETL (Extract-Transform-Load), coupled with star / snowflake schema data models and OLAP engines to achieve multidimensional analysis. However, with the acceleration of enterprise digital transformation, the scale of data is growing exponentially. For example, internet companies process petabytes of user behavior data daily. Traditional technology systems rely on manual rules for data cleaning, resulting in more than 30% of invalid records going unidentified, such as crawler traffic in user click logs on e-commerce platforms. Cross-departmental data sharing requires cumbersome permission application processes, leading to data silos and the risk of uncontrolled access. Summary of the Invention
[0004] The present invention aims to provide a method for forming a data warehouse using AI intelligence, in order to solve the problem that traditional technologies rely on manual intervention and cannot adapt to high-frequency data updates, and the time-consuming end-to-end processing has become a key bottleneck for business agility.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for forming a data warehouse using AI intelligence, comprising: In the data extraction step, an AI agent is added to filter the information extracted from the data. The data transformation step involves adding a transformation AI agent to achieve data transformation. The data loading step involves writing the processed data into the target system. The output step of the analysis table involves adding an AI agent to label and filter the generated data warehouse to ensure data accuracy.
[0006] Preferably, as an improvement, the data extraction stage deploys a multimodal information filtering intelligent agent cluster for efficient access and filtering of cross-type data sources. The multimodal information filtering intelligent agent cluster includes an NLP agent, an OCR agent, and a data source Holon coordination system, and adopts a triple filtering mechanism of semantic understanding, entropy verification, and pattern matching. The NLP agent performs entity recognition on unstructured text through semantic analysis, and combines regular expression matching agents to perform pattern filtering on structured data; the OCR agent performs optical character recognition on image data, converting tabular data into analyzable text, and accurately locates table boundaries and field areas through layout analysis technology; the Holon data source coordination system achieves unified access to heterogeneous data sources and dynamic adaptation of filtering rules through the collaboration of structured data agents, semi-structured data agents, and unstructured data agents.
[0007] Preferably, as an improvement, the triple filtering mechanism includes: Field-level information entropy calculation: The original data fields are discretized, and the information entropy value of each field is calculated using the Shannon information entropy formula; NLP entity recognition uses a BERT pre-trained model to identify entities in contract text, extract key entities, and simultaneously calculate entity-level information entropy to verify semantic consistency. Regular expression pattern matching uses a predefined regular expression library to filter patterns in structured data and dynamically adjusts the matching threshold based on information entropy values. A dynamic weight adjustment mechanism is established by creating an information entropy-filtering rule mapping table and dynamically adjusting the weights of filtering rules based on the real-time calculated information entropy value.
[0008] Preferably, as an improvement, a data governance intelligent agent network is constructed during the data transformation stage to complete the entire chain of data cleaning, standardization, and correlation fusion. The data cleaning agent uses a hybrid approach of rule engine and machine learning. It fills missing values with similar case interpolation algorithm, detects and corrects outliers using information entropy algorithm, and performs intelligent deduplication of duplicate records. The standardized Agent is based on knowledge graph-driven field mapping to achieve currency unit conversion, date format unification, and coding standard alignment; The association fusion agent uses association rule mining technology to discover implicit relationships between fields across tables, and combines knowledge graphs to achieve multi-source data fusion, forming a composite data view that includes business logic relationships.
[0009] Preferably, as an improvement, a dual-engine approach combining information entropy outlier detection and knowledge graph is integrated into the data cleaning agent; Information entropy distribution calculation involves binning numerical fields and calculating the information entropy value of each bin; outliers are identified through the information entropy distribution map. Knowledge graph traceability and verification: Combine knowledge graphs to build a transaction link model and perform multi-dimensional verification of marked outliers; A dual-engine collaborative decision-making system is established, creating a joint decision-making model of information entropy and knowledge graph. An outlier handling strategy is determined through a weighted voting mechanism. When the information entropy detection and knowledge graph verification results are consistent, data correction is automatically executed. When results conflict, a manual review process is triggered to ensure the interpretability and accuracy of the decision.
[0010] Preferably, as an improvement, the data loading stage achieves automated writing and real-time verification through an intelligent loading mechanism. The loading process is monitored throughout by Holon, a data quality monitoring platform. Real-time stream processing technology is used to write the transformed structured data into the target data warehouse, while triggering the data verification agent to perform integrity verification.
[0011] Preferably, as an improvement, Holon data quality sets an information entropy priority loading strategy, which prioritizes loading low-entropy data segments based on field-level information entropy evaluation results; the implementation method of the entropy priority loading strategy includes: Field-level information entropy evaluation calculates the information entropy of the transformed structured data fields, uses a sliding window mechanism to calculate local information entropy, identifies low-entropy data segments, and marks low-entropy values as priority loading objects; The incremental loading mechanism, combined with change data capture technology and timestamp mechanism, enables accurate capture of data changes in the source system. By monitoring information entropy changes, when a sudden change in the information entropy value of a key field is detected, the incremental loading process is triggered to ensure that the data warehouse reflects business dynamics in real time. Real-time verification and feedback: During the loading process, the data verification agent is triggered to perform integrity verification. Abnormal data is automatically triggered to trigger the repair process. A dual-buffer mechanism is used to achieve parallel processing of loading and verification, ensuring a balance between loading efficiency and data quality.
[0012] Preferably, as an improvement, a data credibility assurance system is built in the analysis table output stage to achieve intelligent collaboration between annotation filtering and visualization output; The data quality inspection agent scans and analyzes tables based on predefined quality rules, generates a quality assessment report, and automatically triggers the remediation process. The annotation and filtering agent uses NLP and machine learning technologies to automatically annotate text data and combines it with a business rule engine to complete conditional filtering. The visualization agent presents the filtered data through dynamic charts, allowing users to drill down into the data through interactive operations.
[0013] Preferably, as an improvement, an ant colony clustering-information entropy collaborative analysis module is introduced on the basis of the annotation and filtering system. This module guides the clustering behavior of the ant colony algorithm through information entropy, optimizing the output quality of the analysis table. The implementation method of the information entropy collaborative analysis module includes: Pheromones are initialized and entropy values are mapped. Ant colony algorithm parameters are initialized, and the mapping relationship between the pheromone matrix and the information entropy value is established. The uncertainty of samples is quantified by the information entropy value. Samples with high entropy values are assigned higher pheromone deposition weights to guide ants to explore high uncertainty areas first. In the clustering process optimization, ants select clustering paths based on pheromone concentration and information entropy value in each iteration. The clustering quality is evaluated using the information entropy minimization objective function, and the clustering strategy is dynamically adjusted through pheromone update rules to achieve entropy-driven clustering optimization. During the clustering process, high-entropy samples will deposit more pheromones, guiding ants to explore the region first; low-entropy samples will reduce pheromone concentration through a evaporation mechanism to avoid over-focusing. Visualized path tracing: When clustering results are output, a dynamic heatmap is used to display the path of information entropy change. Users can interactively trace the clustering trajectory of high-entropy samples. The system automatically marks core product data segments and displays their clustering paths, supporting full-link insights from data to decision-making.
[0014] The beneficial effects of this solution are: by using a multimodal intelligent agent cluster and a dynamic optimization mechanism for information entropy, a closed-loop intelligent governance system is built across the entire ETL chain.
[0015] A multimodal triple-filtering engine is deployed in the data extraction phase, integrating NLP agents, OCR agents, and the Holon coordination system to achieve efficient access and filtering across different data sources. NLP Agent: Employs BERT pre-trained models to perform entity recognition on unstructured data such as contract texts and logs, and combines regular expressions to match structured data to achieve semantic-level filtering; OCR Intelligent Agent: Accurately locates table boundaries through layout analysis, converts image data into analyzable text, and solves the pain point of high misrecognition rate in traditional OCR; Holon Coordination System: Through three-body collaboration (structured / semi-structured / unstructured agents) to dynamically adapt filtering rules, and combined with information entropy to calculate field-level uncertainty, it achieves unified governance of heterogeneous data sources.
[0016] A knowledge graph-driven intelligent governance network is used in the data transformation phase to construct a full-link intelligent agent network for data cleaning, standardization, and correlation fusion. Data Cleaning Agent: Integrates information entropy outlier detection and knowledge graph dual engines, fills missing values through similar case interpolation, identifies outliers using information entropy binning, and verifies transaction chain logic using knowledge graph; Standardized Agent: Based on knowledge graph-driven field mapping, it enables standardized operations such as currency unit conversion and date format unification; Association Fusion Agent: Utilizes association rule mining technology to discover implicit cross-table relationships, and combines it with knowledge graphs to construct composite data views, solving the traditional data silo problem.
[0017] The data loading phase employs an entropy-prioritized intelligent loading mechanism. Holon monitors the loading process for data quality, uses real-time stream processing technology to write data to the target repository, and simultaneously triggers a verification agent to perform integrity checks. Entropy-priority loading: Low-entropy fields are loaded first, and the incremental loading mechanism is combined to reflect business dynamics in real time. Double-buffered verification: Loading and verification are processed in parallel to ensure a balance between efficiency and quality.
[0018] To ensure credibility and facilitate collaborative analysis during the output stage of the analysis table, an intelligent collaborative system for data quality inspection, annotation filtering, and visualization output is constructed. Ant colony clustering-information entropy collaboration module: By mapping pheromones to entropy values, it guides ants to prioritize exploring high-uncertainty regions, thereby optimizing clustering quality; Visualized path tracing: Dynamic heatmaps display the path of entropy changes, supporting end-to-end insights from data to decision-making.
[0019] Multimodal triple filtering combined with a knowledge graph dual-engine architecture improves data accuracy by 40%, achieving an invalid record identification rate of ≥95%, thus addressing the problem of low data quality. End-to-end automated ETL and entropy-priority loading reduce processing time by 50%, supporting second-level response for petabyte-scale data, resolving processing efficiency bottlenecks. An intelligent agent collaboration mechanism and dynamic permission control reduce manual intervention costs by 70%, with permission control precision reaching the field level, lowering management costs. A modular intelligent agent architecture and cloud-native expansion reduce hardware costs by 20%, supporting elastic scaling and automatic scaling down, addressing the issue of limited scalability.
[0020] This solution utilizes an information entropy-filtering rule mapping table to dynamically adjust rules, adapting to changing business needs and enhancing dynamic self-adaptability. Ant colony clustering and information entropy collaborative analysis improve the accuracy of complex decision-making scenarios, significantly increasing decision credibility. The Holon data source coordination system and dynamic anonymization technology meet GDPR / HIPAA regulatory requirements, ensuring data security and compliance. End-to-end auditing and tracing from data extraction to analysis output ensures a transparent, explainable, and fully traceable decision-making process. Attached Figure Description
[0021] Figure 1 This is a flowchart illustrating an embodiment of the present invention. Detailed Implementation
[0022] The following detailed description illustrates the specific implementation method: Example The basics are as follows: Figure 1 As shown, the methods for creating a data warehouse using AI include: In the data extraction step, an AI agent is added to filter the information extracted from the data. The data transformation step involves adding a transformation AI agent to achieve data transformation. The data loading step involves writing the processed data into the target system. The output step of the analysis table involves adding an AI agent to label and filter the generated data warehouse to ensure data accuracy.
[0023] In the data extraction phase, a multimodal information filtering intelligent agent cluster is deployed to achieve efficient access and accurate filtering across different data sources. This cluster includes an NLP agent, an OCR agent, and the Holon data source coordination system: The NLP agent performs entity recognition on unstructured text through semantic analysis, such as identifying key information like company names and project addresses in contract texts; combined with a regular expression matching agent, it performs pattern filtering on structured data, forming a dual-engine filtering mechanism of "semantic understanding + pattern matching." The OCR agent performs optical character recognition on image data, converting tabular data in scanned documents into analyzable text, and accurately locating table boundaries and field areas through layout analysis technology. The Holon data source coordination system achieves unified access and dynamic adaptation of filtering rules for heterogeneous data sources such as databases, log files, and PDF documents through the collaboration of structured data agents, semi-structured data agents, and unstructured data agents.
[0024] Implementation methods for multi-agent Holonic coordination systems, including Step 1: Holon hierarchical modeling, using a four-layer Holon architecture: Physical layer: includes device Holon, such as database interface and OCR engine; Implementation layer: includes data source Holon, and structured / semi-structured / unstructured agents; Collaboration layer: includes task assignment Holon and conflict resolution Holon; Application layer: Includes business logic Holon and visualization Holon; Step 2, cross-type data source adaptation; Holon aggregation mechanism enables unified access to heterogeneous data sources. For example, when processing mixed data sources of database logs and PDF scans, the system automatically coordinates the collaboration of the structured data agent (processing logs), the semi-structured data agent (processing XML / JSON), and the unstructured data agent (processing PDFs), and uses a dynamic rule mapping table to achieve automatic data format conversion and quality verification.
[0025] Step 3: Information Fusion and Self-Similarity Preservation; An information fusion algorithm is implemented during Holon aggregation to ensure seamless integration of lower-level Holon information into higher-level Holons. Fractal coding techniques are employed to maintain system self-similarity; for example, unit Holon inherits local data features from device Holon while simultaneously forming a new global identifier, supporting cross-level data traceability.
[0026] During the data extraction phase, an information entropy-driven intelligent filtering module is implemented. This module calculates the information entropy value of the raw data and performs double verification on unstructured text: first, it uses semantic analysis to identify key entities, such as contract amounts and project dates, and simultaneously calculates field-level information entropy to quantify data uncertainty. For example, when extracting financial data, the system automatically marks the "abnormal invoice amount" field with abnormally high information entropy, and combines it with regular expressions to complete structured data pattern filtering, forming a triple filtering mechanism of "semantic understanding - entropy value verification - pattern matching". The data source Holon coordination system ensures a unified quality standard across data sources by dynamically adjusting the filtering rule weights of each agent. Cross-data source situations include the conversion between database logs and PDF scans.
[0027] The implementation method of the triple filtering mechanism includes: Step 1, Field-level Information Entropy Calculation: Discretize the original data fields and calculate the information entropy value of each field using the Shannon information entropy formula. The Shannon information entropy formula is as follows: ; Where H(X) is the information entropy value of random variable X (unit: bits), which quantifies the uncertainty of data; X is a discrete random variable, such as the set of values for the "invoice amount" field; n is the total number of possible values for variable X, such as the number of amount intervals; Let the variable x take the value The probability can be calculated through frequency statistics, for example... = (Number of occurrences) / (Total number of samples). A base-2 logarithmic operation ensures that the information entropy unit is compatible with the binary system.
[0028] For example, in financial data filtering scenarios, if the "invoice amount" field shows abnormally high frequency of small-amount transactions, such as a large number of 1-yuan transactions, what is the probability distribution? This will result in a spike distribution, causing a significant decrease in the information entropy value H(X), triggering anomaly labeling.
[0029] Step 2, NLP-regular expression dual-engine collaborative filtering; NLP entity recognition uses a BERT pre-trained model to identify entities in contract text, extracting key entities such as company name and project address, and simultaneously calculating entity-level information entropy to verify semantic consistency.
[0030] Regular expression pattern matching filters structured data based on a predefined library of regular expressions (such as date and amount formats), and dynamically adjusts the matching threshold based on information entropy values. For example, when the information entropy value exceeds the threshold, the precision of the regular expression matching is automatically tightened.
[0031] Step 3, dynamic weight adjustment mechanism; An information entropy-filtering rule mapping table is established, and the weight of the filtering rules is dynamically adjusted based on the real-time calculated information entropy value. For example, in a scenario where database logs and PDF scans are processed together, the system automatically identifies fields with high entropy values and prioritizes NLP filtering, while fields with low entropy values are matched using regular expressions, thus achieving adaptive optimization of the filtering strategy.
[0032] The data transformation phase constructs a data governance intelligent agent network to complete the end-to-end processing of data cleaning, standardization, and correlation fusion. The data cleaning agent employs a hybrid approach of rule engines and machine learning, using similar case interpolation algorithms to fill in missing values, information entropy algorithms to detect and correct outliers, and intelligent deduplication of duplicate records. The standardization agent, based on a knowledge graph, drives field mapping to achieve currency unit conversion, date format unification, and coding standard alignment; for example, unifying the "customer ID" field from different systems into a standard coding system. The correlation fusion agent uses association rule mining technology to discover implicit relationships between cross-table fields, combining it with knowledge graphs to achieve multi-source data fusion, forming a composite data view that includes business logic connections such as customer profiles and transaction chains.
[0033] An information entropy outlier detection engine is integrated into the data cleaning agent. This engine accurately locates outliers by calculating the information entropy distribution of numerical fields. For example, when processing sales data, the system automatically identifies "abnormal sales" records with sudden increases in information entropy, and uses a knowledge graph to trace the transaction chain to confirm whether it is a fraudulent transaction or a data entry error. The standardization agent optimizes field mapping rules using the principle of minimizing information entropy, ensuring that operations such as currency unit conversion and date format consistency are minimized to reduce data uncertainty. The association and fusion agent introduces an ant colony clustering algorithm, using pheromones, i.e., information entropy, to guide the data clustering process and achieve efficient association of cross-table fields.
[0034] The entropy-knowledge graph dual-engine implementation method for outlier detection includes: Step 1: Information entropy distribution calculation; bin the numerical fields and calculate the information entropy value of each bin. Identify outliers using the information entropy distribution map, such as "abnormal sales" records with a sudden increase in information entropy in the sales field.
[0035] Step 2, Knowledge Graph Traceability and Verification: A transaction link model is constructed using the knowledge graph to verify marked outliers from multiple dimensions. For example, when abnormal sales figures are detected, the system automatically traces the customer-product-transaction link and uses knowledge graph reasoning to verify whether there are fraudulent transactions or data entry errors.
[0036] Step 3: Dual-engine collaborative decision-making; establish an information entropy-knowledge graph joint decision-making model and determine the outlier handling strategy through a weighted voting mechanism. For example, when the information entropy detection and knowledge graph verification results are consistent, data correction is automatically executed; when the results conflict, a manual review process is triggered to ensure the interpretability and accuracy of the decision.
[0037] The data loading phase utilizes an intelligent loading mechanism to automate writing and real-time verification. The entire loading process is monitored by Holon, a data quality management platform. Real-time stream processing technology is used to write the transformed structured data to the target data warehouse, while a data verification agent is triggered to perform integrity checks, such as verifying the uniqueness of primary key fields and the absence of missing required fields. The system supports incremental loading, capturing changes to source system data through change data capture technology and combining this with a timestamp mechanism to achieve efficient data synchronization, ensuring the data warehouse reflects business changes in real time.
[0038] Holon's data quality features an information entropy priority loading strategy. Based on field-level information entropy assessments, the system prioritizes loading data segments with low entropy values (high importance). For example, when loading customer profile data, the system automatically writes "high-value customer" records to the data warehouse first and triggers a real-time verification process. The incremental loading mechanism, combined with CDC technology and information entropy change monitoring, enables precise capture and synchronization of data changes in the source system, ensuring the data warehouse reflects business dynamics in real time.
[0039] The implementation method of the entropy priority loading strategy includes: Step 1, Field-level Information Entropy Assessment: Information entropy is calculated for the transformed structured data fields. A sliding window mechanism is used to calculate local information entropy, identifying data segments with low entropy values (high importance). For example, in customer profile data, the "high-value customer" field is marked as a priority loading object due to its low information entropy value.
[0040] Step 2, Incremental Loading Mechanism Implementation: Combining Change Data Capture (CDC) technology and timestamp mechanism, accurate capture of data changes in the source system is achieved. Through information entropy change monitoring, when a sudden change in the information entropy value of a key field is detected, the incremental loading process is triggered to ensure that the data warehouse reflects business dynamics in real time.
[0041] Step 3, Real-time Verification and Feedback: During the loading process, a data verification agent is triggered to perform integrity checks (such as primary key uniqueness checks), and an automatic repair process is triggered for abnormal data. A double-buffer mechanism is used to achieve parallel processing of loading and verification, ensuring a balance between loading efficiency and data quality.
[0042] The data credibility assurance system is built during the analysis table output phase, enabling intelligent collaboration between annotation filtering and visualization output. The data quality inspection agent scans the analysis table based on predefined quality rules such as completeness, accuracy, and consistency, generating a quality assessment report and automatically triggering the remediation process. The annotation filtering agent uses NLP and machine learning technologies to automatically annotate text data, such as performing sentiment analysis on customer reviews and subject classification on contract texts; it also combines with a business rule engine to perform conditional filtering, such as filtering sales data by quarter or filtering large transaction records by amount range. The visualization agent presents the filtered data through dynamic charts such as heatmaps and trend lines, supporting users to interactively drill down into the data, forming a progressive output from raw data to business insights. Ultimately, it builds a highly accurate and interpretable data asset system to support enterprise-level decision analysis needs.
[0043] Building upon the labeling and filtering system, an ant colony clustering-information entropy collaborative analysis module is introduced. This module guides the clustering behavior of the ant colony algorithm through information entropy, optimizing the output quality of the analysis table. For example, when generating a sales trend analysis table, the system automatically identifies the "core product" data segment with low information entropy, combines it with sentiment analysis to label customer feedback, and forms a composite analysis view including multi-dimensional indicators such as product popularity and customer satisfaction. The visualization of the agent presents data distribution through dynamic heatmaps, allowing users to trace the path of information entropy changes through interactive operations, achieving end-to-end insight from data to decision-making.
[0044] The implementation method of the information entropy collaborative analysis module includes: Step 1: Pheromone Initialization and Entropy Mapping; Initialize ant colony algorithm parameters (ant count, pheromone evaporation coefficient) and establish a mapping relationship between the pheromone matrix and information entropy values. Information entropy values quantify sample uncertainty, assigning higher pheromone deposition weights to samples with high entropy values to guide ants to prioritize exploring areas of high uncertainty.
[0045] Step 2, Clustering Process Optimization; In each iteration, the ants select a clustering path based on pheromone concentration and information entropy value. The clustering quality is evaluated using the information entropy minimization objective function. The expression for the information entropy minimization objective function is as follows: ; Where J is the clustering objective function value, i.e., the entropy minimization index, the smaller the value, the better the clustering effect; M is the total number of clusters, such as the number of customer groups; N is the total number of samples, such as the number of transaction records. The probability that sample i belongs to class j is calculated using the membership function. The natural logarithm is used, consistent with the Shannon entropy definition.
[0046] For example, in sales trend analysis, the system optimizes the clustering results by minimizing the J value, so that the "core product" data segment (low entropy value region) and the "peripheral product" data segment (high entropy value region) form a clear boundary, thereby improving the interpretability of the analysis table.
[0047] Entropy-driven clustering optimization is achieved by dynamically adjusting the clustering strategy using pheromone update rules. The expression for the pheromone update rule is as follows: ; in, ρ represents the pheromone concentration at time t along path (i,j), which affects the ant selection probability; ρ is the pheromone evaporation coefficient (0<ρ<1), which controls the degree of retention of historical information. This refers to the amount of pheromone added during the period from time t to t+1. The amount of pheromone is related to the sample entropy value, i.e. α is the pheromone deposition coefficient.
[0048] During the clustering process, high-entropy samples (with greater uncertainty) will deposit more pheromones, guiding ants to explore the area first; low-entropy samples will reduce pheromone concentration through a evaporation mechanism to avoid over-focusing.
[0049] Step 3, Visual Path Tracing: When outputting clustering results, a dynamic heatmap is used to display the path of information entropy changes. Users can interactively trace the clustering trajectory of high-entropy samples. For example, in sales trend analysis, the system automatically labels core product data segments and displays their clustering paths, supporting end-to-end insights from data to decision-making.
[0050] This embodiment also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of forming a data warehouse using AI intelligence.
[0051] The above descriptions are merely embodiments of the present invention, and common knowledge such as specific technical solutions and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solutions of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.
Claims
1. A method for forming a data warehouse using AI intelligence, characterized in that, include: In the data extraction step, an AI agent is added to filter the information extracted from the data. The data transformation step involves adding a transformation AI agent to achieve data transformation. The data loading step involves writing the processed data into the target system. The output step of the analysis table involves adding an AI agent to label and filter the generated data warehouse to ensure data accuracy.
2. The method for forming a data warehouse using AI according to claim 1, characterized in that: The data extraction phase deploys a multimodal information filtering intelligent agent cluster for efficient access and filtering across different data sources. The multimodal information filtering intelligent agent cluster includes an NLP agent, an OCR agent, and a data source Holon coordination system, and adopts a triple filtering mechanism of semantic understanding, entropy verification, and pattern matching. The NLP agent performs entity recognition on unstructured text through semantic analysis, and combines regular expression matching agent to perform pattern filtering on structured data; the OCR agent performs optical character recognition on image data, converts table data into analyzable text, and accurately locates table boundaries and field areas through layout analysis technology. The Holon data source coordination system achieves unified access to heterogeneous data sources and dynamic adaptation of filtering rules through the collaboration of three agents: structured data agent, semi-structured data agent, and unstructured data agent.
3. The method for forming a data warehouse using AI intelligence according to claim 2, characterized in that, The triple filtering mechanism includes: Field-level information entropy calculation: The original data fields are discretized, and the information entropy value of each field is calculated using the Shannon information entropy formula; NLP entity recognition uses a BERT pre-trained model to identify entities in contract text, extract key entities, and simultaneously calculate entity-level information entropy to verify semantic consistency. Regular expression pattern matching uses a predefined regular expression library to filter patterns in structured data and dynamically adjusts the matching threshold based on information entropy values. A dynamic weight adjustment mechanism is established by creating an information entropy-filtering rule mapping table and dynamically adjusting the weights of filtering rules based on the real-time calculated information entropy value.
4. The method for forming a data warehouse using AI according to claim 3, characterized in that: During the data transformation phase, a data governance intelligent agent network is constructed to complete the entire process of data cleaning, standardization, and correlation fusion. The data cleaning agent uses a hybrid approach of rule engine and machine learning. It fills missing values with similar case interpolation algorithm, detects and corrects outliers using information entropy algorithm, and performs intelligent deduplication of duplicate records. The standardized Agent is based on knowledge graph-driven field mapping to achieve currency unit conversion, date format unification, and coding standard alignment; The association fusion agent uses association rule mining technology to discover implicit relationships between fields across tables, and combines knowledge graphs to achieve multi-source data fusion, forming a composite data view that includes business logic relationships.
5. The method for forming a data warehouse using AI according to claim 4, characterized in that: Integrate a dual-engine approach—information entropy outlier detection combined with knowledge graph—into the data cleaning agent; Information entropy distribution calculation involves binning numerical fields and calculating the information entropy value of each bin; outliers are identified through the information entropy distribution map. Knowledge graph traceability and verification: Combine knowledge graphs to build a transaction link model and perform multi-dimensional verification of marked outliers; A dual-engine collaborative decision-making approach is adopted, establishing an information entropy-knowledge graph joint decision-making model and determining outlier handling strategies through a weighted voting mechanism. When the information entropy detection and knowledge graph verification results are consistent, data correction is automatically performed; when the results conflict, a manual review process is triggered to ensure the interpretability and accuracy of the decision.
6. The method for forming a data warehouse using AI according to claim 5, characterized in that: During the data loading phase, an intelligent loading mechanism enables automated writing and real-time verification. The loading process is monitored throughout by Holon, a data quality management platform. Real-time stream processing technology is used to write the transformed structured data into the target data warehouse, while simultaneously triggering the data verification agent to perform integrity verification.
7. The method for forming a data warehouse using AI according to claim 6, characterized in that: Holon sets an information entropy priority loading strategy for data quality, prioritizing the loading of low-entropy data segments based on field-level information entropy evaluation results. The implementation method of the entropy priority loading strategy includes: Field-level information entropy evaluation calculates the information entropy of the transformed structured data fields, uses a sliding window mechanism to calculate local information entropy, identifies low-entropy data segments, and marks low-entropy values as priority loading objects; The incremental loading mechanism, combined with change data capture technology and timestamp mechanism, enables accurate capture of data changes in the source system. By monitoring information entropy changes, when a sudden change in the information entropy value of a key field is detected, the incremental loading process is triggered to ensure that the data warehouse reflects business dynamics in real time. Real-time verification and feedback: During the loading process, the data verification agent is triggered to perform integrity verification. Abnormal data is automatically triggered to trigger the repair process. A dual-buffer mechanism is used to achieve parallel processing of loading and verification, ensuring a balance between loading efficiency and data quality.
8. The method for forming a data warehouse using AI according to claim 7, characterized in that: In the analysis table output stage, a data credibility assurance system is built to achieve intelligent collaboration between annotation filtering and visualization output; The data quality inspection agent scans and analyzes tables based on predefined quality rules, generates a quality assessment report, and automatically triggers the remediation process. The annotation and filtering agent uses NLP and machine learning technologies to automatically annotate text data and combines it with a business rule engine to complete conditional filtering. The visualization agent presents the filtered data through dynamic charts, allowing users to drill down into the data through interactive operations.
9. The method for forming a data warehouse using AI intelligence according to claim 8, characterized in that: Based on the annotation and filtering system, an ant colony clustering-information entropy collaborative analysis module is introduced. This module uses information entropy to guide the clustering behavior of the ant colony algorithm, optimizing the output quality of the analysis table. The implementation method of the information entropy collaborative analysis module includes: Pheromones are initialized and entropy values are mapped. Ant colony algorithm parameters are initialized, and the mapping relationship between the pheromone matrix and the information entropy value is established. The uncertainty of samples is quantified by the information entropy value. Samples with high entropy values are assigned higher pheromone deposition weights to guide ants to explore high uncertainty areas first. In the clustering process optimization, ants select clustering paths based on pheromone concentration and information entropy value in each iteration. The clustering quality is evaluated using the information entropy minimization objective function, and the clustering strategy is dynamically adjusted through pheromone update rules to achieve entropy-driven clustering optimization. During the clustering process, high-entropy samples will deposit more pheromones, guiding ants to explore the region first; low-entropy samples will reduce pheromone concentration through a evaporation mechanism to avoid over-focusing. Visualized path tracing: When clustering results are output, a dynamic heatmap is used to display the path of information entropy change. Users can interactively trace the clustering trajectory of high-entropy samples. The system automatically marks core product data segments and displays their clustering paths, supporting full-link insights from data to decision-making.