Data index intelligent carding optimization method and system based on AI enhancement

Abstract

The invention discloses a data index intelligent sorting optimization method and system based on AI enhancement. The system comprises a multi-source data access module used for accessing and preprocessing multi-source data, the multi-source data comprising structured data and unstructured data; the NLP index extraction module is used for realizing automatic extraction and structured output of business indexes in the unstructured data; the dynamic optimization decision module is used for outputting a business index carding strategy so as to adjust the weight of the business index in real time; the visualization and interaction module is used for generating an interpretable blood relationship map and a natural language report and realizing visual monitoring and intelligent interaction of a business index treatment process; and the feedback learning module is used for realizing dynamic optimization of the business index carding strategy through a real-time man-machine cooperation mechanism. According to the method, the efficiency and the accuracy of intelligent sorting of the data indexes are improved, and the self-adaptive capability and the stability are enhanced.

Description

Technical Field

[0001] This invention relates to the field of enterprise data indicator management technology, and more specifically, to an AI-enhanced intelligent sorting and optimization method and system for data indicators. Background Technology

[0002] Data metrics analysis refers to the process of identifying, defining, and classifying business metrics from multi-source heterogeneous data (such as databases, logs, and documents). It is essential to ensure logical consistency, business relevance, and interpretability of these metrics. A systematic approach can establish a standardized and reusable metric system, eliminating redundancy and conflicts.

[0003] Data governance refers to ensuring the quality, security, and availability of data through technical and management means, encompassing aspects such as data standardization, metadata management, and data lineage tracing. Data governance helps enterprises unify data definitions, formats, and calculation logic (e.g., consistent indicator definitions), establishing a data standardization system; ensure data credibility through data cleaning, verification, and monitoring, establishing a data quality assurance system; establish data lineage, providing data traceability capabilities, tracing the entire data chain from generation to consumption; control data access permissions to meet privacy protection regulations (such as GDPR and CCPA), establishing a data security system; and improve data utilization, reduce redundancy and costs, and maximize data value through efficient management.

[0004] AI-Augmented Technology refers to a technological paradigm that enhances the automation, intelligence, and decision-making efficiency of traditional processes by enabling artificial intelligence algorithms to work collaboratively with human intelligence. Its core goal is not to completely replace humans, but rather to empower relevant fields by replacing repetitive and time-consuming manual operations (such as data cleaning and rule configuration), providing data-driven analytical suggestions (such as indicator optimization strategies), and responding in real-time to environmental changes (such as business rule adjustments and data distribution shifts). These automated, decision-supporting, and dynamically adaptive methods empower related fields.

[0005] Intelligent optimization systems are technological systems based on artificial intelligence and optimization algorithms. Through automated modeling, dynamic decision-making, and feedback iteration, they enable real-time adjustment and improvement of complex business indicators or processes. At its core, it achieves the optimal or near-optimal solution to the objective function through a combination of data-driven and algorithm-driven approaches.

[0006] With the continuous development of internet technology, future data service management platforms will evolve in the following directions: 1) Ease of use: Data service management platforms will become more user-friendly, lowering the barrier to entry and improving user experience. 2) Security: Data service management platforms will place greater emphasis on data service security, providing more comprehensive security measures. 3) Efficiency: Data service management platforms will offer higher data processing capabilities to meet the high-efficiency data service needs of internal enterprise applications. 4) Reliability: Data service management platforms will provide higher reliability, ensuring the stable operation of data services. 5) Scalability: Data service management platforms will support more data formats and protocols, facilitating integration with other systems and meeting the ever-changing business needs of enterprises.

[0007] Data metrics analysis is a core component of data governance and business analytics. It aims to systematically identify, define, classify, and correlate key metrics within an enterprise's data assets to support decision-making and optimization. Traditional technologies primarily rely on human experience and rule engines, with core processes including manually defining metrics, static calculations based on structured data, and cross-departmental collaborative management. However, as data scale and business complexity increase, existing technologies are increasingly revealing significant limitations in areas such as multi-source heterogeneous data processing, dynamic adaptability, and scalability.

[0008] 1) Traditional methods rely on business experts to manually define metrics, using tools such as Excel and SQL to complete data extraction, calculation, and cross-departmental collaboration. This approach requires repeated confirmation of metric definitions and logic, is time-consuming (usually several days to weeks), and is prone to ambiguity or conflict in metric definitions due to communication errors, such as differences in the calculation logic for "active users" in different scenarios. Its shortcomings lie in its high dependence on human experience, its inability to cover massive amounts of unstructured data (such as logs and documents), and its inability to adapt to dynamic business needs.

[0009] 2) Existing technologies typically utilize predefined rules (such as SQL scripts and ETL processes) to generate and manage metrics, relying on traditional data warehouse architectures for batch processing of structured data. While such tools can improve efficiency to some extent, the rigidity of rules leads to insufficient flexibility and an inability to respond to temporary changes in business rules (such as adding metrics for promotional activities). Furthermore, static tools lack automated redundancy detection capabilities and struggle to identify logically overlapping metrics (such as semantic duplication between "DAU" and "Daily Active Users"), resulting in wasted storage resources and reduced analytical efficiency.

[0010] 3) Existing technologies lack systematic governance capabilities for addressing noise, missing data, and multi-source heterogeneity in raw data. Abnormal data (such as negative order amounts) often enters the indicator calculation process directly without cleaning, affecting the reliability of the results. Furthermore, indicator lineage (data sources and dependency paths) largely relies on manual drawing, making end-to-end traceability difficult. For example, in financial risk control scenarios, data anomalies may distort the "delinquency rate," but traditional methods struggle to pinpoint the root cause in a timely manner. These shortcomings are concentrated in the lack of automated cleaning and verification mechanisms, and the fragmentation of lineage analysis.

[0011] 4) Existing technologies are mostly designed for specific scenarios, making it difficult to dynamically adjust the indicator system according to changes in business rules (such as policy adjustments or market activities). For example, when a "flash sale conversion rate" indicator needs to be added quickly during a major e-commerce promotion, manual intervention is required to reconfigure the process. In addition, the reusability of indicators across different fields (such as finance and manufacturing) is poor, leading to redundant development and wasted resources. Its core deficiency lies in the lack of real-time feedback and adaptive optimization capabilities, making it difficult to cope with highly complex and time-sensitive scenarios.

[0012] See Figure 1 As shown, existing data indicator processing systems typically consist of the following core modules: Data Acquisition and Preprocessing Module: Responsible for connecting to structured data sources (such as databases and APIs), performing basic data cleaning (duplicate removal and format standardization), and implementing batch data extraction and transformation through an ETL (Extract-Transform-Load) process; Indicator Definition and Management Module: Provides a user interface for business experts to define indicator names, calculation formulas, and business scenarios, and stores indicator metadata (such as indicator definition tables and calculation logic) in a relational database; Indicator Calculation Engine Module: Responsible for performing batch indicator calculations based on predefined rules (such as SQL scripts and statistical formulas), and supports scheduled task scheduling (such as generating reports every morning); Lineage Tracking and Document Management Module: Records indicator source tables and field dependencies through manual annotation or simple rules, and supports generating or exporting static documents (such as Excel or PDF) describing indicator metadata and business meaning; Visualization and Reporting Module: Generates indicator trend charts based on BI tools (such as Tableau and Power BI), and outputs standardized reports (such as weekly and monthly reports) for management decision-making reference.

[0013] Analysis shows that existing data indicator sorting systems meet the needs of enterprises in data indicator sorting and management to a certain extent, but still have the following shortcomings.

[0014] 1) High reliance on manual intervention. Metric definition, data parsing, and lineage maintenance heavily depend on manual operations, leading to inefficiency and a high risk of errors. For example, business experts need to manually define metric logic (such as "GMV = order volume × average order value"), and cross-departmental collaboration can easily result in ambiguities. Furthermore, unstructured data (documents, logs) requires manual annotation and parsing, making it difficult to process at scale.

[0015] 2) Insufficient dynamic adaptability. Rules and logic are rigid and cannot respond to dynamic changes in the business environment (such as adding new data sources or temporary adjustments to metric definitions). For example, when a promotional activity needs to add a "flash sale conversion rate" metric, the ETL script or SQL logic must be manually rewritten. Another example is that preprocessing rules cannot automatically adapt to data distribution shifts (such as changes in field meaning causing cleaning to fail).

[0016] 3) Limited ability to process multi-source heterogeneous data. Existing modules lack sufficient support for unstructured data (text, charts) and real-time streaming data. For example, when extracting indicator definitions from requirements documents, manual parsing is relied upon instead of automated NLP technology. Furthermore, multimodal data such as logs and sensor data are difficult to access and standardize in a unified manner.

[0017] 4) Low level of automation and intelligence. It lacks the ability to automatically detect conflicts, perform root cause analysis, and provide optimization suggestions. For example, it cannot automatically identify semantic duplication issues between "DAU" and "Daily Active Users." Furthermore, abnormal metrics (such as a sudden increase in "user churn rate") require manual investigation of the data source, and intelligent diagnostic reports cannot be generated.

[0018] 5) High maintenance costs and poor scalability. Delayed updates to lineage relationships and documentation, along with data silos across systems, lead to escalating maintenance costs. For example, after a database table structure change, the lineage graph needs to be manually re-annotated, easily causing version inconsistencies. Furthermore, cross-domain (such as finance and manufacturing) indicators have poor reusability, requiring the repeated development of similar logic.

[0019] 6) Performance and resource efficiency bottlenecks. In high-concurrency scenarios, the computing engine experiences response latency, and resource allocation strategies lack optimization. For example, when computing petabyte-scale data, fixed resource allocation leads to task timeouts or uneven cluster load. Furthermore, redundant metrics (such as logically overlapping "user retention rate" and "repeat purchase rate") are not merged, wasting storage and computing power.

[0020] In existing technologies, cloud-based data service management platforms, such as AWS Lambda and Azure Functions, utilize the computing, storage, and network resources provided by the cloud platform to manage data services. A cloud-based data service management platform mainly consists of the following modules: a service registration and discovery module, responsible for registering data services and their corresponding interface information to the platform and enabling automatic service discovery; a load balancing module, responsible for distributing request traffic across multiple service instances to improve system availability and stability; a routing and gateway module, responsible for receiving requests and forwarding them to the appropriate service instances based on the request content; a security and authentication module, responsible for performing security checks on requests, such as identity authentication and authorization, to ensure the secure use of data services; a monitoring and statistics module, responsible for monitoring and statistically analyzing the operational status and performance indicators of data services, enabling administrators to optimize and adjust the system; and a configuration and expansion module, responsible for managing the platform's configuration information and supporting horizontal scaling of the platform.

[0021] Cloud-based data service management platforms, to a certain extent, meet the needs of enterprises in data service management, possessing characteristics such as elasticity and high availability, but still have some shortcomings: 1) Data security and compliance issues. Data services provided by cloud platforms may have data security and compliance issues, which could lead to data leaks and unauthorized use.

[0022] 2) Cost issues. Using data services provided by cloud platforms may incur additional fees, which may not be suitable for cost-sensitive enterprises.

[0023] 3) Dependency issues. Enterprises may need to rely on services provided by cloud platforms, which may lead to technology stack lock-in and affect the enterprise's technological independence.

[0024] Furthermore, data service management platforms based on microservice architecture, such as Dubbo and Spring Cloud, implement data service management through microservice architecture. A data service management platform based on microservice architecture mainly consists of the following modules: Service registration and discovery module: responsible for registering data services and their corresponding interface information to the platform and achieving automatic service discovery; Load balancing module: responsible for distributing request traffic to multiple service instances to improve system availability and stability; Routing and gateway module: responsible for receiving requests and forwarding them to the appropriate service instances based on the request content; Security and authentication module: responsible for performing security checks on requests, such as identity authentication and authorization, to ensure the secure use of data services; Monitoring and statistics module: responsible for monitoring and statistically analyzing the running status and performance indicators of data services, enabling administrators to optimize and adjust the system; Configuration and extension module: responsible for managing the platform's configuration information and supporting horizontal scaling of the platform; Service governance module: responsible for the lifecycle management of microservices, such as service registration, service discovery, service invocation, and service monitoring.

[0025] Data service management platforms based on microservice architecture meet the data service management needs of enterprises to a certain extent, and have features such as modularity and decoupling, but they still have the following shortcomings: 1) High development and maintenance costs. Microservice architecture requires advanced technical skills for development and maintenance, which may result in high development and maintenance costs.

[0026] 2) Higher system complexity. Microservice architecture divides the system into multiple modules, which may lead to higher system complexity and increase the difficulty of system management and maintenance.

[0027] 3) Data consistency issues. Since each service instance in a microservice architecture may have independent data storage, data consistency issues may arise, requiring additional mechanisms to resolve them.

[0028] In summary, existing technologies need to be improved to provide a more efficient and accurate solution for index analysis. Summary of the Invention

[0029] The purpose of this invention is to overcome the shortcomings of the prior art and provide an AI-enhanced intelligent data indicator sorting and optimization method and system. According to a first aspect of the present invention, an AI-enhanced intelligent data indicator sorting and optimization system is provided. The system includes: a multi-source data access module, an NLP indicator extraction module, a dynamic optimization decision-making module, a visualization and interaction module, and a feedback learning module, wherein: The multi-source data access module is used to access and preprocess multi-source data, which includes structured data and unstructured data. The NLP indicator extraction module is used to automatically extract and structure business indicators from unstructured data. The dynamic optimization decision-making module is used to output business indicator analysis strategies to adjust the weights of business indicators in real time. The visualization and interaction module is used to generate interpretable lineage maps and natural language reports, enabling visualized monitoring and intelligent interaction of the business indicator governance process; The feedback learning module is used to dynamically optimize the business indicator sorting strategy through a real-time human-machine collaboration mechanism.

[0030] According to a second aspect of the present invention, an AI-enhanced intelligent data metric sorting and optimization method is provided. The method includes the following steps: Acquire and preprocess multi-source data, including structured and unstructured data; Perform automated extraction and structured output of business metrics from unstructured data; Obtain business metrics analysis strategies to adjust the weights of business metrics in real time; Generate interpretable lineage maps and natural language reports to enable visualized monitoring and intelligent interaction of the business indicator governance process; The business indicator analysis strategy is dynamically optimized through a real-time human-machine collaboration mechanism.

[0031] Compared with existing technologies, the advantages of this invention lie in the fact that the AI-enhanced intelligent data indicator management and optimization method and system it provides systematically solves the core problems of traditional data indicator governance, such as strong reliance on manual intervention, poor dynamic adaptability, and difficulty in processing multi-source heterogeneous data, through a multi-modal AI fusion architecture (such as the DeepSeek-MoE model, sparse attention mechanism, and PPO reinforcement learning) and a real-time feedback learning loop. This invention utilizes an AI-enhanced technical approach to transform data governance from a "cost center" to a "value center," providing enterprises with an efficient, stable, and transparent full-stack solution for achieving data-driven, accurate decision-making in dynamic business environments.

[0032] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0034] Figure 1 This is an architecture diagram of a data indicator processing system in existing technology; Figure 2This is an architecture diagram of an AI-enhanced intelligent data index sorting and optimization system according to an embodiment of the present invention; Figure 3 This is a flowchart of an AI-enhanced intelligent data index sorting and optimization method according to an embodiment of the present invention. Detailed Implementation

[0035] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0036] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0037] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0038] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0039] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0040] See Figure 2 As shown, the provided AI-enhanced intelligent data indicator sorting and optimization system generally includes a multi-source data access module, an NLP indicator extraction module, a dynamic optimization decision-making module, a visualization and interaction module, and a feedback learning module. The multi-source data access module supports unified access to both structured and unstructured data. The NLP indicator extraction module performs indicator extraction and lineage modeling based on a hybrid expert model (MoE). The dynamic optimization decision-making module can adjust indicator weights in real time using a PPO algorithm. The visualization and interaction module generates interpretable lineage maps and natural language reports. The feedback learning module enhances model capabilities based on human feedback collection, effect evaluation, and online learning.

[0041] 1. Multi-source data access module The multi-source data access module adopts a hybrid architecture design, supporting unified access and preprocessing of structured, semi-structured, and unstructured data, which can be collectively referred to as unstructured data. The module includes a dual-engine architecture: a structured gateway and an unstructured parser. For example, the structured gateway is optimized based on the Apache Arrow columnar in-memory format, supporting access from multiple protocols such as MySQL and Kafka, achieving high-throughput, low-latency data stream processing. The unstructured parser integrates the DeepSeek multimodal model, capable of parsing complex formats such as PDFs, Word documents, and scanned documents, achieving an OCR accuracy of 98.5%, a table data extraction accuracy of 96%, and supporting mixed Chinese / English / Japanese text processing. The data preprocessing layer incorporates a dynamic cleaning rule engine, automatically marking dirty data through an anomaly detection algorithm (Isolation Forest) and filling missing values ​​using context-aware interpolation technology, improving cleaning efficiency by 3 times compared to traditional ETL tools. The module employs a distributed caching architecture (Redis Cluster + local memory) to achieve real-time synchronization and version control of multi-source data metadata, supporting second-level metadata retrieval for TB-level data.

[0042] In one embodiment, the multi-source data access module is compatible with 12 types of data source formats (compared to only 5 types in traditional solutions), improves heterogeneous data parsing speed by 220% (the measured PDF parsing time was reduced from 2.1s to 0.7s), and dynamically allocates bandwidth resources through an adaptive flow control algorithm to ensure data access stability (99.9% SLA) in high-concurrency scenarios.

[0043] 2. NLP Indicator Extraction Module The NLP (Natural Language Processing) metric extraction module is built on the DeepSeek-MoE hybrid expert model and is specifically designed for multi-source heterogeneous data. It enables the automated extraction and structured output of business metrics from unstructured documents. The module first processes PDF / Word documents using an OCR and text cleaning engine to remove noise and extract text paragraphs. Then, it employs domain-adaptive prompting engineering (such as "extract the metric name, calculation formula, and business scenario from the text") to guide the DeepSeek-MoE model in parsing semantics. This involves 16 expert sub-models that automatically route based on text features (routing accuracy >95% for scenarios such as finance and e-commerce), outputting structured triples (metric name, formula, and scenario).

[0044] In one embodiment, key technologies of the NLP metric extraction module include: a sparse attention mechanism for handling long text dependencies (supporting 128k token context), multimodal feature fusion (jointly parsing text and tabular data), and a confidence calibration algorithm (automatic data entry when the threshold is >0.9, and manual review triggered when the threshold is below). Compared with traditional methods, this module improves the F1 score of metric extraction in cross-language mixed documents from 0.65 to 0.91, achieves a processing speed of 800 documents / minute (compared to only 120 documents / minute for traditional rule engines), and supports dynamic fine-tuning, such as incrementally updating model parameters every 24 hours through an online learning engine to adapt to changes in business terminology. In terms of interpretability, the module generates a metric extraction source traceability report, annotating key semantic segments and confidence scores to meet enterprise auditing needs.

[0045] 3. Dynamic Optimization Decision Module The dynamic optimization decision-making module, built upon a reinforcement learning framework, comprises a state-aware module, a policy decision-making module, and an action execution module, enabling real-time adjustment of indicator weights and optimal resource allocation. The state-aware module continuously collects multi-dimensional data (indicator weights, resource utilization, business priority) to construct a normalized state vector. The policy decision-making module employs the Proximal Policy Optimization (PPO) algorithm, generating action instructions through an Actor-Critic dual-network architecture. The Actor network predicts the probabilities of actions such as weight adjustment and indicator merging, while the Critic network evaluates state value and calculates the advantage function, ensuring the stability of policy updates (learning rate 3e-4, discount factor 0.99). The action execution module translates decisions into concrete operations, including Kubernetes resource reallocation (response time <500ms) and graph database indicator merging (automatically generated by Neo4j instructions), supporting ±0.1 weight increments and merging of highly similar (>0.85) indicators. The system drives strategy iteration through a real-time reward function (R = 0.6Δ storage cost + 0.3Δ latency - 0.1Δ business impact), updates model parameters hourly, and combines an online learning engine to achieve a feedback loop.

[0046] 4. Visualization and Interaction Module The visualization and interaction module can be built based on a multi-dimensional rendering engine and natural language generation technology to achieve visualized monitoring and intelligent interaction of the indicator governance process. The module includes a lineage graph engine (D3.js + WebGL accelerated rendering), a natural language report generator, and an interactive dashboard (React / Vue framework), which supports three-level lineage tracing (indicator → field → data source), heatmap dependency strength (edge ​​weight 0-1 color gradient), and dynamic drill-down analysis.

[0047] Key technologies in the visualization and interaction module include: Level of Detail (LOD) technology, which automatically simplifies the rendering complexity of ultra-large-scale graphs; and a real-time semantic synchronization mechanism, which feeds back user interaction behaviors (such as node dragging and weight adjustment) to the optimization decision layer via API, triggering real-time policy updates (latency <200ms). A natural language report generator, combined with the DeepSeek model and rule templates, outputs structured optimization suggestions (such as "merging metrics A / B can reduce storage costs by 22%)," and uses attention visualization to annotate key decision-making criteria. Compared to traditional BI tools, this module supports dynamic business scenario adaptation (template generation efficiency improved by 5 times) and interactive response speed improved by 3 times (actually, the rendering time for a graph with tens of thousands of nodes decreased from 8s to 2.5s). It also integrates multi-device synchronous collaboration functions, supporting real-time annotation and comment synchronization across PC and mobile devices, meeting enterprise-level collaborative governance needs.

[0048] 5. Feedback Learning Module The feedback learning module achieves dynamic strategy optimization through a real-time human-machine collaboration mechanism, including human feedback collection, effect evaluation, and an online learning engine. Human feedback collection supports multimodal input (JSON commands / natural language), which is parsed into structured operation commands (such as weight adjustment and indicator merging) by the DeepSeek model, achieving a parsing accuracy of 98.2%. The effect evaluation module employs time-series causal inference, quantifying the long-term impact of the strategy on business indicators (such as GMV and user retention rate) based on the Prophet model, and calculating the comprehensive benefit index of ΔBI (business impact) and resource saving rate. The online learning engine achieves hot model deployment through incremental parameter updates: for the DeepSeek-MoE model, it dynamically adjusts the routing weights of the expert sub-models (learning rate η=0.01); for the PPO policy network, it optimizes the reward function coefficients (α / β / γ) in real time, processing 100,000 feedback data points daily with a model update latency of <10 minutes (compared to >6 hours for traditional batch training).

[0049] The feedback learning module also achieves the following technological breakthroughs: a two-way confidence calibration mechanism, which prioritizes operations with higher confidence (>0.9) when manual corrections conflict with AI suggestions; and a drift suppression algorithm, which controls parameter shifts during model iteration (drift rate <0.1%) through KL divergence constraints (λ=0.2). In practical applications, this module reduces the strategy iteration cycle from 24 hours to 2 hours, increases the manual intervention adoption rate from 38% to 89%, and reduces the volatility (σ) of business indicators after strategy adjustments by 53% in e-commerce scenarios, ensuring the system continuously adapts to dynamic business needs.

[0050] Accordingly, this invention also provides an AI-enhanced intelligent data indicator sorting and optimization method, see [link to relevant documentation]. Figure 3As shown, the method includes: input access: structured, semi-structured and unstructured data can be accessed; DeepSeek processing: generate prompt engineering, and then output multimodal feature fusion; output index triples, such as including {name, formula and scene}; constructing a lineage graph: sparse attention can be used to calculate dependency weights; dynamic optimization decision, such as using a PPO strategy network to update the optimization decision every hour to dynamically adjust the index weights.

[0051] In summary, this invention belongs to the interdisciplinary field of artificial intelligence and data governance. It designs an automated method and system for data indicator analysis and dynamic optimization based on multimodal AI technology, making it particularly suitable for scenarios such as enterprise data asset management and real-time business analysis. Compared to existing technologies, this invention has the following main advantages: 1) The efficiency and accuracy of indicator analysis have been significantly improved. Based on the DeepSeek-MoE hybrid expert model and sparse attention mechanism (supporting 128k token context windows), the system achieves automated metric extraction from unstructured documents, improving accuracy from 72% to 95% compared to traditional methods. The time for sorting through thousands of metrics is reduced from 5 person-days to 0.5 person-days (efficiency improvement of 90%). Furthermore, the system resolves semantic ambiguity issues in cross-language scenarios through a multilingual routing mechanism (mixed parsing of Chinese / English / Japanese with an F1 score of 0.91).

[0052] 2) Dynamic resource utilization optimization and cost reduction Driven by the PPO reinforcement learning algorithm-driven dynamic optimization engine, the system adjusts the indicator weights (in ±0.1 increments) and resource allocation (dynamic CPU / memory allocation) in real time, achieving a 33% reduction in storage costs (82% redundancy indicator merging rate) and a 40% increase in computing resource utilization (CPU utilization in e-commerce scenarios decreased from 83% to 58% in actual tests). The reward function (R=0.6Δstorage + 0.3Δlatency - 0.1Δbusiness impact) ensures that the optimization direction is strongly correlated with business objectives.

[0053] 3) Enhanced system adaptability and stability By combining an online incremental learning engine with a feedback loop mechanism, the strategy iteration cycle is shortened from 24 hours to 2 hours, and the volatility (σ) of long-term business indicators is reduced by 53%. The model parameter drift is suppressed by KL divergence constraint (λ=0.2) (drift rate <0.1%), and the long-term impact of the strategy is quantified by the Prophet model (ΔBI error ±5%), ensuring the stability of the system in dynamic business environments.

[0054] 4) Breakthrough in compatibility of multi-source heterogeneous data Based on Apache Arrow columnar storage optimization and DeepSeek-Vision multimodal parsing technology, it supports access to 12 types of data sources (including PDF, logs, and APIs), improves the parsing speed of unstructured data by 220% (reducing the parsing time of a single document from 2.1s to 0.7s), and achieves OCR accuracy of 98.5% (compared to 92% for traditional solutions), effectively solving the problem of data silos across systems (reducing manual connection costs by 80%).

[0055] 5) Optimization of interpretability and decision-making efficiency The visualization module achieves sub-second response times for lineage graphs of tens of thousands of indicators (frame rate ≥ 60 FPS, time reduced from 8s to 2.5s) through LOD multi-level detail technology and WebGL accelerated rendering; the natural language report generator (T5-3B model) combined with the attention visualization mechanism outputs structured optimization suggestions (such as "merging indicators A / B saves 22% of storage"), which increases the adoption rate of human decision-making from 38% to 89%, and improves decision-making efficiency by 3 times.

[0056] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0057] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0058] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0059] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0060] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0061] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0062] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0063] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0064] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. An AI-enhanced intelligent data indicator sorting and optimization system, comprising: The module includes a multi-source data access module, an NLP metric extraction module, a dynamic optimization decision-making module, a visualization and interaction module, and a feedback learning module, among which: The multi-source data access module is used to access and preprocess multi-source data, which includes structured data and unstructured data. The NLP indicator extraction module is used to automatically extract and structure business indicators from unstructured data. The dynamic optimization decision-making module is used to output business indicator analysis strategies to adjust the weights of business indicators in real time. The visualization and interaction module is used to generate interpretable lineage maps and natural language reports, enabling visualized monitoring and intelligent interaction of the business indicator governance process; The feedback learning module is used to dynamically optimize the business indicator sorting strategy through a real-time human-machine collaboration mechanism.

2. The system according to claim 1, characterized in that, The multi-source data access module includes a structured gateway, an unstructured parser, and a data preprocessing layer. The structured gateway is optimized based on a columnar memory format and supports access via multiple protocols. The unstructured parser integrates the DeepSeek multimodal model for parsing documents in various formats. The data preprocessing layer is used to preprocess the accessed data, including a built-in dynamic cleaning rule engine that automatically marks dirty data using anomaly detection algorithms and fills in missing values ​​in the data using context-aware interpolation technology.

3. The system according to claim 1, characterized in that, The NLP metric extraction module is built based on the DeepSeek-MoE hybrid expert model, which includes multiple expert sub-models, and performs the following steps: The text is processed using OCR and a text cleaning engine to remove noise and extract text paragraphs from unstructured text in various formats. Domain-adaptive prompting engineering is employed to guide the DeepSeek-MoE hybrid expert model in parsing semantics. The multiple expert sub-models automatically route based on text features and output structured triples, which include the indicator name, formula, and scenario.

4. The system according to claim 3, characterized in that, The dynamic optimization decision-making module includes a state awareness module, a strategy decision-making module, and an action execution module. The state awareness module continuously collects multi-dimensional data and constructs a normalized state vector. The strategy decision-making module adopts a near-end policy network optimization algorithm and generates action instructions through an Actor-Critic dual-network architecture. The Actor network predicts the probability of actions, which include adjusting the weights of business indicators or merging business indicators. The Critic network evaluates the state value and calculates the advantage function. The action execution module translates the decision into specific operations.

5. The system according to claim 1, characterized in that, The visualization and interaction module is built on a multi-dimensional rendering engine and natural language generation technology to realize the visualization monitoring and intelligent interaction of the business indicator governance process. It includes a lineage graph engine, a natural language report generator and an interactive dashboard, and supports three-level lineage tracing, heat mapping dependency strength and dynamic drill-down analysis.

6. The system according to claim 4, characterized in that, The feedback learning module includes a human feedback collection module, an effect evaluation module, and an online learning engine. The human feedback collection module supports multimodal input of commands and natural language and parses it into structured operation commands. The effect evaluation module uses time series causal inference to quantify the long-term impact of the strategy on business indicators and calculates a comprehensive benefit index of business impact and resource saving rate. The online learning engine achieves hot model deployment through incremental parameter updates.

7. The system according to claim 6, characterized in that, The online learning engine achieves hot model deployment through incremental parameter updates, including: dynamically adjusting the routing weights of expert sub-models for the DeepSeek-MoE hybrid expert model; and optimizing the reward function coefficients in real time for the near-end policy network.

8. The system according to claim 6, characterized in that, The long-term impact of the quantification strategy on business metrics is implemented based on the Prophet model.

9. A data indicator intelligent sorting and optimization method based on AI enhancement, comprising the following steps: Acquire and preprocess multi-source data, including structured and unstructured data; Perform automated extraction and structured output of business metrics from unstructured data; Obtain business metrics analysis strategies to adjust the weights of business metrics in real time; Generate interpretable lineage maps and natural language reports to enable visualized monitoring and intelligent interaction of the business indicator governance process; The business indicator analysis strategy is dynamically optimized through a real-time human-machine collaboration mechanism.

10. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, it implements the steps of the method according to claim 9.

Images