Full-process visual global data index analysis system and method

Abstract

The invention discloses a full-process visual global data index analysis system and method, and relates to the technical field of big data analysis. The system comprises a base layer, a large model layer, a storage layer, a business layer, a storage layer physical isolation module, a self-adaptive micro-service module, a multi-modal large model agent, an AI driving attribution analysis module, a data management module, a real-time multi-dimensional analysis module, an elastic scheduling module, a multi-dimensional right management and control module, an intelligent tenant management module and the like. According to the method, an attribution analysis result can be obtained more quickly, data changes can be captured in time, a faster decision process can be supported, the root of a problem can be positioned more accurately, powerful support is provided for problem solving, the data value plays a greater role in business optimization and process improvement, deep conversion and improvement of the data value are realized, and the method is suitable for large-scale popularization and application. A solid technical support is provided for data-driven decision making, and the continuity and stability of data value conversion are ensured.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, and in particular to a full-process, visualized, and comprehensive data metrics analysis system and method. Background Technology

[0002] In today's data-driven era, enterprises, despite possessing massive amounts of data, often find themselves in a predicament of "rich data, poor information." Traditional data analysis technologies have many shortcomings in transforming data value, failing to meet enterprises' needs for in-depth data utilization. For example, the inefficiency of real-time multidimensional analysis prevents the timely release of data value, hindering enterprises from quickly seizing market opportunities; insufficient accuracy in attribution analysis makes it difficult to accurately uncover the deeper value behind the data, resulting in a lack of strong support for enterprise decision-making; complex system interactions impede the widespread dissemination of data value, making it difficult for non-technical personnel to directly participate in the data value creation process. Furthermore, inflexible resource scheduling, insufficient granularity of access control, inefficient tenant management, risks of data isolation across multiple projects, and reliance on manual governance in microservice architectures further exacerbate the difficulty of transforming data value. Summary of the Invention

[0003] The technical problem to be solved by this invention is to overcome the above-mentioned problems existing in the prior art. It proposes a full-process visual full-domain data indicator analysis system, which realizes the full life cycle management of enterprise-level indicator data by integrating multi-source heterogeneous data, dynamically constructing indicator analysis models, and displaying them in real time interactively.

[0004] The present invention solves the above-mentioned technical problems through the following technical solution: Firstly, a full-process, visualized, and comprehensive data indicator analysis system is provided, comprising a foundation layer, a large model layer, a storage layer, and a business layer. The foundation layer includes a storage layer physical isolation module, servers, and adaptive microservice modules; The large model layer includes industry knowledge graphs, multimodal large model intelligent agents, and AI-driven attribution analysis modules, etc. The storage layer includes the database, interactive data, indicator data, and data management module; The business layer includes a real-time multidimensional analysis module, an elastic scheduling module, a multidimensional access control module, and an intelligent tenant management module.

[0005] Secondly, a method for full-process visualization and comprehensive data indicator analysis is provided, the method comprising: Identify the data source and collect the data, then perform data access and preprocessing; The real-time multidimensional analysis module stores data and optimizes its structure, quickly displaying the corresponding query results; The AI-driven attribution analysis module builds and trains attribution models, updates the attribution models in real time to identify key factors, and obtains analysis results. Upon receiving a query submitted via natural language, the multimodal large model agent performs semantic parsing, retrieves relevant data, generates query results, and displays them visually. Upon receiving a submitted task, the elastic scheduling module evaluates and allocates resources, executes the task and monitors its progress, and reclaims resources for reuse after the task is completed. The multi-dimensional access control module ensures user authentication, dynamically controls data access permissions, and records audit logs. The intelligent tenant management module monitors the load, generates resource profiles, dynamically adjusts resource allocation, and provides early warnings of resource bottlenecks. Each project is allocated an independent storage cluster through a physical isolation module at the storage layer, which controls data access and tracks data lineage.

[0006] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0007] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described method.

[0008] The positive and progressive effects of this invention are as follows: This invention proposes a full-process, visualized, and comprehensive data indicator analysis system and method. By improving the efficiency of real-time multidimensional analysis, it enables faster acquisition of analysis results and timely capture of data changes, thereby supporting a faster decision-making process. Secondly, the enhanced accuracy of attribution analysis allows for more precise identification of the root causes of problems, providing strong support for problem-solving and enabling data value to play a greater role in business optimization and process improvement. Furthermore, simplified interaction lowers the technical barrier to entry, allowing more interactive terminals to directly participate in the data analysis process, promoting the widespread dissemination and sharing of data value, and achieving deep transformation and enhancement of data value. Simultaneously, improved resource utilization efficiency, enhanced data security, improved tenant management efficiency, and guaranteed system stability all provide solid technical support for data-driven decision-making, ensuring the continuity and stability of data value transformation. The combined effect of these technical benefits gives this invention significant technical advantages and market competitiveness in the field of data processing and analysis. Attached Figure Description

[0009] Figure 1 A schematic diagram of the structure of a full-process visual full-domain data index analysis system provided as an exemplary embodiment of the present invention; Figure 2 The flowchart illustrates a full-process visualization and global data indicator analysis method provided as an exemplary embodiment of the present invention. Detailed Implementation

[0010] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0011] Example 1 Figure 1 This embodiment illustrates the module composition of the full-process visualization and full-domain data indicator analysis system provided, such as... Figure 1 As shown, the system includes a base layer, a large model layer, a storage layer, a business layer, and a presentation layer.

[0012] The foundation layer includes a storage layer physical isolation module, servers, and adaptive microservice modules.

[0013] The large model layer includes industry knowledge graphs, multimodal large model intelligent agents, and AI-driven attribution analysis modules.

[0014] The industry knowledge graph can be generated based on knowledge from one or more industries. This invention is particularly applicable to digital and intelligent decision support in multiple industry scenarios such as manufacturing, energy, finance, retail, and healthcare.

[0015] The storage layer includes the database, interactive data, indicator data, and data management module.

[0016] The business layer includes a real-time multidimensional analysis module, an elastic scheduling module, a multidimensional access control module, and an intelligent tenant management module.

[0017] The presentation layer includes a visualization module and an interactive input module.

[0018] The following provides a detailed description of several improved core modules in the embodiments of the present invention.

[0019] The data management module is responsible for collecting data from multiple data sources in real time or in batches, and for cleaning, transforming and standardizing the data to ensure its quality and consistency.

[0020] In a preferred embodiment, the data source includes, for example, a database, an API, a log file, a data table, etc.

[0021] The AI-driven attribution analysis module can automatically uncover multi-factor association rules in data, improving the accuracy and depth of attribution analysis. Specifically, this module constructs an attribution model based on a graph neural network and updates model parameters in real time through an incremental learning mechanism to adapt to business changes, thereby enhancing the accuracy and depth of attribution analysis. The data used for mining includes indicator data and multi-dimensional analysis results.

[0022] The AI-driven attribution analysis module acquires various data, including indicator data and multidimensional analysis results, and performs knowledge extraction and processing to generate a vector database and an indicator domain knowledge base. The resulting knowledge base is used to train a graph neural network-based attribution model and can also be used to generate attribution analysis thought chains, updating the attribution model parameters in real time through a continuously updating incremental learning mechanism. Finally, the attribution model is used to analyze the causes of indicator fluctuations in real-time indicator queries input through interactive methods and outputs the analysis results.

[0023] The multimodal large-scale model intelligent agent, by deploying a large-scale parameter-level multimodal model and integrating industry knowledge graphs through continuous pre-training technology, achieves zero-sample semantic parsing of domain terms, thereby improving the accuracy of indicator identification in the business scenarios of complex group enterprises. Context-aware multi-turn dialogue capabilities allow users to drill down into analytical dimensions layer by layer using natural language. A dynamic visualization orchestration engine can automatically generate composite visualization templates including trend charts, geographic heatmaps, and relationship networks based on query results, enabling business personnel to obtain decision-level insights without writing code. Users can interact with the system through natural language to obtain decision-level insights, significantly lowering the technical barrier to entry and improving the user experience.

[0024] The real-time multidimensional analysis module employs columnar storage and vector computation methods, combined with dynamic materialized view technology, to achieve sub-second query responses to massive amounts of data and obtain multidimensional analysis results. Furthermore, this module can automatically optimize the storage structure, reduce I / O overhead, and improve query efficiency.

[0025] The elastic scheduling module dynamically allocates computing resources using a reinforcement learning-based scheduling algorithm to ensure the priority of critical tasks. Furthermore, it incorporates fine-grained task isolation technology to reduce the impact of failures and improve resource utilization efficiency.

[0026] The multi-dimensional access control module implements dynamic access control at the row and field levels by constructing an attribute-based access control (ABAC) model. Simultaneously, it combines end-to-end audit logs to record the entire data access and operation process, enhancing data security. This effectively addresses the insufficient granularity of traditional RBAC models.

[0027] In a preferred embodiment, permission management for indicator data includes permission management for objects such as topics, data sources + tables, indicator definitions, and other management dimensions. On the other hand, it also requires permission management for usage scenarios such as indicator definition and development, indicator application data source application, etc.

[0028] The intelligent tenant management module, through the development of a tenant resource profiling engine, automatically adjusts CPU / memory quotas based on business load. Combined with a tenant-level performance monitoring panel, it provides real-time warnings of resource bottlenecks, improving business continuity and resource utilization efficiency, and solving the shortcomings of traditional resource quota adjustments that require downtime for maintenance.

[0029] Regarding the physical isolation system for multi-project data, traditional logical isolation mechanisms pose a risk of data obfuscation. This invention improves the physical isolation module at the storage layer by: 1) allocating an independent distributed storage cluster to each project and achieving strong data distribution isolation through an improved Ceph CRUSH algorithm, ensuring data security during parallel development of multiple projects; 2) developing a cross-project data access gateway and employing a hardware-level trusted execution environment (TEE) to ensure data transmission security; and 3) constructing a project-level metadata lineage graph to automatically track the entire lifecycle of data from acquisition to destruction, ensuring absolute isolation of data assets during parallel development of multiple projects. The improved module reduces the risk of data leakage and improves project switching efficiency.

[0030] The adaptive microservice module introduces service mesh technology to achieve real-time monitoring and automatic scaling of inter-service call chains. Combined with an intelligent circuit breaker mechanism, it ensures system availability, improves fault recovery capabilities, and solves the shortcomings of traditional service governance that relies on manual configuration.

[0031] In this embodiment of the invention, the improved system solution achieves significant breakthroughs in real-time multidimensional analysis, attribution analysis, large model intelligent agents, elastic scheduling, multidimensional permission control, intelligent tenant management, physical isolation of multi-project data, and adaptive microservice architecture through deep integration of AI and system architecture innovation. It has built an intelligent analysis platform covering the entire data lifecycle and provided a new generation of infrastructure for enterprise data-driven decision-making.

[0032] The collaboration of the aforementioned key modules not only improves the efficiency, accuracy, and user experience of data analysis, but also enhances data security, resource utilization efficiency, and system stability, giving this invention significant technical advantages and market competitiveness in the field of data processing and analysis.

[0033] Example 2 Corresponding to the aforementioned embodiment of the full-process visualization and full-domain data indicator analysis system, the present invention also provides an embodiment of the full-process visualization and full-domain data indicator analysis method.

[0034] like Figure 2 As shown, this is a method for full-process visualization and comprehensive data indicator analysis, including: First, data access and preprocessing are carried out. Data sources are identified and collected. Data is then cleaned, transformed, and standardized to ensure data quality.

[0035] Next, the real-time multidimensional analysis module is used to store data and optimize its structure, quickly process user queries, and display the results to support further analysis.

[0036] Meanwhile, the AI-driven attribution analysis module prepares data, builds and trains attribution models, updates them in real time to identify key factors, and displays the analysis results.

[0037] Upon receiving a query submitted via natural language, the large model agent performs semantic analysis, retrieves relevant data, generates query results, and visualizes them, thereby improving the user experience.

[0038] In terms of resource scheduling, after the system submits a task, the elastic scheduling system evaluates and allocates resources, executes the task and monitors the progress, and reclaims resources for reuse after the task is completed.

[0039] In addition, multi-dimensional access control ensures that after user authentication, data access permissions are dynamically controlled and audit logs are recorded.

[0040] The intelligent tenant management module monitors the load, generates resource profiles, dynamically adjusts resource allocation, and provides early warnings of resource bottlenecks.

[0041] The data layer physical isolation module allocates an independent storage cluster to each project, controls data access, and tracks data lineage.

[0042] Finally, the adaptive microservice architecture deploys services, monitors and alerts on faults, and automatically scales up or down service instances to ensure system stability and availability.

[0043] In a preferred embodiment, the foregoing methods or steps can obtain or implement the corresponding modules.

[0044] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative, and the units described as separate modules may or may not be physically separate. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present invention. Those skilled in the art can understand and implement this without creative effort.

[0045] Example 3 This embodiment provides a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement the full-process visualization and full-domain data indicator analysis method described in any of the foregoing embodiments.

[0046] Example 4 This embodiment provides a computer program product, which includes a computer program that, when executed by a processor, is used for the full-process visualization and comprehensive data indicator analysis method described in any of the foregoing embodiments. This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in any of the foregoing embodiments.

[0047] In summary, this invention introduces columnar storage and vector computation techniques, combined with dynamic materialized view technology, to achieve sub-second query response. This method significantly improves real-time analysis efficiency, enabling enterprises to capture valuable information from data more quickly and providing strong support for rapid decision-making. Secondly, to address the issue of insufficient accuracy in attribution analysis, this invention constructs an attribution model based on graph neural networks, updating model parameters in real time through an incremental learning mechanism. This model can automatically mine multi-factor association rules in the data, improving the accuracy and depth of attribution analysis, thereby helping enterprises more accurately locate the root cause of problems and uncover the deeper value behind the data. Furthermore, addressing the complex interaction issues of intelligent data query systems, this invention deploys a multimodal large-scale model with hundreds of billions of parameters, achieving zero-sample semantic parsing of domain terms, multi-turn dialogue capabilities, and dynamic visualization orchestration, thereby easily acquiring valuable information and promoting the widespread dissemination and sharing of data value. Meanwhile, this invention also comprehensively improves the system's resource utilization efficiency, data security, tenant management efficiency, and system stability through technical solutions such as elastic scheduling system, multi-dimensional permission control, intelligent tenant management, multi-project data physical isolation system, and adaptive microservice architecture, providing a solid technical guarantee for the deep transformation of data value.

[0048] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A full-process visual full-domain data indicator analysis system, characterized in that, The system comprises a foundation layer, a large model layer, a storage layer, and a business layer. The foundation layer includes a storage layer physical isolation module, servers, and adaptive microservice modules; The large model layer includes industry knowledge graphs, multimodal large model intelligent agents, and AI-driven attribution analysis modules, etc. The storage layer includes the database, interactive data, indicator data, and data management module; The business layer includes a real-time multidimensional analysis module, an elastic scheduling module, a multidimensional access control module, and an intelligent tenant management module.

2. The full-process visual full-domain data indicator analysis system according to claim 1, characterized in that, The system also includes a presentation layer, which comprises a visualization module and an interactive input module.

3. The full-process visual full-domain data indicator analysis system according to claim 2, characterized in that, The data management module is used to collect data from multiple data sources in real time or in batches, and to clean, transform and standardize the data.

4. The full-process visual full-domain data indicator analysis system according to claim 1, characterized in that, The AI-driven attribution analysis module is used to mine multi-factor association rules in data and perform attribution analysis on enterprise indicator data by combining a graph neural network-based attribution model.

5. The full-process visual full-domain data indicator analysis system according to claim 1, characterized in that, Multimodal large-scale model intelligent agents deploy multimodal large-scale models and integrate industry knowledge graphs through continuous pre-training technology to perform zero-shot semantic parsing of domain terms; based on context-aware multi-turn dialogue capabilities, they support users to drill down and analyze dimensions layer by layer through natural language. Using a dynamic visualization orchestration engine, it automatically generates composite visualization outputs that include trend charts, geographic heatmaps, and relationship networks based on query results.

6. The full-process visual full-domain data indicator analysis system according to claim 5, characterized in that, The real-time multidimensional analysis module uses columnar storage and vector computation methods, combined with dynamic materialized view technology, to achieve sub-second query response to massive amounts of data and obtain multidimensional analysis results. The elastic scheduling module dynamically allocates computing resources by employing a reinforcement learning-based scheduling algorithm to ensure the priority of critical tasks, and reduces the impact of failures by combining fine-grained task isolation technology. The multi-dimensional access control module constructs an attribute-based access control model to achieve dynamic access control at the row and field levels, while combining full-link audit logs to record the entire data access and operation trajectory. The intelligent tenant management module automatically adjusts CPU / memory quotas based on business load by developing a tenant resource profiling engine.

7. The full-process visual full-domain data indicator analysis system according to claim 1, characterized in that, The storage layer physical isolation module allocates an independent distributed storage cluster to each project, develops a cross-project data access gateway, adopts a hardware-level trusted execution environment, constructs a project-level metadata lineage graph, and automatically tracks the entire lifecycle of data from collection to destruction. The adaptive microservice module uses service mesh technology to achieve real-time monitoring and automatic scaling of the inter-service call chain, and combines it with an intelligent circuit breaker mechanism to ensure system availability.

8. A method for full-process visualization and comprehensive data indicator analysis, characterized in that, The method is implemented based on the full-process visualization and full-domain data indicator analysis system described in any one of claims 1-7, and the method includes: Identify the data source and collect the data, then perform data access and preprocessing; The real-time multidimensional analysis module stores data and optimizes its structure, quickly displaying the corresponding query results; The AI-driven attribution analysis module builds and trains attribution models, updates the attribution models in real time to identify key factors, and obtains analysis results. Upon receiving a query submitted via natural language, the multimodal large model agent performs semantic parsing, retrieves relevant data, generates query results, and displays them visually. Upon receiving a submitted task, the elastic scheduling module evaluates and allocates resources, executes the task and monitors its progress, and reclaims resources for reuse after the task is completed. The multi-dimensional access control module ensures user authentication, dynamically controls data access permissions, and records audit logs. The intelligent tenant management module monitors the load, generates resource profiles, dynamically adjusts resource allocation, and provides early warnings of resource bottlenecks. Each project is allocated an independent storage cluster through a physical isolation module at the storage layer, which controls data access and tracks data lineage.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the method of claim 8 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of claim 8.

Images