Data analysis process automatic construction method based on large model

Abstract

The invention relates to the technical field of data analysis, in particular to a data analysis process automatic construction method based on a large model. Firstly, a process construction demand input by a user is obtained; analyzing a process construction demand according to the large language model, and determining a process construction factor output by the large language model; and according to the process construction factors, self-adaptive nodes are generated through a model. And then, according to a directed acyclic graph algorithm and the adaptive nodes, constructing a data analysis process, and optimizing the data analysis process through a genetic algorithm. And finally, performing closed-loop verification on the data analysis process according to the large-model automatic verification system, and displaying a visual flow chart corresponding to the optimized data analysis process to a user. According to the scheme, the actual demand of the user is directly acquired as the starting point, and the close correlation between the construction process and the business target of the user is ensured. And a large language model is utilized to analyze and understand complex natural language input, so that key elements in user requirements can be accurately identified.

Description

Technical Field

[0001] This manual relates to the field of data analysis technology, and in particular to a method for automating the construction of data analysis processes based on large models. Background Technology

[0002] In the era of rapid development of big data and artificial intelligence technologies, data analysis has become a core driving force for enterprise decision-making and scientific research innovation. To lower the technical barriers to data analysis and improve work efficiency, a series of graphical user interface-based visual data analysis platforms have emerged and are widely used. Existing tools, such as open-source platforms like KNIME, provide rich pre-built functional nodes, allowing users to build data processing workflows through drag-and-drop and connection methods, thus intuitively designing and executing complex data analysis tasks. This approach reduces reliance on traditional programming skills to some extent, making workflow construction more intuitive and convenient.

[0003] However, these mainstream visualization platforms still have significant limitations in practical applications. While graphical operation lowers the programming barrier, users still need strong data analysis expertise to accurately understand the function of each node, rationally select node types, and effectively combine and configure them. The entire process relies heavily on the user's personal experience and domain knowledge, requiring manual selection, setup, and connection of nodes one by one. For users lacking a professional background, this manual construction method still poses a significant barrier to use, increasing learning costs and limiting its applicability.

[0004] Therefore, this specification provides a method for automating the construction of data analysis processes based on large models. Summary of the Invention

[0005] This specification provides a method for automating the construction of data analysis processes based on large models, in order to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification: This manual provides a method for automating the construction of data analysis workflows based on large models, including: S1. Requirements for constructing a process to obtain user input; S2. Analyze the process construction requirements based on the large language model, and determine the process construction factors output by the large language model; S3. Based on the factors constructed in the process, adaptive nodes are generated through a dedicated node generation model based on the Transformer architecture; S4. Based on the directed acyclic graph algorithm and the adaptive nodes, construct a data analysis process, and optimize the data analysis process using a genetic algorithm; S5. Based on the preset large-scale model automated verification system, perform closed-loop verification on the optimized data analysis process, and after the closed-loop verification is passed, display the visual flowchart corresponding to the optimized data analysis process to the user.

[0007] Based on the aforementioned technical means, this solution starts by directly acquiring the user's actual needs, ensuring that the subsequent data analysis process is closely related to the user's business objectives. Utilizing the capabilities of large language models to parse and understand complex natural language input accurately identifies key elements in user needs, providing a foundation for subsequent process design. The Transformer architecture effectively processes sequential data and generates corresponding nodes based on factors identified in the previous step. These nodes are "adaptive," meaning they can dynamically adjust according to specific needs, improving the flexibility and applicability of process construction. A directed acyclic graph (DAG) algorithm is used to organize the logical relationships between nodes, ensuring the rationality and executability of the process. The application of genetic algorithms aims to further optimize this process, finding optimal or near-optimal solutions by simulating natural selection processes, improving efficiency and result quality. A closed-loop verification mechanism ensures that the final data analysis process not only meets the initial requirements but is also feasible and effective in practice. Finally, a visual flowchart is provided to the user, helping non-technical personnel understand and confirm the final design results.

[0008] Furthermore, the adaptive nodes described in S2 include data cleaning nodes, time analysis nodes, cause classification nodes, and geographic analysis nodes.

[0009] Based on the aforementioned technical means, the large language model can not only identify abstract "process construction factors" but also accurately map them to specific functional node types, thus enabling it to cope with multimodal and multidimensional business scenarios.

[0010] Furthermore, S5 also includes step S51: After the closed-loop verification is passed, the user is shown the analysis results data of the closed-loop verification corresponding to the optimized data analysis process, the execution report and verification record of the optimized data analysis process, and the data analysis process template to be saved.

[0011] Based on the aforementioned technical means, this solution not only automates construction and optimization but also strengthens three key capabilities: interpretability, traceability, and reusability. Through the analysis results of closed-loop verification, it presents users with specific metrics generated during the verification phase (such as accuracy, runtime, resource consumption, and anomaly detection results), enabling users to quantitatively assess process quality rather than relying solely on black-box output. Execution reports include information such as the execution status, time consumption, input / output samples, and intermediate features of each process node, helping users understand "how the process works" and increasing trust in the automated results. Verification records document test cases, baseline comparisons, and failed attempts during the verification process, forming an audit trail to meet compliance or post-event review needs. Furthermore, it can abstract the successfully verified process into a parameterizable template for direct use or fine-tuning in future similar tasks.

[0012] Furthermore, the large language model described in S2 includes a BERT-based pre-trained model and a RoBERTa-based pre-trained model.

[0013] Furthermore, S3 specifically includes: A dedicated node generation model based on the Transformer architecture calls a private knowledge base related to node and process generation to determine the guidance text related to node and process generation. Based on the aforementioned guidance text, determine the control instructions; The control command is sent to a dedicated MCP using MCP technology to determine the adaptive node generated by the dedicated MCP.

[0014] Based on the aforementioned technical means, a dedicated MCP is used to actually generate nodes, realizing intelligent node generation and automated process construction based on high-quality templates. This significantly improves the professionalism and accuracy of the analysis process and enables large models to deeply call private knowledge bases.

[0015] Furthermore, in S5, after the closed-loop verification is passed, a visual flowchart corresponding to the optimized data analysis process is displayed to the user, specifically including: After the closed-loop verification is passed, a responsive user interface is built using the Web technology stack, and a visual flowchart corresponding to the optimized data analysis process is displayed to the user in the responsive user interface. The visual flowchart is adjusted in response to the user's editing operation on the responsive user interface.

[0016] Based on the above technical means, users can intuitively understand the generated results, flexibly intervene, and collaboratively optimize them.

[0017] Furthermore, S4 also includes step S41: An adaptive node selection technique based on the ASEA mechanism and a pre-set node library are used to adaptively adjust each node in the optimized data analysis process.

[0018] Furthermore, S5 uses a pre-defined large-scale model automated verification system to perform closed-loop verification of the optimized data analysis process, specifically including: Through a pre-set large-scale automated verification system, various test sample data are generated based on the optimized data analysis process. The various test sample data include positive sample data that characterizes the normal operation of the process and various negative sample data that characterize the abnormal operation of the process. For each type of test sample data, the optimized data analysis process is constructed, and the processing result of each node in the optimized data analysis process is determined; Determine whether the processing result matches the expected result contained in the test sample data; If so, the optimized data analysis process is verified through this type of test sample data; If not, the optimized data analysis process is tested, the test results are determined, and the data analysis process is corrected based on the test results until it passes verification.

[0019] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for automating the construction of a data analysis workflow based on a large model.

[0020] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement an automated construction method for a data analysis process based on a large model.

[0021] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: This solution starts by directly understanding the user's actual needs, ensuring that the subsequent data analysis process is closely aligned with the user's business objectives. Leveraging the capabilities of large language models to parse and understand complex natural language input accurately identifies key elements in the user's needs, providing a foundation for subsequent process design. The Transformer architecture effectively processes sequential data and generates corresponding nodes based on factors identified in the previous step. These nodes are "adaptive," meaning they can dynamically adjust according to specific needs, improving the flexibility and applicability of the process. A directed acyclic graph (DAG) algorithm is used to organize the logical relationships between nodes, ensuring the process's rationality and executability. The application of genetic algorithms further optimizes the process by simulating natural selection processes to find optimal or near-optimal solutions, improving efficiency and result quality. A closed-loop verification mechanism ensures that the final data analysis process not only meets the initial requirements but is also feasible and effective in practice. Finally, a visual flowchart is provided to the user, helping non-technical personnel understand and confirm the final design results. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating an automated construction method for a data analysis workflow based on a large model, provided as an embodiment of this specification. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0024] In embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0025] While current data visualization platforms offer user-friendly interfaces and rich functional modules, they still fall short in terms of intelligent and automated workflow support. These platforms generally lack the ability to intelligently recommend or automatically generate workflows based on analytical objectives, data characteristics, or user behavior. This means the entire analysis process remains heavily reliant on human experience and expertise, failing to truly achieve efficient and universally accessible intelligent support for data analysis.

[0026] Therefore, this specification provides a method for automating the construction of data analysis processes based on large models to solve the above problems.

[0027] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0028] Figure 1 A flowchart illustrating an automated data analysis workflow construction method based on a large model, provided in this specification, includes the following steps: S1: Requirements for building a process to obtain user input.

[0029] This specification describes the process of automating the construction of a data analysis workflow based on a large model. In the embodiments described herein, this automated construction process can be executed by a server. However, this specification does not limit the type of device or platform used to perform this automated construction process; for example, a personal computer, mobile terminal, or other such device or platform can also be used. For ease of description, the following description uses a server as the executing entity.

[0030] In one or more embodiments of this specification, the entire automated data analysis workflow construction process begins with the user's natural language input, such as a description of the workflow construction requirement: "I want to analyze road vehicle accident data to identify peak accident times and main causes." Therefore, the server can obtain the user's input workflow construction requirement.

[0031] S2: Analyze the process construction requirements based on the large language model, and determine the process construction factors output by the large language model.

[0032] In one or more embodiments of this specification, after a user inputs a process construction requirement, the server can invoke a large language model to analyze the requirement, understand it, and determine the process construction factors output by the large language model. This large language model can be a BERT-based pre-trained model, a RoBERTa-based pre-trained model, or a generative pre-trained transform model (GPT) such as GPT-4 or Claude. Therefore, the server can use a large language model, such as a BERT-based or RoBERTa-based pre-trained model, to understand the requirement and accurately extract the analysis target (e.g., accident statistical analysis), data type (accident time, location, cause, etc.), and processing requirements (e.g., time period statistics, cause classification, etc.) using advanced natural language processing techniques to obtain the process construction factors. Specifically, the Chinese text input by the user can be preprocessed using the jieba word segmentation tool, and then pre-trained models such as BERT and RoBERTa can be used to extract text features. The deep learning algorithm based on the Transformer architecture can then be used to analyze the user's input process construction requirements. Combined with professional knowledge and historical experience in the private knowledge base, process construction factors can be obtained.

[0033] In road traffic accident analysis scenarios, it can identify that "collision angle" represents a numerical feature, "damage degree" requires hierarchical processing, and "relationship analysis" requires correlation analysis algorithms. It can also infer possible data preprocessing steps based on the knowledge base, such as outlier removal and missing value imputation.

[0034] S3: Based on the process construction factors, generate adaptive nodes through a dedicated node generation model based on the Transformer architecture.

[0035] In one or more embodiments of this specification, the server can generate adaptive nodes based on process construction factors and a dedicated node generation model based on the Transformer architecture. Furthermore, it can further incorporate the ASEA (Adaptive Selection with External Attention) mechanism, referencing a pre-defined node library, to intelligently generate adaptive nodes.

[0036] The adaptive nodes include data cleaning nodes, time analysis nodes, cause classification nodes, and geographic analysis nodes. For example, for accident data analysis, the server will automatically generate data cleaning nodes (handling missing and outlier values ​​in accident records), time analysis nodes (statistically counting accident frequency by hour, day, and week), cause classification nodes (classifying causes based on accident description text), and geographic analysis nodes (analyzing accident location distribution patterns), etc. Each node also has the ability to dynamically adjust its processing strategy based on the characteristics of the input data.

[0037] Specifically, a dedicated node generation model based on the Transformer architecture invokes a private knowledge base related to node and process generation to determine guidance texts relevant to node and process generation. Based on the guidance texts, control instructions are determined. These control instructions are then sent to a dedicated MCP employing Model Context Protocol (MCP) technology to determine the adaptive nodes generated by the dedicated MCP.

[0038] This specification describes how a specialized MCP Server (i.e., a dedicated MCP) was built to enable large models to deeply access a private knowledge base. The MCP Server provides a standardized Application Programming Interface (API), allowing large models to directly access a knowledge base containing thousands of professional data analysis templates. For example, in the field of traffic accident analysis, this includes professional templates such as accident statistics templates, time series analysis templates, and geospatial analysis templates. Through the MCP protocol, large models can invoke domain-specific analysis rules and constraints, such as AIS scoring rules and accident liability determination rules in traffic accident analysis. The MCP Server provides an intelligent retrieval interface, enabling large models to search for similar historical success cases based on current needs, providing a reference for node generation.

[0039] The large-scale model can analyze user needs through the MCP interface and intelligently match the most suitable analysis template from the private knowledge base. For example, for "accident high-incidence period analysis," the system will automatically select a time series analysis template and a statistical summary template. Based on the selected professional template, the large-scale model will obtain the standard parameter configuration and best practice values ​​of the template through the MCP interface to ensure that the generated nodes meet professional standards. The MCP Server supports the combined use of multiple templates, and the large-scale model can intelligently combine different professional templates to build a complete analysis workflow.

[0040] Large models can also dynamically adjust specialized templates based on specific data characteristics through the MCP interface, optimizing parameter configurations and processing strategies. The MCP Server provides a domain knowledge fusion interface, enabling large models to organically combine general pre-trained knowledge with specialized domain knowledge. Furthermore, based on user history and preferences, the MCP Server supports personalized template customization and optimization.

[0041] Therefore, this specification utilizes the MCP protocol to enable large models to access professional templates in a private knowledge base through standardized interfaces. Based on these templates, the system generates configurations and code for data analysis and processing nodes, resulting in nodes with higher professionalism and accuracy. For example, in traffic accident analysis, the system can call professional AIS scoring templates to generate professional analysis nodes that conform to industry standards. The MCP Server supports real-time updates and synchronization of the private knowledge base, ensuring that large models always have access to the latest professional templates and analysis rules. When industry standards or best practices change, the knowledge base update is immediately reflected in the node generation process of the large model. It supports access to knowledge bases from multiple professional fields. Through different MCP Server instances, large models can generate specialized analysis nodes for different industries. For example, financial analysis, medical diagnosis, and traffic management all have dedicated knowledge bases and template libraries. This intelligent private knowledge base calling architecture based on the MCP protocol enables this solution to build a truly professional knowledge-driven automated analysis process, providing a more professional and reliable AI solution for the data analysis field.

[0042] S4: Based on the directed acyclic graph algorithm and the adaptive nodes, construct the data analysis process and optimize the data analysis process using a genetic algorithm.

[0043] In one or more embodiments of this specification, the server can construct a data analysis process based on a directed acyclic graph algorithm and adaptive nodes, and optimize the node connection relationships in the data analysis process through a genetic algorithm to ensure the logic and efficiency of the data analysis process. For example, it can automatically determine the reasonable execution order of data cleaning, time series analysis, and causal analysis.

[0044] In this process, the server can also use adaptive node selection technology based on the ASEA mechanism and a preset node library to further adaptively adjust each node in the optimized data analysis process.

[0045] In the adaptive node selection process, each selected node undergoes parameter adaptive configuration using reinforcement learning. Each node's parameter configuration file contains the specific algorithm selection, parameter values, and execution order.

[0046] S5: Based on the preset large-scale model automated verification system, perform closed-loop verification on the optimized data analysis process, and after the closed-loop verification is passed, display the visual flowchart corresponding to the optimized data analysis process to the user.

[0047] In one or more embodiments of this specification, the server can perform closed-loop verification of the optimized data analysis process according to a preset large-scale model automated verification system. After each node is executed, it immediately calls the large-scale model to perform operations such as result verification and error detection. For example, after the time analysis node is executed, the large language model will verify the rationality of the results and check for data anomalies or statistical errors.

[0048] Specifically, the server uses a pre-defined large-scale automated verification system to generate various test sample data based on an optimized data analysis process. These test sample data include positive samples indicating normal process operation and negative samples indicating abnormal process operation. For each type of test sample data, an optimized data analysis process is constructed, and the processing result of each node in the optimized process is determined. It is then determined whether the processing result matches the expected result contained in that type of test sample data. If yes, the optimized data analysis process passes the verification for that type of test sample data. If not, the optimized data analysis process is tested. The large model thoroughly examines the settings parameters, algorithms used, and output results of each node, using multi-dimensional analysis to identify the specific cause of the error, determine the detection result, and automatically correct the data analysis process based on the detection result. This includes correcting node configurations, adjusting algorithm parameters, or optimizing the process structure. The verification process is then rerun until it passes verification.

[0049] This mechanism enables truly automated quality control. The entire execution process is monitored based on an event-driven architecture, and the execution progress is fed back to the user in real time via the WebSocket protocol. The user can clearly see the running status and verification results of each process, such as incident data cleaning, time analysis, and cause classification.

[0050] Furthermore, after the closed-loop verification is passed, the server will display a visual flowchart corresponding to the optimized data analysis process to the user.

[0051] Of course, after the closed-loop verification is passed, the server will also show the user the analysis results data of the closed-loop verification corresponding to the optimized data analysis process, the execution report and verification record of the optimized data analysis process, and the data analysis process template to be saved, thereby providing the user with an end-to-end automated intelligent data analysis solution.

[0052] This solution automates the construction of data analysis workflows using a visual approach, with the workflow built from any number of adaptive nodes. These nodes are automatically generated by a dedicated node generation model based on the Transformer architecture, and during workflow processing, they invoke a larger model to perform closed-loop self-verification of the processing results. The solution can employ a layered architecture design, achieving precise control over the construction of automated data analysis workflows through deep integration of a large language model, the MCP protocol, and a visual interface.

[0053] based on Figure 1 This paper presents an automated data analysis workflow construction method based on a large model. By directly obtaining the user's actual needs as a starting point, it ensures that the subsequently constructed data analysis workflow is closely related to the user's business objectives. Utilizing the capabilities of a large language model to parse and understand complex natural language input, it accurately identifies key elements in the user's needs, providing a foundation for subsequent workflow design. The model employing the Transformer architecture can effectively process sequential data and generate corresponding nodes based on the factors determined in the previous step. These nodes are "adaptive," meaning they can dynamically adjust according to specific needs, improving the flexibility and applicability of workflow construction. A directed acyclic graph (DAG) algorithm is used to organize the logical relationships between nodes, ensuring the rationality and executability of the workflow. The application of a genetic algorithm aims to further optimize this workflow by simulating the selection process in nature to find the optimal or near-optimal solution, improving efficiency and result quality. A closed-loop verification mechanism ensures that the final data analysis workflow not only meets the initial requirements but is also feasible and effective in practice. Finally, a visual flowchart is provided to the user, helping non-technical personnel understand and confirm the final design results.

[0054] Furthermore, in this specification, after the closed-loop verification is passed, the server can use a web technology stack to build a responsive user interface, displaying the optimized data analysis process and its corresponding visual flowchart to the user. Moreover, in response to user editing operations on the visual flowchart within the responsive user interface, the visual flowchart is adjusted accordingly.

[0055] In one or more embodiments of this specification, the server can implement an event-driven architecture based on a publish-subscribe pattern. For standardized events such as "process construction requirements," seamless collaboration between large models and visual analytics is achieved. The server can build an event-driven layer to implement an event-driven architecture based on a publish-subscribe pattern, including functional modules such as event publishers, event subscribers, event queues, and event handlers, supporting asynchronous event processing and real-time event response. When a user needs to create an analysis process, the server triggers the large model to generate adaptive nodes through an event-driven mechanism. When a node completes execution, the server can trigger a closed-loop self-verification process through the event-driven mechanism.

[0056] This specification also covers data processing involved in the construction of the data analysis workflow. The server can build a data processing layer responsible for actual data operations, including data loading, cleaning, transformation, analysis, and storage. It can utilize Apache Spark and Apache Flink streaming architectures, supporting both batch and real-time processing modes, and capable of handling data scales from KB to TB. Specifically, it can employ data cleaning algorithms (statistical outlier detection methods such as Z-Score and IQR), data transformation algorithms (PCA-based feature reduction and t-SNE-based visualization reduction), data analysis algorithms (integrating the Scikit-learn machine learning library, supporting classification, regression, and clustering algorithms), and data storage algorithms (using Parquet columnar storage format and ORC-optimized row-column hybrid storage).

[0057] Furthermore, for the various types of data involved in the data analysis process, the server can adopt a hybrid storage strategy to meet the storage needs of different data types. This can be achieved through relational databases for storing configuration and user information, graph databases for storing knowledge graphs and process topologies, distributed file systems for storing large data files, and caching systems for providing high-performance data access capabilities.

[0058] In one or more embodiments of this specification, the present invention may adopt a modular architecture design, comprising eight mutually cooperating core functional modules. These modules achieve data interaction and collaborative work through standardized interfaces, forming a complete technological ecosystem. These modules include a rule generation module, a node management module, an automated process construction module, a verification and testing module, a knowledge base module, an event-driven module, an MCP protocol module, and a visualization module.

[0059] Among them, the rule generation module: The rule generation module is responsible for building a standardized node construction rule base, providing fundamental support for the system's intelligent decision-making. This module analyzes the usage patterns of mainstream visualization analysis tools, extracts common node combination rules and connection logic, and forms a reusable rule system. The module employs machine learning methods to analyze numerous successful cases, automatically identifies best practice patterns, and continuously optimizes the content and quality of the rule base.

[0060] Node Management Module: The node management module provides complete adaptive node lifecycle management capabilities, covering the entire process of node creation, configuration, execution, monitoring, and destruction. This module maintains a scalable node library, where each adaptive node is automatically generated from a large model and features standardized input / output interfaces, parameter configuration items, and execution logic. The module supports a dynamic node registration mechanism, allowing third-party developers to develop custom nodes through a standardized Application Programming Interface (API) and seamlessly integrate them into the system, achieving excellent scalability.

[0061] Automated process building module: This module supports two workflow orchestration modes: manual drag-and-drop and AI-driven. In manual mode, users can drag and drop elements through a visual interface, with the system providing real-time connection verification and intelligent error feedback. In AI-driven mode, the system automatically recommends the optimal node combination based on user requirements, intelligently configures node parameters, and optimizes the workflow topology. The module uses a directed acyclic graph to represent the workflow structure and employs a topology sorting algorithm to ensure the correctness of the execution order and the integrity of the data flow.

[0062] Verification test module: The verification testing module implements a multi-level verification system based on closed-loop self-verification, including node-level verification, process-level verification, and system-level verification. This module integrates an intelligent test case generator, capable of automatically generating comprehensive test datasets based on process functional characteristics. The core technological innovation of this module lies in building a complete automated verification process based on a large model. First, the large model automatically generates various test sample data containing normal conditions and various abnormal boundary conditions. Then, it automatically executes the complete analysis process and obtains the processing results. Next, it intelligently compares and analyzes the processing results with the expected results to verify the correctness of the process. If discrepancies are found, the large model deeply examines the parameter settings, algorithm selection, and output results of each node, identifies the cause of the error through multi-dimensional analysis, automatically corrects it, and reruns the verification until all tests pass. The module can employ professional test design methods such as boundary value analysis and equivalence class partitioning, combined with the reasoning capabilities of the large model, to generate test sample data (i.e., test cases) covering various boundary conditions.

[0063] Knowledge base module: The knowledge base module, as the intelligent core of the system, is responsible for storing and managing core knowledge assets such as domain expertise, user preference models, historical success stories, and lessons learned from failures. The module employs knowledge graph technology to construct a unified knowledge representation model, supporting unified storage and efficient retrieval of structured, semi-structured, and unstructured knowledge. The module implements a continuous learning mechanism, automatically extracting new knowledge from user feedback and dynamically updating the knowledge graph to maintain the timeliness and accuracy of the knowledge.

[0064] Event-driven module: The event-driven module implements a high-performance event-driven architecture based on a publish-subscribe pattern, serving as a key technological infrastructure for enabling collaborative work between AI and application systems. The module defines standardized event types such as "process construction requirement analysis requests," "adaptive node generation requests," and "closed-loop verification requests," establishing a seamless collaboration mechanism between large models and visual analysis. The architecture includes core components such as event publishers, event subscribers, priority queues, and asynchronous processors, supporting real-time and batch event processing and ensuring efficient communication and coordination among system components.

[0065] MCP protocol module: The MCP protocol module provides standardized external system integration interfaces, enabling deep integration of large models with various data analysis tools. This module fully implements the four-layer architecture of the MCP protocol: the session layer is responsible for establishing and maintaining persistent connections between AI models and application systems; the transport layer provides a reliable data transmission mechanism, supporting message acknowledgment and retransmission; the message layer defines standardized message formats and protocol specifications; and the security layer provides comprehensive security mechanisms such as authentication, access control, data encryption, and audit logs.

[0066] Visualization module: The visualization module utilizes modern web technology stacks to build a responsive user interface, providing an intuitive graphical interface and real-time feedback. The module integrates a rich library of chart components, supporting functions such as visual editing of processes, real-time status monitoring, and multi-dimensional result display. The interface design follows best practices for user experience, supports custom themes and personalized configurations, and can generate various professional data visualization charts, including statistical charts, geographic information maps, and network topology diagrams, helping users intuitively understand and analyze complex data analysis results.

[0067] The loosely coupled interface design between modules can achieve the architectural principle of high cohesion and low coupling. Each module can be deployed and upgraded independently, supporting the horizontal scaling and high availability deployment of the system.

[0068] In one or more embodiments of this specification, an event-driven module collaboration mechanism is constructed. Deep integration of the large language model and the visualization analysis system is achieved through standardized event interfaces and asynchronous message passing. The event-driven requirement analysis mechanism forms the starting point of the entire automated collaboration process. When a user inputs a process construction requirement description on the interface, the user interface module automatically publishes a "Process Construction Requirement Analysis Request" event. This event structure contains key data such as the user's original input text, session ID, and user context information. The event structure is defined as follows: the payload data field carries complete information such as the user requirement description text, session identifier, user identifier, user operation context, and additional metadata, providing rich contextual support for subsequent intelligent processing. The large model engine precisely listens for such specific events through a subscription mechanism, and immediately initiates the semantic understanding processing flow upon receiving the "Process Construction Requirement Analysis Request" event. This event-driven architecture based on the publish-subscribe pattern cleverly achieves decoupled communication between the large model and the user interface, completely avoiding the performance bottlenecks caused by traditional polling or synchronous calls, and significantly improving system response efficiency. A sophisticated event processing pipeline mechanism is also designed. After the large model engine completes the requirements analysis, it will automatically publish a "Node Query Request" event containing information such as analysis results, functional requirement tags, and data type identification. The node management module intelligently responds to this event, efficiently filters candidate nodes that meet the functional requirements from the node library, and publishes a "Candidate Node Ready" event so that the data analysis process can be adjusted according to the candidate nodes.

[0069] The server also constructs a complete event processing chain, from demand analysis requests to node query requests, candidate node readiness, process plan readiness, and finally process execution requests. Each event has a clear triggering condition and response mechanism, and the various links are closely connected, forming a highly intelligent automated processing pipeline.

[0070] In this process, inter-module communication adopts a fully asynchronous messaging mechanism, which not only avoids direct dependencies between modules, significantly improving the system's scalability and maintainability, but also supports event replay and state recovery functions, significantly enhancing the system's fault tolerance. The load balancing mechanism allows a single event to be processed in parallel by multiple consumers, while event filtering and routing functions further improve processing efficiency. The central event bus architecture provides a solid foundation for the entire event-driven system. The event bus is fully responsible for core functions such as event registration and routing, persistence and replay, priority management, monitoring and statistics, and isolation and handling of abnormal events, ensuring stable and reliable event processing. Through this innovative event-driven mechanism, the system truly achieves a loosely coupled architecture; each module can be deployed and upgraded independently. As long as the external event interface remains unchanged, it will not affect the normal operation of other modules, providing a flexible architectural foundation for the system's continuous evolution.

[0071] In one or more embodiments of this specification, the present invention implements adaptive node technology, a core technical feature not found in traditional visualization and analysis tools. Adaptive nodes overcome the limitations of traditional fixed-function nodes, enabling them to dynamically adjust their behavior patterns and parameter configurations in real time based on data characteristics and analytical needs. The core technical features of adaptive nodes are reflected in their intelligent capabilities across multiple dimensions. Regarding the dynamic node behavior adjustment mechanism, the system implements an adaptive node selection technology based on the ASEA mechanism: nodes can dynamically select the most suitable processing algorithm based on the characteristics of the input data, optimizing node behavior in real time through a selective attention mechanism without predefined assumptions. It supports self-adjustment and parameter optimization functions during node execution. Regarding the adaptive indexing and data processing mechanism, each adaptive node has a built-in adaptive indexing system based on tiles and tree structures, exhibiting powerful data processing flexibility. The system can automatically select the optimal data processing strategy based on data scale and complexity, supporting intelligent switching between real-time data stream processing and batch processing modes, and implementing dynamic optimization and caching strategies for data preprocessing. A deep collaborative learning mechanism between users and adaptive nodes has been established. Nodes can learn from users' operating habits and adjust their behavior patterns. User preferences and node performance form a two-way adaptive optimization loop, supporting personalized node configuration and workflow optimization. In terms of the technical implementation of adaptive nodes, the adaptive node architecture adopts a carefully designed three-layer structure: the node basic information layer contains a unique node identifier and core data processing functions; the adaptive layer integrates four functional modules: a data feature analyzer, a behavior selector, a parameter optimizer, and a performance monitor; the learning module contains three core components: a user preference learner, a pattern recognizer, and an adaptive engine, which together realize the node's intelligent adaptive capabilities.

[0072] Regarding adaptive node types, this invention supports multiple specialized types of adaptive nodes to meet diverse analytical needs: Adaptive data input nodes can automatically select the optimal data loading and parsing strategy based on data source characteristics. Adaptive data cleaning nodes dynamically adjust cleaning rules and methods based on data quality issues. Adaptive analysis nodes select the most suitable analysis algorithm based on the analytical objectives and data characteristics. Adaptive visualization nodes automatically select the optimal visualization method based on data characteristics and user preferences.

[0073] In one or more embodiments of this specification, during closed-loop verification, a pre-defined large-model automated verification system is used as the input for a completed data analysis process (a complete process including multiple adaptive nodes and their connections). The process includes a functional description, such as the analysis objective and expected results for analyzing the relationship between vehicle collision angle and damage degree. Data characteristic information includes basic data information such as data scale, field types, and distribution characteristics. The relevant domain knowledge base includes professional knowledge such as relevant business rules, data constraints, and expected patterns.

[0074] Implementation of an intelligent test data generator based on a large model: Normal test sample data generation: Based on process functions and data characteristics, normal test sample data that conforms to business logic is generated. In the accident analysis scenario, test data containing 1000 accident records is generated, where the collision angle and damage degree show a reasonable negative correlation.

[0075] Anomaly Test Sample Data Generation: Generates test sample data for various boundary conditions and anomaly scenarios. This includes: Generate test sample data containing 5%, 10%, and 20% missing values.

[0076] Test sample data including extreme collision angles (such as -90°, 270°) and unreasonable damage levels.

[0077] The test sample data includes 10% and 30% duplicate records.

[0078] Generate test sample data to address issues such as incorrect date format and incorrect numeric type.

[0079] The large model will generate standard expected results for each test sample based on business logic and statistical principles.

[0080] Then, the entire analysis process is executed fully automatically: Automated process execution: The complete process, including data cleaning, correlation analysis, and visualization, is executed automatically. Each test sample data is run independently to avoid mutual interference between samples.

[0081] Intermediate result collection: Collect the output results of each node, including the number of records after data cleaning, outlier detection results, correlation coefficient calculation results, and other intermediate processing data.

[0082] Performance metrics monitoring: Record performance metrics such as execution time, memory usage, and CPU consumption for each node to ensure efficient process execution.

[0083] Then, in the intelligent result comparison and difference analysis stage, the actual execution results are compared with the expected results from multiple dimensions: Numerical result comparison: Compare the calculated results with the expected results. For example, the difference between the actual calculated correlation coefficient (-0.72) and the expected value (-0.70) is 0.02, which is within the acceptable range.

[0084] Logical consistency verification: Checks whether the processing logic meets expectations. It verifies that outlier nodes were successfully identified and 95% of the outlier data was removed, meeting the expected identification rate requirement of over 90%.

[0085] Boundary condition handling verification: Tests whether the handling of various boundary conditions is correct. It verifies that the missing value handling node can correctly handle missing values ​​up to 20%, but a warning is given when the missing value exceeds 30%.

[0086] Finally, the process moves to the error diagnosis and root cause analysis stage, and the automated correction and re-verification stage. When discrepancies are found between the results and expectations, in-depth error diagnosis is performed, followed by automated correction and re-verification.

[0087] Furthermore, it can output verification reports, including: Test execution report: 50 test samples were generated, including 20 normal samples and 30 abnormal samples. Validation pass rate: Initial pass rate 76%, corrected to 100% Error diagnosis results: 3 parameter configuration errors, 2 inappropriate algorithm selections, and 1 process structure defect were found. Correction suggestions implemented: 6 issues were automatically corrected, reducing verification time from an average of 45 minutes to 8 minutes. Verification history: Saves the complete verification process, including the entire process of error discovery, diagnosis, and correction. The system achieved a fully automated verification process, increasing test coverage from 30% to 95%, reducing verification time from hours to minutes, and improving the error detection rate by 300%. In practical applications of road traffic accident analysis, the system successfully detected several subtle configuration errors, such as inappropriate data preprocessing algorithm selection and unreasonable parameter settings, ensuring the accuracy and reliability of the analysis results. Crucially, the system can automatically correct problems based on error diagnosis results without manual intervention, truly achieving intelligent quality assurance.

[0088] This specification also includes Example 2. A computer-readable storage medium storing a computer program, which, when executed by a processor, implements an embodiment of an automated construction method for a data analysis process based on a large model.

[0089] This specification also includes Example 3. An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described embodiment of an automated construction method for a data analysis process based on a large model.

[0090] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for automating the construction of a data analysis workflow based on a large model, characterized in that, include: S1. Requirements for constructing a process to obtain user input; S2. Analyze the process construction requirements based on the large language model, and determine the process construction factors output by the large language model; S3. Based on the factors constructed in the process, adaptive nodes are generated through a dedicated node generation model based on the Transformer architecture; S4. Based on the directed acyclic graph algorithm and the adaptive nodes, construct a data analysis process, and optimize the data analysis process using a genetic algorithm; S5. Based on the preset large-scale model automated verification system, perform closed-loop verification on the optimized data analysis process, and after the closed-loop verification is passed, display the visual flowchart corresponding to the optimized data analysis process to the user.

2. The method for automating the construction of a data analysis workflow based on a large model as described in claim 1, characterized in that, The adaptive nodes described in S2 include data cleaning nodes, time analysis nodes, cause classification nodes, and geographic analysis nodes.

3. The method for automating the construction of a data analysis workflow based on a large model as described in claim 1, characterized in that, S5 also includes step S51: After the closed-loop verification is passed, the user is shown the analysis results data of the closed-loop verification corresponding to the optimized data analysis process, the execution report and verification record of the optimized data analysis process, and the data analysis process template to be saved.

4. The method for automating the construction of a data analysis workflow based on a large model as described in claim 1, characterized in that, The large language model described in S2 includes a BERT-based pre-trained model and a RoBERTa-based pre-trained model.

5. The method for automating the construction of a data analysis workflow based on a large model as described in claim 4, characterized in that, S3 specifically includes: A dedicated node generation model based on the Transformer architecture calls a private knowledge base related to node and process generation to determine the guidance text related to node and process generation. Based on the aforementioned guidance text, determine the control instructions; The control command is sent to a dedicated MCP using MCP technology to determine the adaptive node generated by the dedicated MCP.

6. The method for automating the construction of a data analysis workflow based on a large model as described in claim 1, characterized in that, S5 displays a visual flowchart of the optimized data analysis process to the user after the closed-loop verification is passed, specifically including: After the closed-loop verification is passed, a responsive user interface is built using the Web technology stack, and a visual flowchart corresponding to the optimized data analysis process is displayed to the user in the responsive user interface. The visual flowchart is adjusted in response to the user's editing operation on the responsive user interface.

7. The method for automating the construction of a data analysis workflow based on a large model as described in claim 1, characterized in that, S4 also includes step S41: An adaptive node selection technique based on the ASEA mechanism and a pre-set node library are used to adaptively adjust each node in the optimized data analysis process.

8. The method for automating the construction of a data analysis workflow based on a large model as described in claim 1, characterized in that, S5 uses a pre-defined large-scale model automated verification system to perform closed-loop verification of the optimized data analysis process, specifically including: Through a pre-set large-scale automated verification system, various test sample data are generated based on the optimized data analysis process. The various test sample data include positive sample data that characterizes the normal operation of the process and various negative sample data that characterize the abnormal operation of the process. For each type of test sample data, the optimized data analysis process is constructed, and the processing result of each node in the optimized data analysis process is determined; Determine whether the processing result matches the expected result contained in the test sample data; If so, the optimized data analysis process is verified through this type of test sample data; If not, the optimized data analysis process is tested, the test results are determined, and the data analysis process is corrected based on the test results until it passes verification.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 8.

10. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 8.

Images