A data exploration method based on analytic tree search and dynamic pruning

Abstract

The application discloses a data exploration method based on analysis tree search and dynamic pruning, comprising receiving an input natural language analysis target and a structured data set, extracting field types, statistical characteristics and representative samples of the data set through a data set abstract generation module to generate a structured data set abstract; calling a question generation module to generate an initial question set based on the analysis target and the data set abstract, corresponding to a first analysis branch in the analysis tree starting from a root node; performing branch expansion recursively on each initial question using a depth-first traversal strategy, including calling an insight generation module, generating sub-questions and recursively expanding until a preset maximum exploration depth is reached by the insight generation module and a dynamic pruning module; and finally, after the traversal is completed, all valid insights are summarized, an analysis summary is generated and the result is output. The application controls the consumption of computing resources under the premise of guaranteeing the logical coherence of multiple rounds of exploration and the reliability of insights, and realizes low-threshold, end-to-end automatic data exploration.

Description

Technical Field

[0001] This invention relates to the field of data search technology applied to scenarios such as Internet platforms, IoT devices, and enterprise information systems, and specifically to a data exploration method based on analytic tree search and dynamic pruning. Background Technology

[0002] With the widespread deployment of internet platforms, IoT devices, and enterprise information systems, the scale of data in areas such as government governance, smart healthcare, and financial risk control has exploded. Data has become a key production factor driving high-quality economic and social development. Data exploration, as a core technological link in realizing the transformation of data value, encompasses multiple steps including data distribution overview, statistical feature calculation, potential pattern discovery, anomaly identification, and visualization. It is characterized by high openness, non-linearity, and iterativeness. Traditional data exploration processes, which rely on manual intervention, require executors to possess statistical analysis knowledge, proficient programming skills, and vertical domain experience. This multi-faceted capability barrier leads to time-consuming exploration processes and heavy cognitive burdens, making it difficult for enterprises to quickly capture patterns and gain effective insights from massive amounts of complex data, severely restricting agile decision-making and business response capabilities.

[0003] To lower the barrier to data exploration, existing technologies have led to two main types of solutions. One type consists of interactive data exploration tools such as Microsoft Power BI and Tableau. These tools lower the programming barrier through graphical components and drag-and-drop operations, but still require manual translation of business needs into specific data extraction and aggregation operations. Analysts must repeatedly try and fail among numerous variables and charts, resulting in high time costs and cognitive burdens. The other type consists of automated data exploration systems such as QuickInsight and DataShot. These systems can automatically mine data patterns based on statistical rules, but the generated insights are fragmented and lack structured logic. Users need to expend significant effort to filter high-value information, making it difficult to form a coherent and business-deep exploration trajectory, and thus unable to effectively support complex decision-making.

[0004] The technological breakthrough of large language models offers a new opportunity to bridge the semantic gap between business semantics and data manipulation. Their capabilities in natural language understanding, logical reasoning, and code generation promise to enable automated data exploration guided by natural language intent. However, directly applying large language models to automated data exploration still has significant limitations: First, large language models rely on autoregressive probability prediction mechanisms, which are prone to logical breaks or "illusions" in long-chain, multi-step reasoning and decision-making processes, causing the exploration trajectory to deviate from business objectives and resulting in insufficient reliability of insights. Second, data exploration inherently possesses the characteristics of multi-round iterations and strong step dependencies. Relying on contextual interactions with full prompts will generate extremely high computational overhead and response latency, limiting its practicality in real-world business scenarios.

[0005] In summary, existing automated data exploration methods struggle to balance exploration depth, result reliability, and system resource consumption. Traditional rule-based methods lack generalization ability, while native large language models suffer from both severe reasoning illusions and high computational costs. Therefore, effectively controlling computational resource consumption while ensuring logical coherence and insight quality across multiple rounds of exploration, and constructing a low-threshold, end-to-end, and resource-efficient intelligent data exploration method, has become a core technical problem urgently needing to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing automated data exploration methods, such as the inability to balance exploration quality and computational efficiency, the tendency for reasoning illusions to occur when large language models are applied to data exploration, and the high resource consumption. This invention provides a data exploration agent method based on analytic tree search and dynamic pruning, which effectively controls computational resource consumption while ensuring the logical coherence of multi-round exploration and the reliability of insights, thereby achieving low-threshold, end-to-end automated data exploration.

[0007] Based on the first main aspect of the present invention, a data exploration method based on analytic tree search and dynamic pruning is provided. This method is based on the collaborative execution of a dataset summarization generation module, a question generation module, an insight generation module, and a dynamic pruning module, and includes the following steps:

[0008] The system receives the input natural language analysis target and structured dataset, and extracts the field types, statistical features and representative samples of the dataset through the dataset summary generation module to generate a structured dataset summary.

[0009] The question generation module is invoked to generate an initial set of questions based on the analysis objectives and dataset summary, corresponding to the first-level analysis branch starting from the root node in the analysis tree;

[0010] For each initial question, a depth-first traversal strategy is used to recursively perform branch expansion: the insight generation module is called to generate insight text, statistical significance score and visualization chart for the corresponding question. If no valid insight is generated, the branch is terminated; the dynamic pruning module is called to integrate statistical significance and semantic relevance to calculate the comprehensive score of the node. If the score is lower than the preset pruning threshold, the branch expansion is terminated; the node results that pass the pruning judgment are retained, sub-questions are generated and recursively expanded until the preset maximum exploration depth is reached;

[0011] After completing the traversal, all valid insights are summarized, an analysis summary is generated, and the results are output.

[0012] In the overall concept of this invention, by constructing an end-to-end automated data exploration paradigm, the open exploration process is structured using an analysis tree structure to ensure logical coherence, and an invalid branch is blocked in real time using a dynamic pruning mechanism to control resource consumption. This solves the core contradictions of traditional automated exploration methods, such as insufficient generalization, the tendency of native large model exploration to produce illusions, and high computational costs, from the overall architecture level, and achieves a balance between exploration depth and operational efficiency.

[0013] Optionally, the dataset summary generation module traverses all structured data tables in the dataset, extracts metadata such as field type, percentage of missing values, and percentage of unique values ​​for each field, and generates descriptive statistics for numerical fields, high-frequency categories for categorical fields, and time ranges for date fields. It then integrates these data with the field sample values ​​into a structured dataset summary in standardized JSON format.

[0014] The structured dataset summary replaces the original full data as a unified input context for all downstream modules, compressing the data input scale and reducing the context window occupation and token consumption of large language models.

[0015] The above solution extracts field metadata, classification statistical features, and sampled values ​​and generates standardized JSON summaries. It replaces the entire original data with lightweight structured information as the input of downstream modules, which significantly reduces the context window occupation and token consumption of large models from the input side. At the same time, the unified data format also reduces the model's understanding bias of heterogeneous data, thereby reducing computing power overhead and improving the running stability and accuracy of subsequent modules.

[0016] Optionally, the process by which the dynamic pruning module calculates the overall score of a node is as follows:

[0017] First, the semantic scoring agent outputs a raw semantic score of 0-5 based on the semantic consistency between the analysis target, the current problem, and the generated insight. The raw semantic score is then normalized and mapped to the [0,1] interval to obtain the final semantic score. ;

[0018] Next, obtain the statistical significance score of the insight generation module, with a value range of [0,1]. ;

[0019] The overall score of the insight nodes is calculated using an equal-weighted average method. The calculation formula is:

[0020] ;

[0021] In the formula, The exploration depth corresponding to the current node. The node number is at the current depth. The overall score is compared with the preset pruning threshold. If the score is lower than the threshold, the branch is determined to have no further expansion value, and the pruning operation is performed to terminate the branch extension.

[0022] The above scheme establishes an objective and quantifiable standard for determining the value of branches. By integrating semantic relevance and statistical significance with equal weights, it avoids the one-sidedness of single-dimensional judgment. It can accurately identify and eliminate low-value branches that deviate from the analysis objectives and lack data support, while avoiding the accidental truncation of effective paths with exploration potential. It provides a rigorous quantitative basis for dynamic pruning and maximizes the quality of exploration while controlling resource consumption.

[0023] Optionally, the insight generation module has a built-in insight recognition pipeline;

[0024] The insight recognition pipeline automatically identifies three typical data patterns in the data based on statistical algorithms: outliers, time trends, and variable correlations, and outputs a statistical significance score with a value range of [0,1].

[0025] The insight recognition pipeline pre-processes the quantitative determination of data patterns, replacing some of the reasoning steps in the large language model that directly analyze data results, reducing the time and computing power cost of model reasoning, and providing quantitative statistical basis for the dynamic pruning module.

[0026] The above solution automatically identifies and quantifies the significance of three typical data patterns—outliers, time trends, and variable correlations—through a pre-processed statistical algorithm. This replaces some of the data analysis and reasoning steps that originally required large language models. It reduces the number of calls to large models and token overhead to improve operational efficiency, while also mitigating the illusionary risks associated with pure large model analysis through objective statistical results. At the same time, it provides reliable quantitative statistical input for the dynamic pruning module.

[0027] Optionally, the specific process of recursively executing branch expansion using a depth-first traversal strategy is as follows:

[0028] The maximum exploration depth is preset to 3, and problem nodes are processed one by one in the order they are arranged, starting from the initial set of problems at level one.

[0029] For each problem node, the corresponding insight result is generated first, and then dynamic pruning is performed. If pruning fails, the process is backtracked to process the next sibling node.

[0030] After retaining insights into the text and visualization results by pruning nodes, the next level of sub-problems is generated and recursively deepened, with the depth parameter increasing synchronously.

[0031] When the recursion depth reaches the preset maximum exploration depth, no more subproblems are generated. The result of the current node is retained and backtracking is performed directly until all branches are traversed.

[0032] The above scheme clarifies the traversal execution rules of the analytic tree. By setting a maximum exploration depth of 3 levels, it sets clear boundaries for the exploration process, avoiding uncontrolled computational power due to unlimited branch expansion. At the same time, the depth-first traversal order, combined with the pruning and backtracking mechanism, can quickly and deeply explore deeper insights along the core path, improving the controllability and execution efficiency of the multi-round exploration process and ensuring the orderly progress of the exploration process.

[0033] Optionally, the question generation module adopts a two-level differentiated generation strategy:

[0034] In the initial problem generation phase, based on the global analysis goal, three non-overlapping initial problems are generated from three complementary dimensions: overall distribution overview, anomaly feature identification, and variable correlation analysis, thus constructing a multi-dimensional parallel exploration entry point;

[0035] In the sub-problem generation stage, based on the insights and conclusions of the previous node, 1 to 3 progressive sub-problems are generated for the core findings. The sub-problems extend along the logical direction of the current insights to avoid meaningless divergence and ensure the logical coherence of the exploration trajectory.

[0036] The above approach controls the exploration branches from the source of problem generation. In the initial stage, problems are generated from three complementary dimensions to ensure the comprehensiveness of the exploration and avoid omission of key dimensions. Sub-problem stages are generated progressively along the upper-level insights to ensure the logical coherence of the exploration and avoid meaningless divergence. At the same time, the number of problems in each round is strictly controlled. While ensuring the breadth and depth of insight coverage, the unnecessary resource consumption caused by redundant branches is reduced from the source.

[0037] Optionally, the insight generation module employs a dual-path processing mechanism:

[0038] Prioritize calling the insight recognition pipeline to perform data pattern recognition. If the recognition is successful, output structured insight text and statistical significance score, and then use a large language model to complete business semantic polishing and interpretation enhancement.

[0039] If the insight recognition pipeline fails to identify a valid pattern, the statistical results are input into the large language model to generate supplementary analysis conclusions. At the same time, the corresponding statistical significance score is set to 0, and the node directly enters the dynamic pruning judgment stage.

[0040] The above solution, through a division of labor model of "statistical algorithms first, large models as a backup," fully leverages the respective advantages of the two types of technologies. Standardized data patterns are processed efficiently through pipelines to reduce overhead, while complex non-standard patterns are supplemented by large models to ensure adaptability. This approach minimizes the average computing power and time cost of the overall exploration while ensuring the ability to generalize to diverse data scenarios, thus balancing operational efficiency and comprehensive exploration.

[0041] Based on a second key aspect of the present invention, a data exploration intelligent agent system based on analytic tree search and dynamic pruning for implementing the aforementioned method is provided, comprising at least a dataset summarization generation module, a question generation module, an insight generation module, and a dynamic pruning module:

[0042] The dataset summary generation module is used to receive structured datasets, extract the field types, statistical features and representative sample information of the datasets, and generate a structured dataset summary.

[0043] The question generation module is used to combine the input natural language analysis target with the structured dataset summary to generate an initial question set, which constitutes the first-level analysis branch of the analysis tree; it is also used to generate progressive sub-questions based on the insight nodes determined by pruning, which expand the next-level branch of the analysis tree.

[0044] The insight generation module has a built-in insight recognition pipeline, which is used to output corresponding insight text, statistical significance scores and visualization charts for problem nodes at each level. When a valid insight cannot be generated, the branch will terminate.

[0045] The dynamic pruning module recursively processes all branches of the analysis tree using a depth-first traversal strategy: for each problem node, it calls the insight generation module to obtain the results, and terminates the current branch directly when the branch termination is triggered; it integrates the statistical significance score and semantic relevance score to calculate the comprehensive value of the node, and terminates the current branch expansion when the comprehensive value is lower than the preset pruning threshold; it retains the results of the pruned nodes and triggers recursive traversal of sub-problems until the preset maximum exploration depth is reached; after the traversal is completed, it summarizes all valid insights, generates an analysis summary, and outputs it.

[0046] The above solution transforms the technical logic at the methodological level into physical system modules with clear functional boundaries and well-defined collaborative relationships, providing a structured implementation carrier for the engineering implementation of the technical solution. The four modules each perform their respective functions and work in a closed loop, which can stably support the entire process of data exploration of analysis tree search and dynamic pruning, ensuring the feasibility and implementability of the methodological solution.

[0047] Based on a third key aspect of the present invention, an electronic device is provided, comprising one or more processors;

[0048] Storage device for storing one or more programs;

[0049] When one or more programs are executed by one or more processors, the one or more processors implement the data exploration method based on analytic tree search and dynamic pruning as described above.

[0050] The above approach maps the methodology onto a hardware execution platform, clarifying that the data exploration technology can be deployed and executed on various electronic devices by running stored programs on a processor, thus expanding the application scenarios and hardware platform scope of the technical solution.

[0051] Based on a fourth key aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed, implements the aforementioned data exploration method based on analytic tree search and dynamic pruning.

[0052] The above solutions solidify the technical solutions into the storage medium in the form of program instructions, covering the software product form, so that the technical solutions can be disseminated and applied through software distribution and installation.

[0053] In terms of the quality and logical reliability of data exploration, this invention represents a significant improvement over existing technologies. Existing automated data exploration systems based on heuristic rules rely on predefined templates, exhibiting weak generalization capabilities and fragmented output insights. They fail to form a coherent exploration trajectory consistent with business logic. Directly employing large language models for multi-round exploration is prone to logical breaks and illusions due to long-range reasoning flaws, resulting in conclusions lacking reliable data support. This invention constructs a problem-driven exploration paradigm using an analytic tree as its core carrier. It achieves progressive deep mining through depth-first traversal and integrates a dynamic pruning mechanism based on statistical significance and semantic relevance to filter effective branches. This ensures both the logical coherence and depth of business analysis in the exploration trajectory, while objective statistical evidence constrains the large model's reasoning process, significantly reducing the risk of illusions and providing insights with stronger decision-support value.

[0054] In terms of computational resource utilization efficiency, this invention effectively addresses the technical problem that increasing the depth of data exploration inevitably leads to a surge in computational overhead. Existing multi-agent exploration solutions driven by large models often significantly increase the number of model interaction rounds to improve analysis depth, resulting in a significant increase in token consumption and execution time, making it difficult to implement in large-scale, routine business scenarios. This invention reduces the size of model input by compressing the data using lightweight structured dataset summaries, thereby reducing context window usage from the source. Standardized data pattern recognition is performed pre-processed through an insight recognition pipeline, replacing part of the large language model inference stage. Then, valueless exploration branches are terminated in real time through dynamic pruning, avoiding redundant computational consumption. With the synergistic effect of multiple mechanisms, the system significantly reduces token consumption and task execution time while maintaining the same insight quality, improving the business practicality and deployment feasibility of the solution. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0056] Figure 1 This diagram illustrates an execution flowchart of a data exploration method based on analytic tree search and dynamic pruning in one embodiment of the present invention.

[0057] Figure 2 This illustration shows a framework diagram of a data exploration method based on analytic tree search and dynamic pruning in one embodiment of the present invention;

[0058] Figure 3 An example algorithm of a data exploration method based on analytic tree search and dynamic pruning is shown in one embodiment of the present invention. Detailed Implementation

[0059] The preferred embodiments of the present invention will be described in detail below to provide a clearer understanding of the purpose, features, and advantages of the invention. It should be understood that the following embodiments are not intended to limit the scope of the invention, but are merely illustrative of the essential spirit of the technical solution of the invention.

[0060] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known techniques associated with the invention may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0061] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0062] The specific meanings of the technical terms that may be used in this invention are explained as follows:

[0063] Analysis Tree: The structured representation of the exploration process proposed in this invention uses the analysis target as the root node, the exploration question as the edge, and the data insight as the node, and uses a tree structure to depict the hierarchical logical relationship of multiple rounds of data exploration.

[0064] Analysis trajectory: The directed sequence path from the root node of the analysis tree to any leaf node, consisting of alternating "problem-insight" links, is the core carrier for measuring the logical coherence of the exploration process.

[0065] Pipeline Insight: The automated data pattern recognition module designed in this invention automatically detects outliers, time trends, variable correlations, and other patterns in the data based on statistical algorithms and outputs statistical significance scores.

[0066] Dynamic pruning strategy: The core optimization mechanism of this invention integrates statistical significance and semantic relevance to calculate the comprehensive value of exploration nodes in real time, and promptly blocks inefficient or deviating reasoning branches.

[0067] Statistical significance score: A quantitative indicator output by the insight identification pipeline, with a value range of [0,1]. It is used to characterize the statistical reliability of data patterns and is one of the core criteria for dynamic pruning.

[0068] Structured Dataset Summary: A standardized, lightweight representation formed by compressing and extracting the field metadata, statistical features, and sampling information of the original structured data table, used to replace the full data input to reduce computational overhead.

[0069] Depth-first traversal exploration: The analytic tree traversal strategy adopted in this invention recursively explores a single exploration branch to the maximum depth before backtracking to process the remaining branches, and combines dynamic pruning to achieve efficient multi-round data exploration.

[0070] In one feasible embodiment, combined with Figure 1 As shown, this invention provides a data exploration method based on analytic tree search and dynamic pruning. This method is based on the collaborative execution of a dataset summarization module, a question generation module, an insight generation module, and a dynamic pruning module, and includes the following steps S100-S400 executed by a computer system:

[0071] Step S100: Receive the input natural language analysis target and structured dataset, extract the field types, statistical features and representative samples of the dataset through the dataset summary generation module, and generate a structured dataset summary;

[0072] Step S200: Call the question generation module to generate an initial question set based on the analysis objective and dataset summary, corresponding to the first-level analysis branch starting from the root node in the analysis tree;

[0073] Step S300: For each initial question, recursively execute branch expansion using a depth-first traversal strategy: call the insight generation module to generate insight text, statistical significance score, and visualization chart for the corresponding question; if no valid insight is generated, terminate the branch; call the dynamic pruning module to integrate statistical significance and semantic relevance to calculate the node's comprehensive score; if the score is lower than the preset pruning threshold, terminate the branch expansion; retain the node results that pass the pruning judgment, generate sub-questions, and recursively expand until the preset maximum exploration depth is reached;

[0074] Step S400: After completing the traversal, summarize all valid insights, generate an analysis summary, and output the results.

[0075] The following basic knowledge may provide the necessary prerequisites for implementing the present invention.

[0076] Prompt word engineering is one of the core foundational technologies for the implementation of this invention and a key means to drive the large language model to adapt to data exploration scenarios. All stages involving the large language model invocation, such as question generation, semantic relevance scoring, business insight refinement, and final analysis and summary generation, are implemented using the zero-sample and few-sample paradigms of prompt word engineering. By designing structured task instructions and format constraints, the general model can be guided to accurately execute subdivided tasks in data exploration scenarios without fine-tuning the parameters of the large language model. This ensures the format standardization, logical consistency, and accuracy of the output content of each module, providing low-cost and highly flexible technical support for the stable operation of each functional module.

[0077] The large language model intelligent agent architecture technology is the underlying architectural foundation for the end-to-end automated data exploration of this invention. The classic intelligent agent architecture includes four core modules: role setting, planning mechanism, memory mechanism, and action execution. This invention, based on this architectural logic, decomposes dataset summarization generation, question generation, insight generation, and dynamic pruning into intelligent agent units with independent functions. Relying on the autonomous planning and state transition capabilities of the intelligent agents, it achieves closed-loop execution from understanding the analysis target to multi-round iterative exploration. This architecture supports the recursive expansion and depth-first traversal of the analysis tree, enabling the system to autonomously complete task decomposition, path decision-making, and result feedback, overcoming the limitations of traditional single-step large model invocation in handling open-ended exploration tasks.

[0078] Statistical analysis and pattern recognition techniques form the theoretical foundation for the insight quantification and dynamic pruning of this invention. The built-in insight recognition pipeline of this invention relies on three classic statistical analysis methods: outlier detection, time trend analysis, and variable correlation analysis. These methods enable the objective quantification and identification of potential patterns in structured data. This technology provides data-driven statistical significance scores for dynamic pruning, complementing the semantic dimension scoring. It avoids the subjectivity and illusion risks of purely large-scale model semantic judgment and replaces some of the large-scale model inference steps with pre-processed statistical algorithms. This is a key technical prerequisite for balancing exploration quality and computational cost in this invention.

[0079] Structured data processing and secure code execution technologies form the engineering foundation for the practical application of this invention. Dataset summary generation requires field-by-field metadata extraction and statistical calculation of structured tables, relying on mature structured data processing technologies to achieve differentiated parsing and aggregation operations for different types of fields. The analysis and visualization code output by the insight generation module must be executed in an isolated, secure sandbox environment. This ensures both data access security and code execution stability, while also providing rendering capabilities for visualization charts. This is the necessary underlying support for transforming the algorithmic logic of this invention into a runnable system.

[0080] Among the aforementioned fundamental technologies, statistical analysis methods and structured data processing are well-known technologies in the fields of data science and computer science, and can be directly implemented by those skilled in the art using general toolkits. However, the scenario-based implementation methods of prompt word engineering, the state transition mechanism of large language model agents, and the configuration rules for secure code execution environments require further explanation as follows.

[0081] All functional modules in this invention that call the large language model adopt a standardized prompt word framework to ensure the stability and consistency of the output results. Specifically, the prompt words for the question generation module must clearly define the number of questions, the direction of exploration, and the progressive logic requirements, and force the output of a plain text question list; the prompt words for the semantic scoring module must clearly define the scoring dimensions and score ranges, requiring the output of integer scores and brief reasons for the judgment; the prompt words for the insight polishing and summary generation modules must limit the output structure and language style to avoid redundant expressions. For stages with strict format requirements, 2-3 sets of few-sample examples can be added to the prompt words to further reduce the randomness of the output. Those skilled in the art can follow this framework and adjust the wording according to specific business scenarios to achieve the corresponding functions.

[0082] This invention employs a centralized scheduling modular intelligent agent architecture. Implementation requires first defining a global analysis tree state data structure. Each node stores six core types of information: question text, insight content, statistical score, semantic score, node depth, and a list of child nodes, serving as the state carrier throughout the entire process. The scheduling logic is implemented using functional recursion: for each node processed, only the necessary context of the current node is passed to the corresponding module, not the entire dialogue history. After processing, the node state is updated. If the node is determined through pruning and its depth is less than the maximum threshold, child nodes are generated and the processing function is recursively called; otherwise, the current branch is terminated and the process backtracks to the previous level. This design avoids meaningless context inflation while ensuring the logical correctness of depth-first traversal.

[0083] The statistical pattern recognition and code execution phases can be implemented using general technologies, requiring only clearly defined and unified configuration rules. The statistical analysis portion can utilize standard libraries such as NumPy and SciPy to perform outlier detection, trend analysis, and correlation calculations. The code execution phase requires deploying an isolated sandbox environment, using Docker containers to limit resource quotas and network permissions, allowing only specified dependency libraries such as Pandas and Matplotlib, setting a 30-second timeout termination mechanism, and extracting only the calculation results and visualization images after execution to ensure safe and stable operation.

[0084] The following provides a possible embodiment. For example... Figure 2 As shown, this embodiment provides an overall framework for implementing the method of the present invention. Its upper layer consists of four core modules: dataset summarization, question generation, insight generation, and dynamic pruning; its lower layer utilizes single or multiple agents in collaboration to implement the functions of each module.

[0085] This embodiment adopts a "problem-driven" data exploration paradigm, whose core operating mechanism is designed as an iterative closed loop of "problem generation - insight feedback - problem regeneration". Under this mechanism, subsequent problem exploration no longer relies on pre-set static rules, but is dynamically deduced based on the problem of the current round and its feedback insights.

[0086] Therefore, the entire exploration process logically unfolds as a progressively expanding analysis tree. To facilitate formal solution, this embodiment models the data exploration process as a depth-first search within the analysis tree. During the search, the system recursively expands along the branches defined by the problem and progressively visits the nodes composed of insights.

[0087] Furthermore, to overcome the computational overhead bottleneck in deep exploration, this embodiment integrates statistical significance and semantic relevance in the search mechanism to calculate node scores, and designs a dynamic pruning strategy. This strategy blocks inefficient or off-target analysis paths by quantifying the value of nodes in real time, thereby improving the system's execution efficiency while ensuring the continuity of multi-round analysis.

[0088] An algorithmic example of the overall process in this embodiment is as follows: Figure 3 As shown. Given a structured dataset With the analysis objective The system first uses the dataset summarization generation module. Constructing a dataset summary (Line 1). Simultaneously initialize the insight results list. With the list of visualization results (Lines 2-3), and call the problem generation module. based on and Generate an initial set of problems (Line 4). The initial problem corresponds to the initial branch in the parse tree that starts from the root node. Subsequently, the system calls the recursive procedure Expand for each initial problem and performs a depth-first traversal starting from depth d = 1 (lines 5–7).

[0089] During the traversal, to ensure the controllability of the analysis process, a depth determination mechanism is introduced. If d > 3, the analysis ends. Otherwise, for the problem... The recursive process first calls the insight generation module. , generation and Corresponding insight text Statistical scores and visualization results (Line 14). If the problem fails to yield effective insights, that is... If the value is NULL, then the node expansion is terminated immediately (lines 15–17). It is a loop variable that represents the number of iterations or the index of an array.

[0090] For successfully generated insight nodes, the system further calls the dynamic pruning module. By comprehensively analyzing the objectives, current problems, insights, and statistical scores, the insight score of each node is calculated. (Line 18). When When the pruning threshold is less than τ, the node is considered to have low value for further expansion, and the search along that branch is stopped (lines 19–21). Here, τ is the pruning threshold, which is set to 0.7 by default in the experiment.

[0091] For nodes identified through pruning, the system saves their insights and corresponding visualizations to the results list (lines 22–23), and then calls the issue generation module again. Based on the analysis objectives Dataset Summary Current issues and its insightful text Generate a set of subsequent subproblems (Line 24).

[0092] Finally, the system recursively calls Expand on all subproblems, thereby continuously expanding the analytic tree in a depth-first manner (lines 25–27).

[0093] After all initial branches and their retainable branches have been traversed, the system summarizes the final retained discrete insights and calls large language models (LLMs) to generate a coherent exploration summary. (Line 8) Finally, the list of insights results is returned. Visualized results list And exploration and summary (Line 9).

[0094] The dataset summary generation process in step S100 is described in one of the following embodiments.

[0095] To support rapid understanding of the dataset, this embodiment designs a unified dataset summary generation module. This module can automatically extract key metadata and detect potential data quality issues in the dataset.

[0096] Given a structured dataset containing one or more structured data tables This module performs the following operations on each data table. First, it extracts the basic structural information and field-level statistical characteristics of the table, including field type, number and proportion of unique values, and number of missing values. For different types of fields, the module employs differentiated statistical strategies:

[0097] For numeric fields, standard descriptive statistics are calculated, including mean, standard deviation, minimum, maximum, and quantiles. For categorical fields, the top k most frequent categories are extracted, sorted by frequency of occurrence (k = 3 by default in this article). For date fields, the time range (minimum and maximum) is extracted. In addition, the module provides a small number of sample values ​​for each column based on random sampling to help understand the semantic content of the fields.

[0098] The above information is organized into a structured JSON format, forming a structured summary of the dataset, denoted as Abstract. The term "dataset summary" will be used consistently below. The ")" indicates this representation. Dataset summarization provides a comprehensive overview of a dataset from multiple dimensions, enabling downstream modules to understand data characteristics without direct access to the raw data. It also provides a consistent dataset summarization interface across heterogeneous data sources, simplifying the preprocessing workflow and offering a unified, lightweight data context interface for external model calls. Dataset summarization generation module The definition of is:

[0099] ;

[0100] The above formula represents the structured dataset Input to dataset summary generation module After processing by the module, a structured dataset summary is output. .

[0101] The problem generation process in step S200 will be described in one of the following embodiments.

[0102] To drive iterative analysis, this embodiment employs a depth-first strategy for recursive expansion of the analysis tree. The root node of the analysis tree does not store information. In the initial stage, the question generation module is based on the analysis objective. and dataset summary The first set of questions is generated; in subsequent stages, a set of sub-questions is generated based on the current question and its generated insights. Formally, the first set of questions is denoted as... Defined as:

[0103]

[0104] in, This represents the problem generation module. The above formula indicates that the analysis target will be... and dataset summary As input, it is passed to the question generation module. Output the initial problem set. Multiple input parameters are separated by commas within parentheses.

[0105] For any problem Generated insight nodes The system further analyzes the target Dataset Summary Generate a set of subsequent subproblems Its definition is:

[0106]

[0107] In the initial stage, three questions are generated to establish diverse analysis entry points; in subsequent rounds, one to three sub-questions are generated each round to control the branch width of the analysis tree and reduce search overhead.

[0108] The insight generation process in step S300 is described in one of the following embodiments.

[0109] Receiving problem and analysis objectives Next, the code generation agent first generates Python code and executes it in a secure sandbox environment to obtain execution results, including structured statistical results and visualization charts. To reduce the time cost and token consumption associated with LLM data analysis results, this embodiment designs an insight recognition tool based on statistical analysis algorithms, named the "Insight Recognition Pipeline." This pipeline has good modularity, can be integrated into different data exploration systems in a plug-and-play manner, and supports reuse and expansion across business scenarios. It has built-in multiple statistical analysis algorithms that can evaluate the significance of candidate patterns, anomalies, or trends from the perspective of statistical significance and output statistical scores. The pipeline can automatically select the corresponding recognition algorithm according to the visualization type and data characteristics, such as identifying significant category differences for bar charts and identifying trend changes and inflection points for line charts. If the recognition is successful, the insight text and a statistical score with a value range of [0,1] are output. After successful recognition, the generated insight text will be input into the auxiliary analysis agent to complete language polishing and business interpretation enhancement. If the insight recognition pipeline fails to identify a valid insight, the statistical results and relevant context are input into the auxiliary analysis agent, which generates supplementary analysis results, setting the statistical score to 0. Finally, the insight text is output, forming a triple together with the statistical score and visualization results.

[0110]

[0111] in, For the generated insight text; For statistical scoring; For the generated visualization charts; This indicates the insight generation module.

[0112] In this embodiment, the insight recognition pipeline is built into the insight generation module as a functional component. As a pre-statistical algorithm unit, it together with the large language model component to form a dual-path processing mechanism. It is the core deployment unit of this invention to control computing power consumption and improve the objectivity of insight. The whole adopts a modular encapsulation and plug-and-play integration method.

[0113] This pipeline is not an independent system-level module, but rather an algorithm processing unit embedded within the insight generation module, deployed in the processing link between the "code execution component" and the "large language model polishing component". Upstream, it connects to the output of the code execution sandbox, downstream it connects to the semantic polishing stage of the large language model, and simultaneously outputs a statistical significance score to the dynamic pruning module as a quantitative basis for pruning decisions.

[0114] The pipeline employs a pluggable sub-unit architecture, deploying three independent algorithm sub-modules and a routing and scheduling unit: the outlier detection sub-unit deploys the Z-score anomaly detection algorithm, corresponding to anomaly pattern recognition for numerical fields; the trend analysis sub-unit deploys the linear regression significance determination algorithm, corresponding to trend pattern recognition for time series data; and the correlation analysis sub-unit deploys the Pearson correlation coefficient calculation algorithm, corresponding to association pattern recognition for multivariate data. The routing and scheduling unit automatically schedules and matches the corresponding algorithm sub-units for computation based on the field type of the input data, the analysis task type, and the visualization chart type. This architecture supports subsequent expansion to add new pattern recognition algorithms and possesses plug-and-play characteristics.

[0115] In the execution chain of insight generation, the pipeline adopts a deployment logic of "priority call, fallback for failure": after the insight generation module completes the execution of data analysis code and obtains structured statistical results, it prioritizes sending the results to the insight recognition pipeline; the pipeline automatically completes pattern recognition and saliency calculation, outputting structured insight text fragments and statistical significance scores with a value range of [0,1]; successfully recognized results are sent to the large language model to complete business semantic polishing and interpretation enhancement. If the pipeline fails to recognize a valid pattern, the statistical results are directly sent to the large language model to generate supplementary analysis conclusions, while the statistical significance score is set to 0, and simultaneously sent to the dynamic pruning module to participate in node value determination.

[0116] In terms of engineering implementation, the insight and recognition pipeline is encapsulated in the form of a Python function library. It relies on general data computing libraries such as Pandas, NumPy, and SciPy to implement underlying statistical algorithms. It is deployed in the same operating environment as the entire data exploration intelligent agent system, without the need for separate deployment of independent services. It has low call latency, low resource overhead, and can be directly integrated and reused.

[0117] The dynamic pruning process in step S300 is described in the following embodiments.

[0118] Given the analytical objectives and search strategies, the number of valid discoverable insights on any dataset is typically limited. Without proper control of the analysis process, ineffective analyses will occur, resulting in a waste of time and resources. Therefore, this embodiment scores each candidate insight based on both statistical significance and semantic relevance, and performs pruning accordingly.

[0119] First, the semantic scoring agent, based on the analysis objective... Current issues Generative Insights To assess consistency between statistical and semantic scores, a semantic score of 0-5 is assigned. Then, to ensure comparability between the statistical and semantic scores, this embodiment normalizes the semantic score to the [0, 1] interval before weighted fusion, resulting in a normalized semantic score. Finally, and The insight score is obtained by performing an equal-weighted average fusion. .

[0120] Finally, the dynamic pruning module The definition is as follows:

[0121]

[0122] Finally, after the traversal of step S400 is completed, all valid insights are summarized, an analysis summary is generated, and the results are output.

[0123] After the depth-first traversal of all branches of the analysis tree is completed, the system first collects all valid insight nodes that have been determined by dynamic pruning and are stored in the global result list. It then organizes them in a structured manner according to the branch hierarchy and exploration logic of the analysis tree, so that the discrete insight nodes correspond to their respective problem derivation links, forming an insight set with logical hierarchy.

[0124] Subsequently, the original analysis objectives, the structured dataset summary, and all the effective insight texts after sorting are input into the large language model. The content framework and expression norms of the summary are constrained by the preset prompt words, guiding the model to extract core discoveries based on existing insights, summarize data patterns, and form a coherent and complete analysis summary. All conclusions are strictly based on the retained effective insights, and no additional content without data support is generated.

[0125] Finally, the system will package and output all effective insight texts, corresponding visualization charts, and final analysis summaries, completing an end-to-end automated data exploration process.

[0126] This invention selects four mainstream methods—PandasAgent, LightVA, AutoGen, and AgentPoirot—as comparative baselines. The base large language model uses two models with different parameter scales: Qwen2.5-72B-Instruct and Qwen2.5-32B-Instruct. The evaluation model is Llama-3-70B-Instruct. The models are deployed using the lmdeploy framework. The 70B-level model is deployed using four NVIDIA A6000 GPUs, and the 32B-level model is deployed using two A6000 GPUs. Comparative verification is carried out from multiple dimensions, including insight quality, trajectory analysis quality, resource overhead, and overall performance.

[0127] Comparative experimental results show that the proposed method exhibits the best overall performance regardless of the base model size. Driven by a 72B model, the method achieves an insight score (I_score) of 0.438, ranking first; under a 32B model, it also consistently ranks second, significantly outperforming basic methods such as PandasAgent.

[0128] In terms of resource consumption, this invention demonstrates a significant architectural advantage. The average execution time (AET) under 32B and 72B settings is 324s and 370s, respectively, with token consumption controlled in the range of 50,000 to 60,000. Compared with AgentPoirot, which has the closest performance, this invention reduces token consumption by an average of more than 45.0% and execution time by an average of about 50.0%. The final comprehensive evaluation index (BPS) is the highest under both model settings, reaching 0.347 and 0.382, respectively, achieving a balance between exploration quality and computational efficiency.

[0129] Module-level ablation experiments verified the necessity of each core component, and the complete model of this invention achieved optimal BPS in the 72B setting. Specifically, removing the insight recognition pipeline (w / o IG) did not significantly decrease the model's insight quality, but the average token consumption in the 37B setting surged from 56949 to 121022, more than doubling, demonstrating that this pipeline can effectively replace the large model inference stage and significantly reduce computational overhead. Removing the dynamic pruning module (w / o DP) increased token consumption by 32.3% and extended execution time, verifying the blocking effect of dynamic pruning on redundant exploration branches. Removing the dataset summarization generation module (w / o SG) caused a decline in all quality indicators, with a slight increase in both time and token consumption, indicating that lightweight summarization can simultaneously ensure analysis quality and operational efficiency.

[0130] Experiments and case studies on the parameter sensitivity of the evaluation indicators further validated the reliability of the proposed method. The trajectory score T_score remained absolutely stable across all methods within the evaluation model temperature range of 0 to 1.0, with the standard deviation controlled within a narrow range of 0.002 to 0.003, demonstrating extremely strong stability. As the comprehensive evaluation indicator BPS was adjusted from the resource efficiency correction coefficient β from 0.10 to 0.20, the proposed method consistently ranked first, indicating that the indicator system is insensitive to parameter settings and exhibits good robustness. Case studies on financial datasets show that the proposed method's quantitative analysis capabilities are superior to the comparative methods, providing clear numerical conclusions while achieving higher information density with minimal runtime and token consumption.

[0131] It should be understood that the program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] The acquisition, storage, and application of user personal information involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0133] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this invention does not impose any limitations on them.

[0134] The technical terms, principles, or means related to the technical solutions of the present invention mentioned in the above embodiments, which are not described in detail above, are all well-known technologies or common practices that are known to those skilled in the art.

[0135] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A data exploration method based on analytic tree search and dynamic pruning, characterized in that, This method is based on the collaborative execution of a dataset summarization generation module, a question generation module, an insight generation module, and a dynamic pruning module, and includes the following steps: The system receives the input natural language analysis target and structured dataset, and extracts the field types, statistical features and representative samples of the dataset through the dataset summary generation module to generate a structured dataset summary. The question generation module is invoked to generate an initial set of questions based on the analysis objectives and dataset summary, corresponding to the first-level analysis branch starting from the root node in the analysis tree; For each initial question, a depth-first traversal strategy is used to recursively perform branch expansion: the insight generation module is called to generate insight text, statistical significance score and visualization chart for the corresponding question. If no valid insight is generated, the branch is terminated; the dynamic pruning module is called to integrate statistical significance and semantic relevance to calculate the comprehensive score of the node. If the score is lower than the preset pruning threshold, the branch expansion is terminated; the node results that pass the pruning judgment are retained, sub-questions are generated and recursively expanded until the preset maximum exploration depth is reached; After completing the traversal, all valid insights are summarized, an analysis summary is generated, and the results are output.

2. The data exploration method based on analytic tree search and dynamic pruning according to claim 1, characterized in that, The dataset summary generation module traverses all structured data tables in the dataset, extracts metadata such as field type, percentage of missing values, and percentage of unique values ​​for each field, and generates descriptive statistics for numerical fields, high-frequency categories for categorical fields, and time ranges for date fields. It then integrates these data with the sampled field values ​​into a structured dataset summary in standardized JSON format. The structured dataset summary replaces the original full data as a unified input context for all downstream modules, compressing the data input scale and reducing the context window occupation and token consumption of large language models.

3. The data exploration method based on analytic tree search and dynamic pruning according to claim 1, characterized in that, The process by which the dynamic pruning module calculates the overall score of a node is as follows: First, the semantic scoring agent outputs a raw semantic score of 0-5 based on the semantic consistency between the analysis target, the current problem, and the generated insight. The raw semantic score is then normalized and mapped to the [0,1] interval to obtain the final semantic score. ; Next, obtain the statistical significance score of the insight generation module, with a value range of [0,1]. ; The overall score of the insight nodes is calculated using an equal-weighted average method. The calculation formula is: ； In the formula, The exploration depth corresponding to the current node. The node number is at the current depth. The overall score is compared with the preset pruning threshold. If the score is lower than the threshold, the branch is determined to have no further expansion value, and the pruning operation is performed to terminate the branch extension.

4. The data exploration method based on analytic tree search and dynamic pruning according to claim 3, characterized in that, The insight generation module has a built-in insight recognition pipeline; The insight recognition pipeline automatically identifies three typical data patterns in the data based on statistical algorithms: outliers, time trends, and variable correlations, and outputs a statistical significance score with a value range of [0,1]. The insight recognition pipeline pre-processes the quantitative determination of data patterns, replacing some of the reasoning steps in the large language model that directly analyze data results, reducing the time and computing power cost of model reasoning, and providing quantitative statistical basis for the dynamic pruning module.

5. The data exploration method based on analytic tree search and dynamic pruning according to claim 1, characterized in that, The specific process of recursively performing branch expansion using a depth-first traversal strategy is as follows: The maximum exploration depth is preset to 3, and problem nodes are processed one by one in the order they are arranged, starting from the initial set of problems at level one. For each problem node, the corresponding insight result is generated first, and then dynamic pruning is performed. If pruning fails, the process is backtracked to process the next sibling node. After retaining insights into the text and visualization results by pruning nodes, the next level of sub-problems is generated and recursively deepened, with the depth parameter increasing synchronously. When the recursion depth reaches the preset maximum exploration depth, no more subproblems are generated. The result of the current node is retained and backtracking is performed directly until all branches are traversed.

6. The data exploration method based on analytic tree search and dynamic pruning according to claim 1, characterized in that, The question generation module employs a two-level differentiated generation strategy: In the initial problem generation phase, based on the global analysis goal, three non-overlapping initial problems are generated from three complementary dimensions: overall distribution overview, anomaly feature identification, and variable correlation analysis, thus constructing a multi-dimensional parallel exploration entry point; In the sub-problem generation stage, based on the insights and conclusions of the previous node, 1 to 3 progressive sub-problems are generated for the core findings. The sub-problems extend along the logical direction of the current insights to avoid meaningless divergence and ensure the logical coherence of the exploration trajectory.

7. The data exploration method based on analytic tree search and dynamic pruning according to claim 1, characterized in that, The insight generation module employs a dual-path processing mechanism: Prioritize calling the insight recognition pipeline to perform data pattern recognition. If the recognition is successful, output structured insight text and statistical significance score, and then use a large language model to complete business semantic polishing and interpretation enhancement. If the insight recognition pipeline fails to identify a valid pattern, the statistical results are input into the large language model to generate supplementary analysis conclusions. At the same time, the corresponding statistical significance score is set to 0, and the node directly enters the dynamic pruning judgment stage.

8. A data exploration agent system based on analytic tree search and dynamic pruning for implementing the method of any one of claims 1-7, characterized in that, It should include at least a dataset summarization generation module, a question generation module, an insight generation module, and a dynamic pruning module. The dataset summary generation module is used to receive structured datasets, extract the field types, statistical features and representative sample information of the datasets, and generate a structured dataset summary. The question generation module is used to combine the input natural language analysis target with the structured dataset summary to generate an initial question set, which constitutes the first-level analysis branch of the analysis tree; It is also used to generate progressive sub-problems based on insight nodes determined by pruning, and to expand the next level of branches of the analysis tree; The insight generation module has a built-in insight recognition pipeline, which is used to output corresponding insight text, statistical significance scores and visualization charts for problem nodes at each level. When a valid insight cannot be generated, the branch will terminate. The dynamic pruning module is used to recursively process all branches of the analysis tree using a depth-first traversal strategy: for each problem node, the insight generation module is called to obtain the result, and the current branch is terminated directly when the branch termination is triggered. The system integrates statistical significance score and semantic relevance score to calculate the comprehensive value of a node. If the comprehensive value is lower than a preset pruning threshold, the current branch expansion is terminated. The results of pruned nodes are retained and sub-problem recursively traversed until the preset maximum exploration depth is reached. After the traversal is completed, all valid insights are summarized, an analysis summary is generated and output.

9. An electronic device, characterized in that, Includes one or more processors; Storage device for storing one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement the data exploration method based on analytic tree search and dynamic pruning as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the data exploration method based on analytic tree search and dynamic pruning as described in any one of claims 1-7.

Images