Structured data automatic acquisition, analysis and execution method and system

Abstract

The application discloses a structured data automatic acquisition analysis and execution method and system. The method comprises the following steps: firstly, acquiring a to-be-processed file and performing quality checking and standardization processing. Then, field-level semantic analysis and vectorization processing are performed based on the standardized data, and a knowledge index is constructed. By analyzing a user natural language request, combining vector retrieval and risk assessment to ensure its safety and executability, an ordered structured analysis plan is generated. Subsequently, the plan is converted into code to run in a restricted environment, and a closed-loop correction mechanism is used to ensure the success of the execution. The analysis result not only automatically identifies a chart type and generates ECharts rendering parameters to realize visual display, but also comprehensively analyzes a process and a result to automatically generate a summary report in a natural language form. Through the method of the application, an end-to-end solution capable of automatically identifying a file structure, safely executing an analysis task and having a self-correction capability is realized.

Description

Technical Field

[0001] This invention relates to data processing methods, and more specifically to methods and systems for the automatic acquisition, analysis and execution of structured data. Background Technology

[0002] With the acceleration of enterprise digital transformation, structured data files such as Excel, CSV, Parquet, and database exported tables have become important methods for business data storage and exchange. However, traditional data analysis processes rely on data analysts or business personnel manually writing scripts in Python, SQL, VBA, etc., for data processing, which brings several challenges. First, these methods require users to have certain programming knowledge and technical background, such as familiarity with library functions like pandas, NumPy, and openpyxl, which increases the learning cost and reduces collaboration efficiency.

[0003] Furthermore, manually written scripts are prone to errors during execution and are difficult to trace because they lack unified syntax checking and runtime protection mechanisms. Even minor changes to field names can disrupt the entire data processing chain, and recovery is exceptionally complex due to the lack of automatic rollback functionality. Meanwhile, local Excel macros or external online tools often fail to meet enterprise-level data security and compliance requirements, particularly regarding data leaving the domain, not being transmitted over the public internet, and not being stored on third-party servers. Current ChatBI platforms on the market mostly focus on database SQL scenarios, offering limited support for local files, typically only providing a basic "upload-preview-fixed chart" mode. This analysis process is relatively rigid and cannot dynamically adjust analysis steps or self-correct errors based on user natural language commands.

[0004] Therefore, it is necessary to design a new approach that can automatically identify file structure, securely perform analysis tasks, and provide a self-correcting end-to-end solution to achieve a reliable and self-improving automated system for structured data analysis that can be operated without programming experience. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for automatic acquisition, analysis and execution of structured data.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for automatic acquisition, analysis, and execution of structured data, comprising: Obtain the structured data file to be processed to obtain the file to be processed; Perform a full-process quality check and standardization process on the file to be processed to obtain standardized data; Based on the standardized data, perform field-level semantic analysis and vectorization processing to construct a searchable knowledge index; The system acquires and parses user natural language requests, and uses vector retrieval and risk assessment to ensure that the analysis intent is executable and safe, thus obtaining a confirmed executable analysis request. The confirmed executable analysis request is transformed into a structured analysis plan containing multiple ordered steps; The structured analysis plan is converted into executable code and run in a constrained environment. Closed-loop correction is used to ensure successful execution in order to obtain the analysis results. The analysis results are used to identify the chart type and output the results, and ECharts rendering parameters that conform to visualization specifications are automatically generated. The analysis results are comprehensively analyzed, including the execution process and various types of results, to generate an analysis summary report in natural language.

[0007] This invention also provides a system for automatic acquisition, analysis, and execution of structured data, characterized in that it includes: The acquisition unit is used to acquire the structured data file to be processed, so as to obtain the file to be processed; The quality inspection unit is used to perform full-process quality inspection and standardization processing on the files to be processed in order to obtain standardized data; The index building unit is used to perform field-level semantic analysis and vectorization processing based on the standardized data to build a searchable knowledge index. The analysis request determination unit is used to acquire and parse the user's natural language request, and ensure that the analysis intent is executable and safe through vector retrieval and risk assessment, so as to obtain the analysis request that is confirmed to be executable; The conversion unit is used to convert the confirmed executable analysis request into a structured analysis plan containing multiple ordered steps; An execution unit is used to convert the structured analysis plan into executable code and run it in a constrained environment, ensuring successful execution through closed-loop correction to obtain the analysis execution results; The rendering unit is used to identify the chart type output of the analysis execution results and automatically generate ECharts rendering parameters that conform to the visualization specifications. The report generation unit is used to comprehensively analyze the analysis execution process and multiple types of results to generate an analysis summary report in natural language.

[0008] The advantages of this invention compared to existing technologies are as follows: This invention ensures data consistency and accuracy by automatically identifying file structures and performing full-process quality checks and standardization. Based on this, a knowledge index built using field-level semantic analysis and vectorization processing supports accurate and efficient natural language request parsing and risk assessment, ensuring the secure executability of analytical intent. Subsequently, the confirmed analysis request is transformed into a structured analysis plan with ordered steps, and code that can run safely in a restricted environment is automatically generated. A closed-loop correction mechanism ensures the smooth execution of the process and the reliability of the results. Finally, the system can intelligently identify the type of analysis results, automatically generate ECharts rendering parameters that conform to visualization specifications, and generate a detailed natural language analysis summary report. This achieves fully automated processing from data acquisition to result presentation, greatly improving data analysis efficiency and reliability.

[0009] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the method for automatically acquiring, analyzing, and executing structured data provided in an embodiment of the present invention. Figure 2 A schematic block diagram of the structured data automatic acquisition, analysis and execution system provided in the embodiments of the present invention; Figure 3 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] Please see Figure 1 , Figure 1 This is a flowchart illustrating the automated structured data acquisition, analysis, and execution method provided in this embodiment of the invention. This method is applied to a server. By automatically acquiring and analyzing files to be processed, it performs full-process quality checks and standardization to ensure data consistency and integrity. Based on standardized data, it constructs a knowledge index using field-level semantic analysis and vectorization, and utilizes natural language processing technology to parse user requests, ensuring the safety and feasibility of the analysis task. Subsequently, the confirmed executable analysis request is transformed into a structured plan of ordered steps and then into executable code to run in a constrained environment, with closed-loop correction ensuring successful execution. Furthermore, the system can automatically identify chart types, generate ECharts rendering parameters to visualize results, and automatically generate a summary report by integrating the analysis process and results. The entire process automates operations from data acquisition and analysis to result presentation, possessing self-correcting capabilities to ensure system reliability and the accuracy of analysis results. This allows even users without programming backgrounds to safely and effectively perform complex data analysis tasks.

[0017] Figure 1 This is a flowchart illustrating the method for automatically acquiring, analyzing, and executing structured data provided in an embodiment of the present invention. For example... Figure 1 As shown, the method includes the following steps S110 to S180.

[0018] S110. Obtain the structured data file to be processed to obtain the file to be processed.

[0019] In this embodiment, the file to be processed refers to a structured data file obtained through a specific method. These files contain raw data that needs to be quality checked, parsed, analyzed, and executed. Obtaining the structured data file to be processed is the first step in the entire automated process, aiming to provide basic input for subsequent data processing. Specifically: Obtaining the structured data files to be processed: This step involves collecting structured data files from various sources, such as local storage, network locations, or cloud storage. These files can be in various formats, such as spreadsheets (.xlsx, .xls), comma-separated values ​​(.csv), etc. The key is to identify and locate the correct file paths and ensure that these files are accessible.

[0020] Obtaining the files to be processed: Once the structured data files are successfully obtained, the next step is to verify that these files meet expectations, meaning they do indeed contain the data required for further processing. This step also includes extracting file-related attribute information, such as file type, name, and access path, for use in subsequent processing. Furthermore, preliminary validity checks may also be performed at this stage to ensure that the obtained files are readable and valid, avoiding unnecessary errors in later processing stages.

[0021] This step lays the foundation for the entire automated system of structured data analysis based on large models, ensuring the quality and applicability of the input data. This supports a series of subsequent operations, including but not limited to data quality checks, knowledge extraction, understanding of analytical intent and execution constraints, analysis plan generation, code generation and execution, results visualization, and analysis summary generation. Through this series of carefully designed steps, the system can automatically identify and process structured data from different sources and formats, providing users with an efficient and reliable analysis platform that can be operated without programming experience.

[0022] S120. Perform full-process quality inspection and standardization processing on the file to be processed to obtain standardized data.

[0023] In this embodiment, standardized data refers to data files obtained after undergoing a series of rigorous quality checks and standardization processes. These steps ensure the quality, consistency, and readability of the structured data, thereby providing high-quality input for subsequent analysis and interpretation based on large models. Specifically, standardized data has the following characteristics: File validity and integrity: By verifying file type consistency, content integrity, and file readability, we ensure that the file format is correct, the content is complete, and it can be read normally by the system.

[0024] Structural consistency: Identifies and processes multi-level header structures, automatically merges levels to generate unified column names, and performs uniqueness and non-nullability checks to ensure the consistency and uniqueness of data field names.

[0025] Field integrity and naming conventions: Clean up empty rows and columns, detect and complete missing fields, and perform unique processing on duplicate fields to ensure that each field has a clear and unique identifier.

[0026] Data content quality: Without changing the main data structure, detect mixed data types, missing values, outliers, and duplicate data, and generate corresponding prompts to improve the accuracy and reliability of data content.

[0027] Data format standardization: Perform type identification, conversion and format standardization processing on date and time fields to unify the data representation and ensure that all date and time fields have a consistent representation in the output stage.

[0028] In one embodiment, step S120 described above may include steps S121 to S126.

[0029] S121. Extract attribute information, including file type, name and path, from the file to be processed; S122. Perform file type consistency, content integrity and readability checks in sequence. If they fail, terminate the process and prompt an exception to obtain data that has passed the validity check. S123. The data that has passed the legality check is identified and processed to form a multi-level header structure. The hierarchy is automatically merged to generate a unified column name and a uniqueness and non-emptiness check is performed to obtain data that has passed the structural consistency check.

[0030] In one embodiment, step S123 described above may include steps S1231 to S1235.

[0031] S1231. Perform a data volume pre-check on the data that has passed the legality verification. S1232. Extract the first three rows as candidate header rows and candidate data rows, and convert them into string sequences; S1233. Perform multi-level header pattern recognition, including: header hierarchy relationship feature judgment, data row difference feature judgment, category row feature judgment, and duplicate classification feature judgment. S1234. When a multi-level header structure is detected, the first row and the second row are merged to generate a unified column name sequence. S1235. Perform column name uniqueness and non-nullability checks on the unified column name sequence. If they pass, remove the first two candidate header rows and reset the index to obtain data that has passed the structural consistency check.

[0032] S124. Clean up empty rows and columns in the data that has passed the structural consistency check, detect and fill in missing fields, and perform unique processing on duplicate fields to ensure naming conventions. S125. Detect mixed data types, missing values, outliers, and duplicate data without changing the main data structure and generate prompts. S126. Perform type identification, conversion, and format standardization processing on the date and time fields to unify the data representation format, obtain standardized data, upload it to the object storage system, and return the file access address to complete data delivery.

[0033] In this embodiment, the structured data file to be processed and its file attribute information are first obtained. The file attribute information includes at least the file type, file name, and file access path. Then, the structured data file undergoes file validity and parsing verification, which includes three levels: First, file type consistency verification, which extracts the actual extension field by parsing the file name and compares it character-wise with the declared type field. If the comparison result is inconsistent, a first exception code is generated and returned; if consistent, a first pass code is generated and the process proceeds to the next step. Second, file content integrity verification, which checks the existence of the data file's storage path by calling the underlying storage interface. If this fails, a second exception code is generated and returned; if successful, the corresponding parsing component is selected based on the declared type field to perform a trial parsing operation. For xlsx or xls type files, the engine detection module is called to obtain the matching table parsing engine type (such as xlrd or openpyxl) and a table file object is constructed. The process involves several steps. First, a test parsing is performed. For other data types like CSV, encoding detection is performed using a byte stream to obtain the target encoding. Then, a test read is performed based on this encoding with a limited number of rows. If an exception is thrown during the test parsing or reading, a third or fourth exception code is generated and returned. If no exception is thrown, a second pass code is generated and the process proceeds to the next step. Finally, file readability is verified. Using the data file's storage path and declared type as input, the corresponding reading component is called to perform a minimal read verification. For XLSX or XLLS types, the table parsing engine is reused to perform file opening and object instantiation verification. For CSV types, a sampling read verification is performed based on the target encoding. If an exception occurs during verification, a fifth exception code is generated and returned. If the verification passes, a third pass code is generated, and the data file is marked as verified to trigger subsequent parsing processes.

[0034] After passing the file validity and parsability checks, a structural consistency check is performed on the structured data. This process first obtains the data table object corresponding to the structured data and performs a data volume pre-check. When the number of rows in the data table object is less than a preset minimum row count threshold (this threshold at least meets the verification requirement of the first two rows as candidate headers and the third row as candidate data rows, and the value range is 3 to 10), it is determined that the conditions for multi-level header recognition are not met, a structural consistency indication message to indicate "no multi-level header detected" is generated, and the original data table object is returned. Based on the successful data volume pre-check, the first, second, and third rows of the data table object are extracted sequentially as candidate header rows and candidate data rows. The candidate rows are then converted into string sequences to eliminate the impact of numerical type differences on header pattern recognition. Subsequently, multi-level header pattern recognition processing is performed on the candidate rows. This recognition processing includes at least four feature constraints: header hierarchy relationship features (statistically calculating the proportion of duplicate values ​​in the first row; when the proportion of duplicate values ​​exceeds the first threshold and the first row contains non-empty units, it is determined that the first row has a superior category merging feature), data row difference features (statistically calculating the proportion of "numerical or character content that can be parsed as numeric" in the third row; when the proportion exceeds the second threshold, it is determined that the third row is a data row), category row features (statistically calculating the proportion of numerical content in the second row; when the proportion is lower than the third threshold, it is determined that the second row is a category row), and duplicate category features (counting the non-empty categories in the first row; when the number of categories with a frequency greater than 1 is not less than a preset threshold, it is determined that there is a duplicate superior category). Under the condition that the multi-level header indication information indicates "it is a multi-level header", the first and second rows are processed to merge the headers to generate the merged column name sequence new_columns. This process concatenates or combines the upper-level categories of the first row with the lower-level fields of the second row to form a unified field description information. Then, column name uniqueness and column name non-nullability checks are performed on new_columns in sequence. If either check fails, a structural consistency exception indication information is generated to indicate "the header merge is unreliable" and the original data table object is returned. If both checks pass, the column names of the data table object are updated to new_columns and the first two candidate header rows are removed from the data area. Specifically, the data area of ​​the data table object from the third row onwards is truncated and the index is reset so that subsequent business parsing is performed with the unified field description information as input. Optionally, structural consistency check prompts are recorded to indicate "multi-level headers were detected and have been automatically merged".In addition, when the structural consistency check step includes multi-level automatic header merging, the system performs a merge rationality check on the candidate column name sequence generated by the header merging. This check includes at least the following: the candidate column name sequence is not empty, the number of non-empty column names in the candidate column names is greater than zero, and the average character length of the candidate column names does not exceed a preset length threshold (the threshold ranges from 20 to 100 characters). If any of these conditions are not met, the merge result is determined to be unreliable and the system will revert to the original column names or skip the merge update. At the same time, a structural consistency exception indication message is generated to identify "unreliable merge" and the original data table object is returned.

[0035] After structural consistency checks, a field integrity check is performed on the structured data to obtain field integrity indication information. This process first cleans up empty rows and columns in the data table object to eliminate interference from completely empty cells on field identification and naming. Specifically, columns and rows containing only null values ​​are deleted (all null values ​​refer to all cells in the column or row containing system missing values, empty strings, or blank characters). Cleanup prompts are recorded when data dimensions change before and after the cleanup. After cleaning up empty rows and columns, missing field detection and completion are performed on the data table object. Specifically, the column name sequence of the data table object is traversed, and each column name is checked for missing values ​​(including whether the column name is a system missing value, an empty string, or a preset missing value). When the i-th column name is detected to be missing, a default column name containing the column number and a fixed prefix is ​​generated as a replacement column name and used to update the i-th column name. Simultaneously, missing field completion prompts are recorded. The fixed prefix is ​​selected from "COL_", "FIELD_", or "column", and the column number increments from 1. Based on the completed field name sequence, duplicate field detection and field uniqueness processing are performed on the data table object. Specifically, an empty set existing_columns is initialized and a uniqueness check is performed on each column name in the field name sequence. If the column name to be checked does not appear in existing_columns, the column name is retained. If it already exists, a new column name with a serial number suffix is ​​generated based on the column name, and the serial number is incremented in a loop until the new column name is no longer in existing_columns to obtain a unique column name. Then, the unique column name is written back to the column name sequence and added to existing_columns to ensure that the column name is globally unique. At the same time, duplicate field processing prompt information is recorded to indicate that "duplicate column names have been automatically renamed".

[0036] After the structured data has undergone field integrity checks, a data content quality check is performed to obtain data quality indication information. This process checks the rationality, consistency, and anomalies of field values ​​without altering the main data structure (i.e., without adding or deleting rows or columns from the data table objects, only adding informational fields or flags). This includes at least mixed data type detection, missing value detection and missing value flag normalization, outlier detection, numeric format standardization detection, and duplicate data detection. Mixed data type detection involves traversing each field of the data table object and counting the data type set corresponding to each field value (including numeric, character, boolean, date / time, and system missing types). When the number of detected data types in a field exceeds a preset threshold (ranging from 2 to 3), the field is determined to be a mixed data type field, and a mixed data type warning message is generated, suggesting manual review. Missing value detection and missing value tag normalization pre-define multiple missing value tagging modes for each field (including at least empty strings, pure space strings, NA, N / A, NULL, NaN, and custom missing value tags). Field values ​​matching any of these modes are uniformly replaced with standard missing value representations (selected from system missing values ​​or empty strings). The percentage of missing values ​​in each field is then calculated, and a corresponding missing value rate alert is generated when the percentage is greater than zero. Outlier detection filters numeric fields from the data table and calculates the outlier range for each numeric field based on the interquartile range method. The outlier range is defined as [Q1 - k × IQR, Q3 + k × IQR], where Q1 and Q3 are the first and third quartiles, respectively, IQR = Q3 - Q1, and k is a preset outlier coefficient (ranging from 1.5 to 3.0). When a field value exceeds the outlier range, it is considered an outlier, and the number of outliers is counted. When the number is greater than zero, an outlier detection alert is generated. The duplicate data detection performs row-level duplicate checks on data table objects to identify completely duplicate data records (complete duplicates mean that two rows are equal in all corresponding field values, and missing values ​​are considered equal). When the number of duplicate records detected is greater than zero, a duplicate data warning message is generated. Users can specify whether to enable deduplication processing to retain only the first occurrence of the record when initiating the check through configuration parameters. The numeric format standardization detection checks whether character fields conform to the characteristics of a pure numeric string (composed only of numbers, decimal points, plus and minus signs, and scientific notation markers). When the proportion of field values ​​that can be successfully converted to numeric type exceeds a preset conversion ratio threshold (ranging from 0.8 to 1.0), the field is automatically converted to a numeric field and a numeric format standardization warning message is generated.

[0037] The method also includes data format standardization processing, particularly date and time type standardization processing. This processing performs date and time feature detection on character fields, randomly selecting a preset number of non-empty field values ​​(ranging from 10 to 100) as sample data. A preset date and time format rule library is used to perform pattern matching on the sample data and count the proportion of field values ​​that conform to the date and time format features. When the proportion is greater than or equal to a preset proportion threshold (ranging from 0.7 to 1.0), the field is determined to be a candidate date and time field and a date and time type conversion is triggered. A secure conversion interface is called to convert the entire field to a date and time type. If the conversion is successful and the field data type changes, a first standardization prompt message is generated. Subsequently, the date and time type fields are uniformly converted to a preset string representation format conforming to the ISO 8601 standard to meet the requirements of subsequent serialization and cross-system transmission. If the conversion fails, a first exception code is generated and the field is skipped. When a date / time field already exists in the data table object, formatting is performed directly to convert it into a preset date / time string format (selected from "YYYY-MM-DD", "YYYY-MM-DD HH:MM:SS" or "YYYY-MM-DDTHH:MM:SSZ"). When formatting is complete and the field value range does not exceed the valid date / time interval, a second standardized prompt message is generated. When formatting fails or the field value exceeds the valid interval, a second exception code is generated and the original field value is retained unchanged.

[0038] Finally, standardized structured data after quality check is generated based on the quality check results of the structured data. After completing structural consistency check, field integrity check, data content quality check, and data format standardization, the system uses the standardized structured data that has passed quality check as the output dataset and performs standardized file generation processing. This process first determines the working file path and completes preprocessing output: when the original data file is a spreadsheet file, preprocessing by merging cells is performed, and the preprocessed working file path is used as the subsequent read / write object; when it is a text table file, the original file path is directly used as the working file path. Then, the output dataset is organized according to the data carrying format: when the working file path corresponds to a spreadsheet file, the data content of each worksheet is read and the standardization process is performed on each worksheet to obtain a set of standardized data tables that correspond one-to-one with the worksheet name and temporarily stored as a key-value mapping structure (the key is the worksheet name, and the value is the corresponding standardized data table). If the processing of a worksheet fails, it is skipped and other worksheets are processed; when it corresponds to a text table file, the file is read based on the detected character encoding to obtain a single standardized data table. The system then generates standardized output files: It constructs the output file name and path (the output file name is obtained by concatenating the base name of the original file name with a preset suffix such as "_clean" and uniformly generating a spreadsheet format). When the output dataset is a set of standardized data tables, a spreadsheet writer is created, and each table is written to a different worksheet before the writer is closed to complete file persistence. When it is a single standardized data table, it is directly written to the file. After the standardized output file is generated, the working files or temporary files generated during processing are deleted to free up storage space. The standardized output file is then uploaded to the preset object storage system. Upon successful upload, the internal access address is obtained, and this address and the standardized output file name are returned as the standardized structured data generation result after quality verification. Simultaneously, locally generated standardized output files are deleted to avoid redundant usage. When the standardized data table set is empty, or the standardized output file generation fails, or the upload fails, the system returns an overall failure result indicating "standardized file generation failed" and terminates result delivery to avoid outputting incomplete or unusable data files downstream.

[0039] S130. Based on the standardized data, perform field-level semantic analysis and vectorization processing to construct a searchable knowledge index.

[0040] In this embodiment, knowledge indexing refers to constructing a knowledge base that supports efficient semantic retrieval by performing field-level semantic analysis and vectorization processing on standardized structured data. This knowledge base not only contains the structural information and content description of the original data, but also captures the semantic features of each field through vector representation, enabling users to retrieve and locate fields, worksheets, or files based on semantic similarity.

[0041] In one embodiment, step S130 described above may include steps S131 to S135.

[0042] S131. Based on the standardized data and attribute information, split the table file according to the worksheet dimension, and convert each worksheet into a two-dimensional sequence data structure.

[0043] In this embodiment, subsequent analysis and processing can be performed on a single worksheet, improving processing efficiency and accuracy.

[0044] S132. Extract the table structure information, field names, positions, and data sets of each worksheet to generate structured description information.

[0045] In this embodiment, table structure information (such as the number of rows and columns) and field details (including field names, positions and their corresponding data sets) are extracted from each worksheet to generate structured description information.

[0046] Provide clear definitions and location identifiers for each field to facilitate subsequent semantic analysis and retrieval.

[0047] S133. Identify the data type of the fields in the structured description information, extract example values ​​and calculate statistical features to generate field-level semantic description text.

[0048] In this embodiment, the data set of each field is type-identified to determine the data type of the field (such as integer, floating-point, boolean, date / time, or string).

[0049] Based on the field type, a certain number of sample values ​​are extracted and returned, which can effectively reflect the actual values ​​of the field.

[0050] The basic statistical characteristics of calculated fields, such as deduplication count, minimum value, maximum value, and average value, are especially important for numeric fields.

[0051] By integrating the above analysis results, a field-level semantic description text is formed, which comprehensively summarizes the semantic features of the field.

[0052] S134. Convert the semantic description text of the fields into high-dimensional semantic vectors in batches for semantic similarity calculation.

[0053] In this embodiment, all field-level semantic description text is input into the vector generation module and converted into high-dimensional semantic vectors in batches. These vectors are used for subsequent semantic similarity calculations.

[0054] Vectorization enables computers to understand and compare semantic relationships between different fields, thereby supporting more intelligent retrieval functions.

[0055] S135. Construct index records containing knowledge identifiers, metadata, and vectors, and write them in batches to the vector retrieval engine.

[0056] In this embodiment, an index record containing knowledge identifiers, metadata (such as file description information, worksheet description information, field name, field type, example value, statistical features and field-level semantic description text) and vector data is constructed for each field, and these records are written to the vector retrieval engine in batches.

[0057] Establish an efficient database of "field semantics-vector-metadata" mapping relationships, enabling users to quickly retrieve the information they need based on semantic similarity.

[0058] Through this series of steps, the system not only achieves deep analysis and understanding of structured data, but also transforms it into easily manageable and searchable knowledge resources through vectorization. This approach greatly enhances data analysis capabilities, enabling users to utilize data assets more flexibly and accurately.

[0059] In this embodiment, a standardized structured data file that has undergone quality inspection is obtained, and the file attribute information corresponding to the structured data file is also obtained. The file attribute information includes at least file type, file name, file access path, and business scenario identification information. The structured data file is parsed according to the file type. When the file is a table file, the structured data is split according to the worksheet dimension to obtain at least one data worksheet. Specifically, the system calls a preset object storage client to download the table file from the file path and load it as a workbook object; it obtains the set of worksheet names contained in the workbook object and generates worksheet description information to characterize the structure of the table file. The worksheet description information includes at least the number of worksheets, the name of each worksheet, and the worksheet index; subsequently, the system performs a sequential traversal of the set of worksheet names: it obtains the corresponding worksheet object based on the current worksheet name and extracts the cell data of the worksheet object. The cell data is obtained by traversing the value iterator of the worksheet object and converted into two-dimensional sequence data. Each row of the two-dimensional sequence data corresponds to a cell row, and each column corresponds to a cell column; the two-dimensional sequence data is used as the structured data content corresponding to the worksheet; the system can optionally encapsulate the two-dimensional sequence data into a payload to be processed and add it to a payload list for subsequent standardization processing, thereby realizing the splitting and parsing of the table file according to the worksheet dimension. The payload list is either a first-in-first-out queue or a key-value mapping structure.

[0060] For each of the data worksheets, extract the table structure information and field information corresponding to the data worksheet. The field information includes at least the field name, the location of the field, and the data set corresponding to the field. Specifically, the system parses the structured data representation corresponding to the current data worksheet to obtain its row and column counts, forming table structure information. This table structure information characterizes the data scale and overall structure of the data worksheet. Further, the system extracts a sequence of field names from the structured data representation. This sequence consists of a set of column names in the structured data representation, and the position of each field is uniquely determined by the column name's index within the set of column names. For each field, the system extracts the corresponding data set from the structured data representation based on its position, characterizing the value distribution of that field in the data worksheet. After extracting the table structure and field information, the system generates corresponding worksheet description information based on the worksheet name, table structure information, and field name sequence. This worksheet description information includes at least the worksheet name, number of rows, number of columns, and a list of field names, providing a summary description of the structured features of the data worksheet and offering structured input for subsequent data quality checks, field semantic understanding, or result summarization.

[0061] Based on the data set corresponding to each field, field semantic analysis is performed on each field. This semantic analysis includes field data type identification, example value extraction, and statistical feature calculation to generate field-level semantic description information. Specifically, for the structured data in each worksheet, its field set is traversed; the corresponding data set for the current field is extracted, and null values ​​are removed to obtain valid field data; when the valid field data is null, the field is skipped and the next field is processed. Field data type identification is performed on the valid field data to obtain a field type identifier. This includes standardizing null values ​​in the field data and inferring the type based on the value format after removing null values; when the field data meets the numeric characteristic, the field type is identified as integer or floating-point; when the field data meets the Boolean characteristic, the field type is identified as Boolean; when the field data meets the date and time parsable characteristic, the field type is identified as date and time; otherwise, the field type is identified as string; when an exception occurs during type inference, a preset default type identifier is output. Based on the field type identifier, sample value extraction processing is performed on the valid field data to obtain a set of field sample values. Specifically, this includes extracting a preset number of sample values ​​from the valid field data and converting the sample values ​​into a serializable data representation. When the field type identifier is date / time type, the sample values ​​are converted to date / time strings and a preset number of sample values ​​are returned. When the field type identifier is non-string type, a preset number of preceding sample values ​​are returned. When the field type identifier is string type, the sample values ​​are truncated and deduplicated to obtain a set of sample values ​​used to represent the field semantics. Based on the field type identifier, statistical feature calculation is performed on the valid field data to obtain field statistical feature information, wherein the field statistical feature information includes at least the deduplication count of the field. When the field type identifier is numeric type, the minimum, maximum, and mean values ​​of the field are further calculated, and the statistical features are output as part of the field-level semantic description information. After obtaining the field type identifier, example value set, and field statistical feature information, the system organizes the field name, field type identifier, and example value set into a field-level semantic description text to represent the field's semantic features. The system then combines the field-level semantic description text with file-level and worksheet-level description information to generate a summary text. Simultaneously, the system constructs a field-level semantic payload containing user identifier, file identifier, file path, worksheet name, field name, field type identifier, example value set, field statistical feature information, and field-level semantic description text, and adds the field-level semantic payload to a pending processing list for subsequent vectorization, retrieval, or intelligent analysis processes.

[0062] Based on the field-level semantic description information, a vectorization representation generation process is performed to obtain field-level semantic vectors. Specifically, this includes inputting the summary text set generated for each field into the vector generation module, and generating a vector set corresponding one-to-one with the summary text set in a batch processing manner. Each vector is used to represent the semantic features of the corresponding field-level semantic description information. Based on the field-level semantic vectors and field-level semantic payloads, a vector index write load is constructed for subsequent semantic retrieval and matching. Specifically, this includes performing alignment matching on the list of field-level semantic payloads and the vector set, and constructing an index write record for each field. The index write record includes at least: a knowledge identifier (knowledgeId), used to aggregate and identify structured knowledge of the same user and the same file; the knowledge identifier is generated from the user identifier and file identifier using a preset unique key generation rule; metadata, used to carry the field-level semantic payload, which includes at least file description information, worksheet description information, field name, field type, example value, statistical features, and field-level semantic description text; vector data, used to carry the field-level semantic vectors; and scenario identification information, including channel identifier, scenario identifier, and index name, used to distinguish vector retrieval spaces of different business domains. The vector index is written to the vector retrieval engine in batches to complete the vectorized storage of semantic knowledge of structured data fields. Specifically, the index write record set is submitted to the batch write interface, which writes the index write records into a preset index library, thereby forming a searchable "field semantics - vector - metadata" mapping relationship in the vector retrieval engine to support subsequent retrieval and location of fields, worksheets or files based on semantic similarity.

[0063] S140. Obtain and parse the user's natural language request, and ensure that the analysis intent is executable and safe through vector retrieval and risk assessment to obtain a confirmed executable analysis request.

[0064] In this embodiment, confirming an executable analysis request means ensuring, through a series of steps, that the natural language analysis request input by the user not only complies with the system's security policy but is also within its capabilities and clearly expresses its intent, thereby ensuring that subsequent data analysis tasks can be executed smoothly and securely.

[0065] In one embodiment, step S140 described above may include steps S141 to S144.

[0066] S141. Obtain the user's natural language analysis request and related session, scenario, and file identification information; S142. Combining the knowledge index and the standardized data, retrieve the semantic knowledge of relevant fields based on the question vector under permission constraints, and construct the data context required for analysis. S143. Based on the context, use a large language model to identify and analyze the intent, and generate a structured recognition result containing action instructions and risk levels to obtain the intent recognition result; S144. Based on the intent recognition result, determine the risk level, generate risk warnings for requests that do not comply with the security policy, determine whether the request exceeds the system capability boundary, provide alternative problem suggestions when it exceeds the boundary, and perform semantic completion on the current request based on historical interaction information to eliminate ambiguous references and semantic omissions in order to obtain a confirmed executable analysis request.

[0067] In this embodiment, a natural language analysis request input by the user is first received, and session identifier information, business scenario identifier information, and available structured data file identifier information associated with the analysis request are obtained. Based on the business scenario identifier information and the available structured data file identifier information, contextual prompt information for intent recognition is constructed. The natural language analysis request and the contextual prompt information are then input into a large language model to perform analysis intent recognition processing and generate analysis intent recognition results. Specifically, the system receives an intent recognition request that includes a user identifier, session identifier, business channel identifier, business scenario identifier, a set of available structured data file identifiers, and a natural language question. In the vector retrieval-driven context construction phase, the system generates a question vector for the natural language question and performs a vector similarity search in a preset vector index library using a set of filtering conditions: business channel identifier, business scenario identifier, user identifier, and available structured data file identifier. This retrieves field semantic knowledge hit results related to the semantics of the natural language question. The hit results include at least file description information, file storage path, worksheet description information, and field-level semantic description information. If no relevant hit results are found, the system returns an exception result indicating "no relevant data found," prompting the user to supplement the question or upload a file. Subsequently, Prompt assembly and structured context injection are performed: the system groups the hit results by file and worksheet dimensions, extracts common information from each group (including file description information, file storage path, and worksheet description information), concatenates the field-level semantic description information within each group according to the field dimension to form field information segments, combines the common information and field information segments, inserts separators between groups, and generates the basic context prompt information base_prompt. During the intent recognition prompt generation phase, the system reads a preset intent recognition system prompt template, injects the basePrompt and the natural language question according to preset placeholder rules, and generates a combined prompt message, combined_prompt. The combined_prompt at least includes available structured data context related to the question, field semantic description information, and the natural language question text to be recognized. Finally, the system calls a large language model to perform intent recognition: the system uses the system role prompt constraint large language model as the intent gatekeeper module, and calls the model with deterministic inference parameters to perform intent recognition inference on the combined_prompt, obtaining the intent recognition result. The intent recognition result includes at least action instruction information (action) and risk level or risk category information, and is returned in a structured data format, which is parsed by the system to obtain a machine-readable intent determination result.

[0068] Based on the analysis intent recognition result, the natural language analysis request is assessed for risk level, which includes at least allowing execution, denying execution, and executing after prompting. If the assessment result does not meet the preset security policy, a corresponding risk warning message or guided question suggestion is generated. When the analysis intent recognition result meets the preset execution conditions, further capability boundary recognition processing is performed based on the natural language analysis request to determine whether the analysis request exceeds the current system's executable capability range. When the analysis request is detected to exceed the system's executable capability range, a capability out-of-bounds warning message is generated, and alternative analysis question suggestions with similar semantics to the analysis request are output to guide the user to adjust the analysis request. When both the analysis intent recognition result and the capability boundary recognition result meet the preset conditions, the natural language analysis request is determined to be an executable analysis request, and a corresponding analysis execution instruction is generated.

[0069] For natural language analysis requests with incomplete semantics or ambiguous referencing, the system acquires historical interaction question information and performs semantic completion processing based on this information and the current analysis request to generate a semantically clear expression for the analysis request. Specifically, this includes receiving a semantic understanding request containing a session identifier, the current natural language analysis request, and a set of historical interaction questions, and using both the historical interaction question set and the current natural language analysis request as semantic completion input conditions. During the historical interaction question information organization phase, the system sequentially organizes the historical interaction question set, numbering the historical interaction questions according to their order of appearance in the session, and concatenating the numbered historical interaction questions into historical context text. This historical context text is used to represent the analysis background and implicit constraints explicitly expressed by the user in the current session. During the semantic completion prompt information construction phase, the system reads a preset semantic completion system prompt template and injects the historical context text and the current natural language analysis request into the system prompt template to generate combined prompt information for semantically explicit recognition. This combined prompt information includes at least historical question context information and the current analysis request text, guiding the model to understand omitted references, ambiguous conditions, or context-dependent expressions within the conversational context. Subsequently, the system calls a large language model to perform semantically explicit recognition: the system inputs the combined prompt information into the large language model to perform semantically explicit recognition processing and generate a semantically explicit recognition result. This result includes at least a determination flag indicating whether the current analysis request is semantically ambiguous, and, when semantic ambiguity is determined, a semantically completed analysis request expression generated based on historical interaction question information. During the result processing and interactive feedback phase, when the semantically explicit recognition result indicates that the current analysis request has no semantic ambiguity, the system treats the current analysis request as a semantically explicit analysis request and continues into the subsequent analysis process. When the semantically explicit recognition result indicates that the current analysis request has semantic ambiguity, the system returns a semantic completion result and uses the semantically completed analysis request as a candidate analysis request to prompt the user for confirmation or directly for use in the subsequent analysis process, thereby achieving a semantic closed loop for natural language analysis requests in multi-round interaction scenarios. Finally, the semantically completed analysis request is used as input for subsequent decomposition and execution processing of structured data analysis tasks based on large models.

[0070] According to a fourth aspect of the embodiments of this application, a method for generating structured data analysis plans based on a large model is provided. This method is executed after completing structured data quality checks, structured data knowledge extraction, and understanding and constraining the analysis request intent. The method first obtains a natural language analysis request input by a user, and then obtains session identifier information, business scenario identifier information, and target structured data file identifier information bound to the request. Specifically, it includes receiving a plan generation request, which carries at least a user identifier, session identifier, business channel identifier, business scenario identifier, a set of target structured data file identifiers, and the natural language analysis request text. The session identifier is used to uniquely identify the context lifecycle of the current interactive session; the business channel identifier and business scenario identifier jointly determine the current business domain and the corresponding strategy configuration space; and the set of target structured data file identifiers is used to delineate the data boundaries that are allowed for retrieval and analysis in subsequent steps. Based on the natural language processing request, a question vector is generated. This vector is then combined with a set of user identifiers, business channel identifiers, business scenario identifiers, and target structured data file identifiers to construct retrieval constraints. Specifically, the vector generation module is invoked to vectorize the natural language processing request text to obtain the question vector. A vector retrieval payload is constructed, which includes at least the index name, question vector, user identifier, business channel identifier, business scenario identifier, and target structured data file identifier set. This payload is used to perform semantic retrieval with permission filtering in the vector index library. Based on the vector retrieval payload, a vector similarity search is performed in the vector index library to obtain structured data context information related to the semantics of the natural language processing request. This context information includes at least file-level description information, file storage path information, worksheet-level description information, and field-level semantic description information. If the search result is empty, an exception result indicating "no relevant data found" is returned to prompt the user to check the question description or upload the target file.

[0071] Based on the user request, data file identification information is obtained, and preset basic prompt information is read. This basic prompt information includes at least task planning specifications, preset output structure constraints, and a set of available operation types. Based on the natural language analysis request and the basic prompt information, combined prompt information for plan generation is constructed. This combined prompt information is input into a large language model to generate a structured data analysis plan. The structured data analysis plan includes at least several sequentially arranged analysis steps, and each analysis step satisfies preset data flow constraints. Finally, the structured data analysis plan is stored in association and output for the controllable execution of subsequent structured data analysis tasks.

[0072] Through these four steps, the system not only understands the user's specific needs but also ensures that these needs are both safe and feasible during execution, thus effectively supporting structured data analysis processes based on large models. This approach significantly improves the flexibility and security of the data analysis process while also enhancing the user experience.

[0073] S150. The confirmed executable analysis request is transformed into a structured analysis plan containing multiple ordered steps.

[0074] In this embodiment, a structured analysis plan refers to transforming a user-approved, executable natural language analysis request into a series of ordered and logically clear data processing steps. These steps follow specific task planning specifications and ensure that the data flow constraints between each step are satisfied, thereby supporting subsequent automated code generation and execution.

[0075] In one embodiment, step S150 described above may include steps S151 to S155.

[0076] S151. Convert the confirmed executable analysis request into a question vector and construct retrieval constraints with permission filtering. S152. Perform semantic retrieval in the vector index library to obtain the structured data context related to the confirmed executable analysis request; S153. Read basic prompt information including task planning specifications, retrieval constraints, and available operation types; S154. Integrate the confirmed executable analysis request, structured data context and basic prompts, construct combined prompts for plan generation, and call the large language model to generate a multi-step structured data analysis plan that meets data flow constraints. S155. Store the structured analysis plan.

[0077] Through the five steps outlined above, the system transforms users' initial requirements into specific, actionable, structured analysis plans. This process not only enhances the understanding and expression of user intent but also lays a solid foundation for subsequent automated code generation and secure execution. Furthermore, this systematic approach helps improve the efficiency and accuracy of data analysis while ensuring data security.

[0078] S160. The structured analysis plan is converted into executable code and run in a restricted environment. Successful execution is ensured through closed-loop correction to obtain the analysis execution results.

[0079] In this embodiment, the analysis execution result refers to the unified format result object that meets the expected format and requirements after the structured analysis plan is converted into executable code and run in a restricted environment, and the execution is ensured by closed-loop correction. The result includes, but is not limited to, data processing results and visualization chart parameters.

[0080] In one embodiment, step S160 described above may include steps S161 to S163.

[0081] S161. Construct prompt words based on the structured analysis plan, and call the large language model to generate data analysis target code; S162. Execute the generated target code in a controlled environment with restricted resource access permissions; S163. The output results of successful execution are encapsulated in a structured manner to generate a result object in a unified format. When execution fails, error information is extracted iteratively, the code is optimized, and execution is repeated until success or the termination condition is met, so as to obtain the analysis results.

[0082] Through these three steps, the system not only transforms the structured analysis plan into executable code, but also ensures that the code runs efficiently in a secure and controllable environment. It also provides effective solutions for execution failures, thus guaranteeing the smooth completion of the entire analysis process and the accuracy of the results. Furthermore, the special handling of chart types enhances the intuitiveness and ease of understanding of the results presentation.

[0083] In this embodiment, a plan execution request is received. This request includes at least a session identifier, a user identifier, a scenario identifier, and a structured analysis plan generated upstream, along with its corresponding step information. Based on the structured analysis plan and its corresponding step information, code generation prompts are constructed and input into a large language model to generate target code. After generating the target code, a code execution module is invoked to execute it. During execution, the code runs in a restricted execution environment, which limits file access permissions, network access permissions, and execution resource quotas. When code execution completes and a success response is returned, the execution output is structured to generate a unified result object.

[0084] When the target code fails to execute, error information is obtained and a code optimization request is constructed. This request is then input into the large language model to generate optimized code, which is then executed again to form a closed-loop correction mechanism. Specifically, when the execution result of the generated code is a failure, the system extracts the error information from the failure result and encapsulates the error information and the currently executed code into a code optimization request. The code optimization request includes at least the error type, error stack, and original code fragment. Based on the code optimization request, the system calls the code generation module to generate optimized code and executes it immediately to obtain a new execution result. If the new execution result is a success, the execution result is output and the closed-loop correction process is terminated. If it is still a failure, new error information is extracted, the optimized code is updated to the currently executed code, and the new execution result is updated to the current execution response. Then, a code optimization request is constructed again with the latest error information, and the next round of correction processing begins.

[0085] The closed-loop correction mechanism iterates continuously until preset execution success conditions are met. These preset success conditions include: a success status code, no exceptions thrown, and an output format that conforms to the expected schema. Through a closed-loop process of "generation—execution—feedback—regeneration," the generated code achieves automatic convergence and stable execution. The closed-loop correction mechanism is repeatedly executed until preset termination conditions are met, ultimately outputting the planned execution result to achieve a traceable and reproducible execution loop.

[0086] S170. Identify the chart type of the analysis results and automatically generate ECharts rendering parameters that conform to visualization specifications.

[0087] In one embodiment, step S170 described above may include steps S171 to S172.

[0088] S171. Identify the chart type in the analysis execution result and trigger the chart rendering process.

[0089] In this step, the system first performs type identification on the structured output results obtained after executing the list of steps. Once it detects that the output type of a certain step is a chart type (such as the operation type being marked as CHART or the output result object being identified as chart), it automatically triggers the chart rendering process. This process ensures that the corresponding chart rendering parameters are only generated when the output content needs to be displayed graphically.

[0090] S172. Convert the dataset required for the chart into a two-dimensional data table in the form of a nested list. Based on the data characteristics and user questions, call the model to select the chart type, call the large language model to generate ECharts plotting parameters including axes, series and styles, extract the plotting parameters in JSON format and fill them back into the output results.

[0091] This step involves several key components: Dataset Transformation: The system retrieves the corresponding structured dataset based on the input data identifiers in the chart steps and converts this data into a nested list format data table. If the original data is a data frame, it needs to be converted into a two-dimensional array; if it is already a two-dimensional array, it can be used directly.

[0092] Chart type selection: Based on the user's question and the transformed data table, the system calls the chart type selection module to determine the most suitable chart type (chart_type). This module constructs prompts using a preset set of chart types and uses a large language model to return the most suitable chart type identifier, such as line chart, bar chart, etc.

[0093] ECharts Parameter Extraction: After obtaining the target chart type, the system constructs a prompt message for extracting plotting parameters and sends the data table, user question, and target chart type as input to the large language model to generate plotting parameters that conform to the ECharts standard. These parameters include at least xAxis, yAxis, and series, and may also include other elements such as title and legend.

[0094] Result backfilling: Finally, the system extracts the plotting parameters in JSON format from the content returned by the large language model and writes them back into the output of the chart step, ensuring that the output of each chart step corresponds to its position in the step list, thus realizing the automated process of generating chart rendering parameters from the execution result.

[0095] These two steps work together to achieve a complete process from identifying chart requirements in the analysis results to automatically generating the corresponding ECharts rendering parameters.

[0096] After executing the list of steps and obtaining structured step output results, the system performs type identification on the step output results. When it detects that the output type corresponding to a certain execution step is a chart type, it triggers the chart rendering parameter generation process. The chart type at least corresponds to the operation type of the step object being CHART or the type identifier of the output result object being chart. During the chart data encapsulation stage, the system obtains the corresponding structured dataset based on the input data identifier of the chart steps and uniformly converts the structured dataset into a nested list format data table to meet the input requirements of the chart parameter extraction module. When the structured dataset is a data frame object, it is converted into a two-dimensional array; if the structured dataset is already a two-dimensional array, it is directly reused. Subsequently, the system selects the chart type based on the user question and the data table, calling the chart type selection module to obtain the target chart type `chart_type`. The chart type selection module constructs prompt information through a preset set of graphic types and calls a large language model to return the most suitable graphic type identifier. The set of graphic types includes at least line charts, bar charts, scatter plots, pie charts, and box plots. After obtaining the target chart type, the system enters the ECharts parameter extraction stage. It constructs prompts for extracting plotting parameters and sends the data table, user question, and target chart type as input to the large language model to generate ECharts plotting parameters conforming to preset visualization specifications. These plotting parameters include at least xAxis, yAxis, and series, and optionally include title, legend, grid, axis titles, and label layout strategies. All data used in these plotting parameters comes directly from the data table. Finally, the system extracts and backfills the results. It performs structured extraction of the content returned by the large language model, obtains the JSON-formatted plotting parameters, and writes these parameters as chart outputs into the output of the chart step. This ensures a one-to-one correspondence between the chart step output and the step list, thereby achieving automatic generation from execution output to chart rendering parameters.

[0097] S180. The analysis execution results are comprehensively analyzed, including the execution process and multiple types of results, to generate an analysis summary report in natural language.

[0098] In this embodiment, the analysis summary report refers to the process of data analysis task execution and its various results being transformed into a concise and comprehensive natural language description through structured summarization and large language model processing, providing users with a clear task summary.

[0099] In one embodiment, step S180 described above may include steps S181 to S182.

[0100] S181. Obtain the list of analysis steps corresponding to the analysis execution results and the corresponding step-level execution result data.

[0101] In this step, the system first obtains relevant information about the completed data analysis task. This includes: Analysis Steps List: A series of predetermined analysis steps output by the analysis plan generation module.

[0102] Step-level execution results: These are the execution results that correspond one-to-one with the list of analysis steps above. These results may include one or more of the following: data results, document results, or chart results. The execution result of each step reflects the output of that step in actual operation.

[0103] S182. Construct structured summary prompts in the order of steps, trim the content of multiple types of results to form a summary input, call the large language model to generate a natural language analysis summary of the execution process and results, and output it.

[0104] This step is further divided into several key stages: Constructing a structured summary prompt: Based on the list of analysis steps and their corresponding step-level execution results, a structured step summary prompt is generated in the order in which the steps occur. This prompt appropriately extracts and trims the content of different types of execution results (such as data, files, and charts) to ensure that the summary is both comprehensive and concise.

[0105] The process involves using a large language model to generate a summary: The natural language analysis request and the previously constructed summary prompts are passed as input to the large language model. The model uses this input to generate a natural language analysis summary of the entire analysis task's execution process and results. This summary not only describes the purpose and methods of each step but also summarizes the final findings or conclusions.

[0106] Output Analysis Summary: Finally, the system outputs the natural language analysis summary generated by the large language model as the final result, providing users with a clear and easy-to-understand task summary report.

[0107] These two steps work together to complete the entire process from collecting relevant information for the analysis task to generating the final natural language analysis summary report. This approach not only improves the efficiency of information processing but also helps users understand complex analysis tasks and their results more intuitively.

[0108] This step is triggered after the completion of the data analysis task based on the natural language analysis request, and is used to summarize the execution process and results of the analysis task. The method first obtains the analysis steps and execution results, specifically including obtaining the list of analysis steps output by the analysis plan generation module, and the step-level execution results returned by the script execution module that correspond one-to-one with the list of analysis steps. The step-level execution results include at least one of data results, file results, or chart results. Then, summary prompts are constructed, specifically including generating structured step summary prompts based on the list of analysis steps and the corresponding step-level execution results, in the order of the steps, and extracting and trimming the result content according to different execution result types to form prompt text for generating the analysis summary. Finally, a natural language analysis summary is generated and output, specifically including inputting the natural language analysis request and the summary prompts into a large language model, which generates a natural language analysis summary of the analysis task execution process and results, and outputting the generated analysis summary as the final result.

[0109] The core of this embodiment lies in dividing the entire data analysis process into two main parts from top to bottom: a conversational entry point and seven independent but closely connected sub-modules. The conversational entry point is primarily responsible for receiving users' natural language requests and presenting the final results; while the seven sub-modules are responsible for tasks such as quality checks, knowledge extraction, intent understanding, plan generation, code generation and restricted execution, visualization processing, and finally, summary output, achieving fully automated operation from data collection to result display. This method not only significantly lowers the technical threshold for users to perform structured data analysis, achieving the goal of "zero modeling, zero programming," but also effectively improves the accuracy and security of data processing through closed-loop design between each stage, ensuring the reproducibility and auditability of the entire process. Furthermore, this solution can compress the originally time-consuming data analysis process to within minutes, significantly improving work efficiency (by up to 90%), while ensuring high system security and scalability, possessing extremely high platform deployment value. This patent specifically addresses the problems in existing technologies, such as high barriers to data analysis, susceptibility to errors during execution, lack of effective closed-loop correction mechanisms, and insufficient data security. It proposes solutions to provide users with a more efficient, secure, and easy-to-use structured data analysis method. The specific implementation process includes, but is not limited to, data quality verification, knowledge extraction, understanding the analytical intent and its execution constraints, analysis plan formulation, code generation and controlled execution, result visualization, and final analysis summary generation, collectively forming a complete automated closed-loop process for structured data analysis.

[0110] The aforementioned method for automatically acquiring, analyzing, and executing structured data ensures data consistency and accuracy by automatically identifying file structures and performing full-process quality checks and standardization. Based on this, a knowledge index built using field-level semantic analysis and vectorization supports precise and efficient natural language request parsing and risk assessment, guaranteeing the secure and executable nature of the analysis intent. Subsequently, confirmed analysis requests are transformed into structured analysis plans with ordered steps, and code that can run safely in constrained environments is automatically generated. A closed-loop correction mechanism ensures smooth execution and reliable results. Finally, the system can intelligently identify the type of analysis results, automatically generate ECharts rendering parameters conforming to visualization specifications, and generate a detailed natural language analysis summary report. This achieves fully automated processing from data acquisition to result presentation, greatly improving data analysis efficiency and reliability.

[0111] Figure 2 This is a schematic block diagram of a structured data automatic acquisition, analysis, and execution system 300 provided in an embodiment of the present invention. Figure 2 As shown, corresponding to the above-described method for automatically acquiring, analyzing, and executing structured data, the present invention also provides a system 300 for automatically acquiring, analyzing, and executing structured data. This system 300 includes a unit for executing the above-described method for automatically acquiring, analyzing, and executing structured data, and can be configured in a server. Specifically, please refer to... Figure 2 The structured data automatic acquisition, analysis and execution system 300 includes an acquisition unit 301, a quality inspection unit 302, an index construction unit 303, an analysis request determination unit 304, a transformation unit 305, an execution unit 306, a rendering unit 307, and a report generation unit 308.

[0112] The system comprises the following components: an acquisition unit 301, which acquires a structured data file to be processed; a quality inspection unit 302, which performs a full-process quality inspection and standardization on the file to be processed to obtain standardized data; an index building unit 303, which performs field-level semantic analysis and vectorization processing based on the standardized data to build a searchable knowledge index; an analysis request determination unit 304, which acquires and parses user natural language requests, and ensures that the analysis intent is executable and safe through vector retrieval and risk assessment to obtain a confirmed executable analysis request; a transformation unit 305, which transforms the confirmed executable analysis request into a structured analysis plan containing multiple ordered steps; an execution unit 306, which converts the structured analysis plan into executable code and runs it in a restricted environment, ensuring successful execution through closed-loop correction to obtain analysis execution results; a rendering unit 307, which identifies the chart type of the analysis execution results and automatically generates ECharts rendering parameters that conform to visualization specifications; and a report generation unit 308, which comprehensively analyzes the analysis execution process and multiple types of results to generate an analysis summary report in natural language.

[0113] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned automatic structured data acquisition, analysis and execution system 300 and its various units can be referred to the corresponding descriptions in the foregoing method embodiments. For the sake of convenience and brevity, these details will not be repeated here.

[0114] The aforementioned structured data automatic acquisition, analysis, and execution system 300 can be implemented as a computer program, which can perform tasks such as... Figure 3 It runs on the computer device shown.

[0115] Please see Figure 3 , Figure 3 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0116] See Figure 3 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0117] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a method for automatically acquiring, analyzing, and executing structured data.

[0118] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0119] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a method for automatic acquisition, analysis and execution of structured data.

[0120] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0121] The processor 502 is used to run the computer program 5032 stored in the memory to implement all steps of the structured data automatic acquisition, analysis and execution method.

[0122] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0123] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0124] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all steps of the structured data automatic acquisition, analysis, and execution method.

[0125] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0126] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0127] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0128] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the system of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0129] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0130] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for automatically acquiring, analyzing, and executing structured data, characterized in that, include: Obtain the structured data file to be processed to obtain the file to be processed; Perform a full-process quality check and standardization process on the file to be processed to obtain standardized data; Based on the standardized data, perform field-level semantic analysis and vectorization processing to construct a searchable knowledge index; The system acquires and parses user natural language requests, and uses vector retrieval and risk assessment to ensure that the analysis intent is executable and safe, thus obtaining a confirmed executable analysis request. The confirmed executable analysis request is transformed into a structured analysis plan containing multiple ordered steps; The structured analysis plan is converted into executable code and run in a constrained environment. Closed-loop correction is used to ensure successful execution in order to obtain the analysis results. The analysis results are used to identify the chart type and output the results, and ECharts rendering parameters that conform to visualization specifications are automatically generated. The analysis results are comprehensively analyzed, including the execution process and various types of results, to generate an analysis summary report in natural language.

2. The method for automatic acquisition, analysis, and execution of structured data according to claim 1, characterized in that, The process of performing full-process quality checks and standardization on the file to be processed to obtain standardized data includes: Extract attribute information, including file type, name, and path, from the file to be processed; The process sequentially performs file type consistency, content integrity, and readability checks. If any check fails, the process terminates and an exception is displayed to obtain data that has passed the validity checks. The data that has passed the legality check is identified and processed to form a multi-level header structure. The hierarchy is automatically merged to generate a unified column name and a uniqueness and non-nullability check is performed to obtain data that has passed the structural consistency check. The data that has passed the structural consistency check is cleaned of empty rows and columns, missing fields are detected and filled, and duplicate fields are made unique to ensure naming conventions. Detect mixed data types, missing values, outliers, and duplicate data, and generate alerts without changing the main data structure; The date and time fields are processed for type identification, conversion, and format standardization to unify the data representation, resulting in standardized data that is then uploaded to the object storage system. The system returns the file access address to complete the data delivery.

3. The method for automatic acquisition, analysis, and execution of structured data according to claim 2, characterized in that, The process of identifying and processing the multi-level header structure of the data that has passed the validity check, automatically merging the levels to generate unified column names, and performing uniqueness and non-nullability checks to obtain data that has passed the structural consistency check includes: Perform a data volume pre-check on the data that has passed the validity verification; Extract the first three rows as candidate header rows and candidate data rows, and convert them into string sequences; Perform multi-level header pattern recognition, including: header hierarchy relationship feature judgment, data row difference feature judgment, category row feature judgment, and duplicate classification feature judgment; When a multi-level header structure is detected, the first and second rows are merged to generate a unified column name sequence. Perform column name uniqueness and non-nullability checks on the unified column name sequence. If they pass, remove the first two candidate header rows and reset the index to obtain data that has passed the structural consistency check.

4. The method for automatic acquisition, analysis, and execution of structured data according to claim 1, characterized in that, The process of performing field-level semantic analysis and vectorization processing on the standardized data to construct a searchable knowledge index includes: Based on the standardized data and attribute information, the table files are split according to the worksheet dimension, and each worksheet is converted into a two-dimensional sequence data structure. Extract the table structure information, field names, locations, and data sets of each worksheet to generate structured description information; The structured description information is used to identify the data type of the fields, extract example values ​​and calculate statistical features to generate field-level semantic description text; Batch convert field semantic description text into high-dimensional semantic vectors for semantic similarity calculation; Build index records containing knowledge identifiers, metadata, and vectors, and write them in batches to the vector retrieval engine.

5. The method for automatic acquisition, analysis, and execution of structured data according to claim 1, characterized in that, The process of acquiring and parsing the user's natural language request, and ensuring the executable and secure nature of the analysis intent through vector retrieval and risk assessment to obtain a confirmed executable analysis request, includes: Obtain user natural language analysis requests and related session, scenario, and file identification information; By combining the knowledge index and the standardized data, and based on the question vector, relevant field semantic knowledge is retrieved under permission constraints to construct the data context required for analysis. Based on the aforementioned context, a large language model is used to identify and analyze the intent, generating a structured recognition result that includes action instructions and risk levels, in order to obtain the intent recognition result; Based on the intent recognition results, risk levels are determined, risk warnings are generated for requests that do not comply with security policies, it is determined whether the request exceeds the system's capability boundaries, and alternative problem suggestions are provided when the boundaries are exceeded. Based on historical interaction information, semantic completion is performed on the current request to eliminate ambiguous references and semantic omissions, so as to obtain a confirmed executable analysis request.

6. The method for automatic acquisition, analysis, and execution of structured data according to claim 1, characterized in that, The step of converting the confirmed executable analysis request into a structured analysis plan containing multiple ordered steps includes: The confirmed executable analysis request is converted into a question vector, and search constraints with permission filtering are constructed. Perform semantic retrieval in the vector index library to obtain the structured data context related to the confirmed executable analysis request; Read basic prompts including task planning specifications, search constraints, and available operation types; By integrating executable analysis requests, structured data context, and basic prompts, combined prompts are constructed for plan generation, and a large language model is invoked to generate a multi-step structured data analysis plan that meets data flow constraints. Store the structured analysis plan.

7. The method for automatic acquisition, analysis, and execution of structured data according to claim 1, characterized in that, The process of converting the structured analysis plan into executable code and running it in a constrained environment, ensuring successful execution through closed-loop correction to obtain the analysis results, includes: Based on the structured analysis plan, prompt words are constructed, and a large language model is called to generate target code for data analysis. Execute the generated target code in a controlled environment with restricted resource access permissions; The output of successful executions is encapsulated in a structured manner to generate a result object in a unified format. When execution fails, error information is extracted iteratively, the code is optimized, and execution is repeated until success or the termination condition is met, so as to obtain the analysis results.

8. The method for automatic acquisition, analysis and execution of structured data according to claim 1, characterized in that, The step of identifying the chart type output of the analysis results and automatically generating ECharts rendering parameters that conform to visualization specifications includes: The step of identifying the chart type in the analysis execution result triggers the chart rendering process; The dataset required for charts is uniformly converted into a two-dimensional data table in the form of nested lists. Based on data characteristics and user questions, the model is called to select the chart type. The large language model is called to generate ECharts plotting parameters containing axes, series and styles. The plotting parameters in JSON format are extracted and backfilled into the output results.

9. The method for automatic acquisition, analysis, and execution of structured data according to claim 1, characterized in that, The analysis and execution results are comprehensively analyzed, and the execution process and multiple types of results are combined to generate an analysis summary report in natural language, including: Obtain the list of analysis steps corresponding to the analysis execution results and their corresponding step-level execution result data; The system constructs structured summary prompts step by step, trims various types of result content to form a summary input, and calls a large language model to generate a natural language analysis summary of the execution process and results, and outputs it.

10. A structured data automatic acquisition, analysis, and execution system, characterized in that, include: The acquisition unit is used to acquire the structured data file to be processed, so as to obtain the file to be processed; The quality inspection unit is used to perform full-process quality inspection and standardization processing on the files to be processed in order to obtain standardized data; The index building unit is used to perform field-level semantic analysis and vectorization processing based on the standardized data to build a searchable knowledge index. The analysis request determination unit is used to acquire and parse the user's natural language request, and ensure that the analysis intent is executable and safe through vector retrieval and risk assessment, so as to obtain an analysis request that is confirmed to be executable. The conversion unit is used to convert the confirmed executable analysis request into a structured analysis plan containing multiple ordered steps; An execution unit is used to convert the structured analysis plan into executable code and run it in a constrained environment, ensuring successful execution through closed-loop correction to obtain the analysis execution results; The rendering unit is used to identify the chart type output of the analysis execution results and automatically generate ECharts rendering parameters that conform to the visualization specifications. The report generation unit is used to comprehensively analyze the analysis execution process and multiple types of results to generate an analysis summary report in natural language.

Images