A method and apparatus for enterprise data analysis

Abstract

The application relates to an enterprise data analysis method and device, and relates to the technical field of data governance, and comprises the following steps: fine-tuning a large language model for a Text2SQL task through a low-rank adaptation technology, constructing a structured knowledge base containing an enterprise data dictionary, a field calling relationship and a national standard, and converting unstructured industry specifications into a structured data standard knowledge base, and realizing knowledge vector storage by using an M3E-base vector model; sampling representative samples based on to-be-arranged data fields and retrieving related rules, integrating the representative samples into thinking chain prompt words to input a fine-tuning model to generate a labeled classification result, outputting a data arrangement result after multi-model evaluation and verification; generating a regular expression to screen non-standard data by retrieving data standards, sampling to generate SQL conversion code for iterative cleaning until reaching the standard, and outputting standardized data; and finally performing index optimization and abnormality detection based on the fine-tuning model, receiving a natural language query to generate SQL, and generating a visual report.

Description

Technical Field

[0001] This application relates to the field of data governance technology, and in particular to a method and apparatus for enterprise data analysis. Background Technology

[0002] In the digital economy era, data has become a core production factor for enterprises, growing exponentially in scale and becoming increasingly complex. Effective data analysis is a key support for enterprises to achieve refined operations, risk management, and strategic decision-making. For example, customer behavior data analysis can optimize product recommendations, and production data mining can improve supply chain efficiency. However, enterprise data generally suffers from problems such as data silos, inconsistent quality, and low standardization. Traditional manual data governance models are no longer sufficient to address these issues. Data processing and standardization account for 60%-80% of the entire data analysis process, severely restricting the release of data value.

[0003] Currently, various technical solutions exist in the field of enterprise data analytics. Typical existing technologies include a hybrid model of traditional rule engines and semi-automated tools: Data preparation stage: Relying on data engineers to manually write cleaning rules, or using ETL tools (such as Informatica and Talend) for batch processing, basic cleaning such as deleting duplicate values ​​and filling missing values ​​is achieved through preset rules; Data standardization stage: Based on industry-standard templates or internal enterprise data dictionaries, field format constraints are manually defined, and format conversion is performed using script tools; Data analysis stage: BI tools are used to connect to the standardized data warehouse, generating visual reports through drag-and-drop operations, or customized analysis is achieved by data analysts writing SQL queries.

[0004] Regarding the aforementioned technologies, traditional machine learning models require a large number of labeled samples, have weak generalization ability, and do not integrate data preparation and standardization processes when applying large language models for single tasks. This results in data preparation relying on manual labor, which greatly increases the labor costs of data analysis for traditional enterprises.

[0005] Based on this, this application provides a method and apparatus for enterprise data analysis. Summary of the Invention

[0006] To address the issues of traditional machine learning models requiring a large number of labeled samples, having weak generalization ability, and failing to integrate data preparation and standardization processes in the application of large language models for single tasks, which leads to reliance on manual data preparation and significantly increases the labor costs of traditional enterprise data analysis, this application provides an enterprise data analysis method and apparatus.

[0007] Firstly, this application provides a method for enterprise data analysis, which adopts the following technical solution: including: We employ low-rank adaptation techniques to fine-tune a large language model for Text2SQL tasks, constructing a structured knowledge base that includes an enterprise data dictionary, field call relationships, and national standards. The structured knowledge base is then stored in vector form using the M3E-base vector model. Based on the data fields to be processed, the data query interface is called, and a greedy algorithm is used to sample representative samples from the structured knowledge base, and to retrieve the most relevant rules in the structured knowledge base. The field attribution relationship, calling relationship, sample data and retrieval rules are linearly integrated into the mind chain prompt words, which are input into the fine-tuned large language model to generate preliminary labeling and classification results. The preliminary labeling and classification results are verified through a multi-model evaluation enhancement mechanism, and the data processing results are output. Unstructured industry standard documents are converted into a structured format, encoded using the M3E-base vector model, and stored in a vector database to form a data standard knowledge base. Based on the fields to be standardized in the data processing results, several data standards with the highest relevance in the data standard knowledge base are retrieved, and a large language model is guided to generate regular expressions corresponding to the fields to be standardized. Non-standard data rows are then filtered using regular expressions to obtain a non-standard dataset. Multiple samples are randomly sampled from the non-standard dataset, a large language model is constructed to input thought chain prompt words, SQL conversion code is generated to perform data standardization, non-standard data rows are iteratively filtered until the non-standard data rate does not exceed a preset threshold, and standardized data is output. Based on the fine-tuned model's SQL generation capabilities and the standardized data, the system performs database index optimization and anomaly detection, receives user natural language queries, generates SQL query statements and visualization scripts, and outputs visualization reports.

[0008] Preferably, the step of fine-tuning the large language model using low-rank adaptation technology for Text2SQL tasks, constructing a structured knowledge base containing enterprise data dictionaries, field call relationships, and national standards, and storing the structured knowledge base in vector form using the M3E-base vector model includes: Build a fine-tuning sample set by collecting the Chinese CSpider dataset and local enterprise data; A pre-trained large language model was selected as the base model and all pre-training parameters were frozen. Dimensionality reduction and dimension increase matrices were inserted into the QKV attention unit of the Transformer layer and fine-tuned. Samples were input into the model in batches and concatenated into a query format of question and objective. The cross-entropy loss function was adopted. The pre-training parameters were fixed and only the matrix dimensionality reduction and dimension increase matrices were updated. Iterative training was carried out until the loss on the validation set no longer decreased after a certain number of consecutive preset periods. The model was then fused and saved. The dimensionality reduction matrix and the dimensionality increase matrix are fused with the pre-trained parameters, and the output layer weights are updated to the sum of the products of the pre-trained weights and the dimensionality reduction matrix and the dimensionality increase matrix, thus saving the fine-tuned model adapted to the enterprise task. Integrating knowledge sources: Extracting an enterprise data dictionary containing field names, data types, corresponding database tables, and business descriptions from database metadata; mining field call relationships of four-tuples from IT system operation logs; The knowledge source is processed in a structured manner to form entries in a unified format, thus obtaining structured knowledge entries; The M3E-base Chinese vector model is used to convert structured knowledge entries into text strings and generate high-dimensional dense vectors to capture the semantic features of structured knowledge entries. The knowledge entry text and corresponding vectors are stored in a vector database to establish a mapping relationship between knowledge text and vector index.

[0009] Preferably, the step of calling a data query interface based on the data fields to be processed, using a greedy algorithm to sample representative samples from the structured knowledge base, and retrieving several relevant rules with the highest relevance from the structured knowledge base includes: Based on the basic metadata information of the data fields to be processed, the business attribute descriptions of the fields are obtained by parsing the structured metadata of the basic metadata information, and a set of field metadata is generated. Based on the aforementioned field metadata set, a data query interface is invoked to connect to the enterprise database, a structured query statement is executed, and non-empty data records of the fields to be processed are filtered out; duplicate values ​​and invalid format data are removed from the non-empty data records to form an original sample set containing the actual values ​​of the fields; Based on the field association information of the original sample set and the structured knowledge base, a greedy algorithm is used to sample representative samples to generate a representative sample set. Based on the aforementioned field metadata set and representative sample set, a retrieval query text is constructed. The retrieval query text is input into the M3E-base vector model to generate a query vector, and semantic similarity retrieval is performed in the vector database of the structured knowledge base. The cosine similarity between the query vector and each rule vector in the knowledge base is calculated, and the rules are sorted from high to low according to the cosine similarity. The top-ranked rules are selected as the set of relevant rules with the highest relevance.

[0010] Preferably, the step of generating a representative sample set by sampling representative samples using a greedy algorithm based on the field association information of the original sample set and the structured knowledge base includes: The field data types include numeric fields and string fields. For numeric fields, the original sample set is standardized, the Euclidean distance similarity matrix between samples is calculated, and samples are selected iteratively through a greedy algorithm: first, an initial sample is randomly selected, then the farthest distance between the remaining samples and the existing sample set is calculated in a loop, and the sample with the farthest distance is selected to be added to the representative sample set until the number of samples reaches a preset threshold. For string-type fields, the string values ​​of the original sample set are converted into text vectors through a word vector model. The cosine similarity matrix and Jaccard similarity matrix between samples are calculated, and the average of the two is taken as the comprehensive similarity matrix. The same greedy algorithm as for numerical fields is used to iteratively select samples to form a representative sample set covering the semantic features of the field.

[0011] Preferably, the process involves linearly integrating field attribution relationships, calling relationships, sample data, and retrieval rules into thought chain prompts, inputting them into a finely tuned large language model, generating preliminary labeling and classification results, and validating these results through a multi-model evaluation enhancement mechanism. The output data processing results include: The field attribution relationships, calling relationships, sample data, and retrieval rules are converted into structured text and integrated in a logical order to obtain the mind chain prompt words; The thought chain prompts are input into the fine-tuned large language model. Based on the contextual information and reasoning guidance in the thought chain prompts, the large language model performs field semantic analysis, sample feature matching, and classification label mapping step by step to generate the preliminary labeling and classification results containing field meaning, data type, business association, and classification label. The field meaning describes the semantic connotation of the field, and the business association information clarifies the core application scenario of the field in the call relationship. A multi-model evaluation framework is constructed to verify the relevance and consistency of the preliminary annotation and classification results. The relevance verification includes concatenating the preliminary results with the field attribution and calling relationships into evaluation text, inputting at least two different pre-trained large language models, generating a relevance score, and measuring the semantic consistency between the preliminary annotation results and the field association information. The consistency verification includes collecting the scores of different evaluation models, calculating the score standard deviation, and measuring the similarity of the evaluation results. If the mean relevance score is greater than a preset threshold and the standard deviation of the score is less than a preset threshold, the preliminary labeling and classification result is determined to pass the verification; otherwise, the thought chain prompts are readjusted, the preliminary labeling and classification result is regenerated and verified. The preliminary labeling and classification results that have passed the verification are encapsulated in a structured manner, and the data processing results containing field names, label meanings, data types, classification labels, and business association information are output.

[0012] Preferably, the unstructured industry standard documents are converted into a structured format, encoded using the M3E-base vector model, and stored in a vector database to form a data standard knowledge base. Based on the fields to be standardized in the data processing results, several data standards with the highest relevance are retrieved from the data standard knowledge base. This guides the large language model to generate regular expressions corresponding to the fields to be standardized. The regular expressions are then used to filter non-standard data rows, resulting in a non-standard dataset, including: Semantic parsing and structured extraction are performed on unstructured industry standard documents, and the extracted elements are encapsulated into structured data standard entries according to a fixed structure. The structured data standard entries are converted into text strings, and the text strings are vectorized using the M3E-base vector model to generate dense vectors of fixed dimensions to capture semantic features. The text strings and their dense vectors are stored in a vector database, and a mapping relationship between entries and vector indexes is established. From the data processing results, fields that need to be formatted are selected, the metadata information of the fields is determined, and the fields to be standardized are obtained. The metadata information of the fields to be standardized is integrated into the retrieval query text. The retrieval query text is converted into a query vector through the M3E-base vector model. The cosine similarity between the query vector and the dense vector of the text string is calculated in the vector database. The similarity scores are sorted from high to low, and the top few data standards are selected as the set of data standard rules with the highest relevance. The metadata information of the field to be standardized is integrated with the set of data standard rules into prompt words. The prompt words include the field data type, target format requirements, and business constraints, and are input into the large language model. Based on the format constraints and business rules in the prompt words, the big language generates regular expressions that can match standard format data. These regular expressions contain key format features such as the character length, character type, and position of special symbols in the fields. The format of each field data is validated by regular expressions, and data rows that do not match the regular expressions are marked as non-standard data rows. All non-standard data rows are aggregated to form the non-standard dataset.

[0013] Preferably, the step of randomly sampling multiple samples from the non-standard dataset, constructing a large language model for inputting thought chain prompt words, generating SQL conversion code to perform data standardization, iteratively filtering non-standard data rows until the non-standard data rate does not exceed a preset threshold, and outputting standardized data includes: Based on the aforementioned non-standard dataset, a stratified random sampling method was used to select multiple non-standard samples, and the sampled samples were deduplicated. Integrate non-standard samples, data standard rules, and task descriptions to convert non-standard data into a format that conforms to data standard rules, and construct mind chain prompts for generating the SQL conversion code; The thought chain prompts are input into the fine-tuned large language model. The large language model generates the SQL conversion code for data standardization based on the non-standard sample features, data standard rules and conversion logic in the thought chain prompts. The SQL conversion code includes data format conversion, outlier handling and business rule adaptation logic. The SQL transformation code is executed to standardize the non-standard dataset, and regular expressions are used to perform secondary filtering on the updated data. Non-standard data rows that still do not meet the standard are marked, and the current non-standard data rate is calculated. If the non-standard data rate exceeds the preset threshold, the non-standard data is resampled and transformed until the non-standard data rate does not exceed the preset threshold.

[0014] Preferably, the integration of non-standard samples, data standard rules, and task descriptions, converting non-standard data into a format conforming to data standard rules, and constructing thought chain prompts for generating the SQL conversion code includes: The non-standard samples are classified, organized, and labeled with features. The non-standard samples are then displayed in a formatted manner according to a fixed structure to form a list of non-standard data that the model can understand. Relevant rules related to the field to be standardized are extracted from the data standard knowledge base. Data type constraints, format requirements and verification logic are extracted and the rules are organized in a preset order to form the target benchmark for model reasoning. Based on the complete logical chain of the preset error analysis, rule matching, and correction scheme, determine the step-by-step derivation template; Based on a non-standard data list, target benchmark, and step-by-step derivation template, a complete thought chain prompt is formed, which forces the model to output step by step according to error analysis, rule matching, and correction scheme, and constrains the SQL code format.

[0015] Preferably, the SQL generation capability based on the fine-tuned model and the standardized data, performing database index optimization and anomaly detection, receiving user natural language queries, generating SQL query statements and visualization scripts, and outputting visualization reports, includes: Collect query logs of standardized data, extract frequently queried fields and multi-table join scenarios; input the query logs into a fine-tuned large language model, and combine the large language model with the table structure information in the structured knowledge base to analyze the query popularity and data distribution characteristics of fields, generate index optimization suggestions, generate index creation SQL statements, and verify the effect by comparing the query response time before and after the index creation SQL statements are executed; Based on the verification rules in the data standard knowledge base, an anomaly detection rule base is constructed, and the rule reasoning capabilities of the standardized data and the large language model are used to detect abnormal data in real time. Upon receiving a user's natural language query, the system converts the user's query text into an executable SQL query statement using a large language model, executes the SQL query statement, obtains the SQL query results, and generates a visualization script based on the SQL query results. This integrates index optimization, anomaly detection, and the output of a visualization report of the user's SQL query results.

[0016] Secondly, this application discloses an enterprise data analysis device, which adopts the following technical solution, including: The model fine-tuning module is used to fine-tune the large language model for Text2SQL tasks using low-rank adaptation technology, and to build a structured knowledge base that includes enterprise data dictionary, field call relationship and national standards. The structured knowledge base is vectorized and stored through the M3E-base vector model. The data sampling module is used to call the data query interface based on the data fields to be processed, use a greedy algorithm to sample representative samples from the structured knowledge base, and retrieve the most relevant rules in the structured knowledge base. The data processing module is used to linearly integrate field attribution relationships, calling relationships, sample data and retrieval rules into mind chain prompts, input them into a fine-tuned large language model, generate preliminary labeling and classification results, and verify the preliminary labeling and classification results through a multi-model evaluation enhancement mechanism, and output the data processing results. The data filtering module is used to convert unstructured industry standard documents into a structured format, encode them using the M3E-base vector model, and store them in a vector database to form a data standard knowledge base. Based on the fields to be standardized in the data processing results, the module retrieves the most relevant data standards from the data standard knowledge base, guides the large language model to generate regular expressions corresponding to the fields to be standardized, and filters non-standard data rows using regular expressions to obtain a non-standard dataset. The data conversion module is used to randomly sample multiple samples from the non-standard dataset, construct a large language model for inputting thought chain prompt words, generate SQL conversion code to perform data standardization, iteratively filter non-standard data rows until the non-standard data rate does not exceed a preset threshold, and output standardized data. The data query module is used to perform database index optimization and anomaly detection based on the SQL generation capability of the fine-tuned model and the standardized data, and to receive user natural language queries, generate SQL query statements and visualization scripts, and output visualization reports.

[0017] Thirdly, this application also provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores computer programs that can be loaded by the processor and executed, such as the enterprise data analysis method described above.

[0018] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above in the enterprise data analysis method.

[0019] In summary, this application uses low-rank adaptation technology to fine-tune a large language model for Text2SQL tasks, constructing a structured knowledge base containing enterprise data dictionaries, field call relationships, and national standards, which is then vectorized and stored using the M3E-base vector model. Based on the data fields to be processed, a data query interface is called, and a greedy algorithm is used to sample representative samples from the structured knowledge base and retrieve the most relevant rules. Field attribution relationships, call relationships, sample data, and retrieval rules are integrated into thought chain prompts, which are input into the fine-tuning model to generate preliminary labeled classification results. After multi-model evaluation and verification, the data processing results are output. Unstructured industry standards are converted into a structured data standard knowledge base. Based on the fields to be standardized in the data processing results, relevant data standards are retrieved, guiding the model to generate regular expressions to filter non-standard data rows to obtain a non-standard dataset. SQL conversion code is generated by sampling and iteratively cleaned until the non-standard data rate meets the standard, and standardized data is output. Based on the SQL generation capability of the fine-tuning model and the standardized data, database index optimization and anomaly detection are performed. The system receives user natural language queries, generates SQL and visualization scripts, and outputs visualization reports. This approach employs low-rank adaptation techniques to fine-tune large language models for Text2SQL tasks, requiring only a small amount of local enterprise data (combined with publicly available datasets) to adapt to business scenarios, avoiding the reliance of traditional machine learning on large-scale labeled samples. A structured knowledge base containing enterprise data dictionaries, field call relationships, and national standards is constructed. Cross-domain dynamic knowledge retrieval is achieved through the M3E-base vector model, and pre-existing information such as attribution and call relationships is integrated using thought chain prompts. This allows the model to understand complex business logic without manual rule definition, significantly improving generalization ability. By integrating the entire data processing, standardization, and analysis process end-to-end, and through automated labeling and classification, iterative regular expression generation and cleaning, the traditionally manual pre-processing steps are transformed into automated model processing, greatly reducing the workload of data engineers in rule writing and maintenance. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a business data analysis method.

[0021] Figure 2 This is a structural block diagram of an enterprise data analysis device. Detailed Implementation

[0022] The following combination Figure 1- Figure 2 This application will be described in further detail.

[0023] To address the technical challenges of high labor costs, weak generalization capabilities, and fragmented processes in traditional enterprise data analysis methods, this invention provides an end-to-end enterprise data analysis method based on a large language model. By integrating technologies such as low-rank adaptive fine-tuning, retrieval-enhanced generation, and thought chain reasoning, it achieves intelligent processing of the entire process of data organization, standardization, management, and analysis.

[0024] Reference Figure 1 The embodiments of this application include at least steps S10 to S60.

[0025] S10 employs low-rank adaptation technology to fine-tune the large language model for Text2SQL tasks, constructs a structured knowledge base containing enterprise data dictionaries, field call relationships, and national standards, and stores the structured knowledge base in vector form through the M3E-base vector model; S20: Based on the data fields to be processed, call the data query interface, use a greedy algorithm to sample representative samples from the structured knowledge base, and retrieve the most relevant rules in the structured knowledge base. S30 linearly integrates field attribution relationships, calling relationships, sample data, and retrieval rules into thought chain prompts, inputs them into a fine-tuned large language model, generates preliminary labeling and classification results, verifies the preliminary labeling and classification results through a multi-model evaluation enhancement mechanism, and outputs data processing results. S40 converts unstructured industry standard documents into a structured format, encodes them using the M3E-base vector model, and stores them in a vector database to form a data standard knowledge base. Based on the fields to be standardized in the data processing results, it retrieves the most relevant data standards from the data standard knowledge base, guides the large language model to generate regular expressions corresponding to the fields to be standardized, and filters out non-standard data rows using regular expressions to obtain a non-standard dataset. S50 randomly samples multiple samples from a non-standard dataset, constructs a large language model for inputting thought chain prompt words, generates SQL conversion code to perform data standardization, iteratively filters non-standard data rows until the non-standard data rate does not exceed a preset threshold, and outputs standardized data. S60, based on the fine-tuned model's SQL generation capabilities and standardized data, performs database index optimization and anomaly detection, and receives user natural language queries, generates SQL query statements and visualization scripts, and outputs visualization reports.

[0026] Specifically, a low-rank adaptation technique is used to fine-tune a large language model for Text2SQL tasks. A structured knowledge base containing enterprise data dictionaries, field call relationships, and national standards is constructed and stored using an M3E-base vector model. Simultaneously, unstructured industry standard documents are converted into a structured format to build a data standard knowledge base. Based on this, a data query interface is invoked based on the data fields to be processed. A greedy algorithm is used to sample representative samples from the structured knowledge base and retrieve the most relevant rules. The field attribution relationships, call relationships, sample data, and retrieval rules are integrated to fine-tune the model for inputting thought chain prompts, generating preliminary labeled classification results. After multi-model evaluation and verification, the data processing results are output. Based on the fields to be standardized in the data processing results, the data standard knowledge base is retrieved to generate regular expressions to filter non-standard data rows and obtain a non-standard dataset. Non-standard samples are randomly sampled to construct mind chain prompt words to input into the model and generate SQL conversion code. The data is iteratively cleaned until the non-standard data rate meets the standard and then standardized data is output. Finally, based on the SQL generation capability of the fine-tuned model and the standardized data, database index optimization and anomaly detection are performed. The system receives user natural language queries, generates SQL query statements and visualization scripts, and outputs visualization reports, realizing the intelligent end-to-end process of enterprise data analysis from data preprocessing to result output.

[0027] In some embodiments, step S10 specifically includes the following steps: constructing a fine-tuning sample set, collecting the CSpider Chinese dataset and enterprise local data; selecting a pre-trained large language model as the base model and freezing all pre-training parameters, inserting a dimensionality reduction matrix and an increasing matrix into the QKV attention unit of the Transformer layer, and performing fine-tuning training; inputting samples into the model in batches and concatenating them into a query format of question and objective; using the cross-entropy loss function, fixing the pre-training parameters and only updating the dimensionality reduction matrix and the increasing matrix; iteratively training until the loss on the validation set no longer decreases after a certain number of consecutive preset periods; fusing and saving the model; fusing the dimensionality reduction matrix and the increasing matrix with the pre-training parameters, and outputting the model. The weights are updated to the sum of the products of the pre-trained weights and the reduced and increased dimensions matrices, saving the fine-tuned model adapted to the enterprise task; knowledge sources are integrated: an enterprise data dictionary containing field names, data types, corresponding database tables, and business descriptions is extracted from database metadata, and field call relationships of quadruples are mined from IT system operation logs; the knowledge sources are structured to form unified format entries, obtaining structured knowledge entries; the M3E-base Chinese vector model is used to convert structured knowledge entries into text strings and generate high-dimensional dense vectors to capture the semantic features of structured knowledge entries; the knowledge entry text and corresponding vectors are stored in a vector database, establishing a mapping relationship between knowledge text and vector indexes.

[0028] Low-rank adaptation (LoRA) fine-tuning primarily reduces computational resource consumption by freezing the pre-trained model parameters and training only the low-rank matrix (reduced-dimensional matrix AA and increased-dimensional matrix BB). Its loss function is formulated as follows: ; Where N is the sample size. For input data, For target tags, The model parameters are defined using cross-entropy loss.

[0029] By combining the CSpider Chinese dataset with local enterprise data, the model's adaptability to Chinese semantics and complex SQL generation is effectively improved.

[0030] Specifically, this embodiment includes two parts: First, fine-tuning of the large language model Text2SQL task. A fine-tuning sample set containing the CSpider Chinese dataset and local enterprise data is constructed. A pre-trained large language model is selected as the base model and all parameters are frozen. Dimensionality reduction and dimension increase matrices are inserted into the QKV attention unit of the Transformer layer. Samples are concatenated in batches to form question and target query formats. The cross-entropy loss function is used to update only the two matrices. Iterative training is performed until the loss no longer decreases on a continuous preset period validation set. The two matrices are fused with the pre-trained parameters (the output layer weights are the sum of the pre-trained weights and the product of the two matrices), and the fine-tuned model adapted to the enterprise task is saved. Second, the construction and vectorized storage of a structured knowledge base. Knowledge sources are integrated (enterprise data dictionaries containing field names are extracted from database metadata, and four-tuple field call relationships are mined from IT system operation logs). Structured processing is performed to form unified format entries. The M3E-base Chinese vector model is used to convert the entries into text strings and generate high-dimensional dense vectors to capture semantic features. The knowledge entry text and vectors are stored in a vector database, and a mapping between knowledge text and vector indexes is established.

[0031] In some embodiments, step S20 specifically includes the following steps: based on the basic metadata information of the data fields to be processed, by parsing the structured metadata of the basic metadata information, the business attribute description of the fields is obtained, and a field metadata set is generated; based on the field metadata set, a data query interface is called to connect to the enterprise database, a structured query statement is executed, and non-empty data records of the fields to be processed are filtered out; duplicate values ​​and invalid format data are removed from the non-empty data records to form an original sample set containing the actual values ​​of the fields; based on the original sample set and the field association information of the structured knowledge base, a greedy algorithm is used to sample representative samples to generate a representative sample set; based on the field metadata set and the representative sample set, a retrieval query text is constructed; the retrieval query text is input into the M3E-base vector model to generate a query vector, and semantic similarity retrieval is performed in the vector database of the structured knowledge base, the cosine similarity between the query vector and each rule vector in the knowledge base is calculated, and the rules are sorted from high to low according to the cosine similarity, and the top-ranked rules are selected as the set of relevant rules with the highest relevance.

[0032] Furthermore, step S20 also includes the following steps: The field data types include numeric fields and string fields. For numeric fields, the original sample set is standardized, the Euclidean distance similarity matrix between samples is calculated, and samples are iteratively selected using a greedy algorithm: first, an initial sample is randomly selected, then the furthest distance between the remaining samples and the existing sample set is calculated in a loop, and the sample with the furthest distance is added to the representative sample set until the number of samples reaches a preset threshold. For string fields, the string values ​​of the original sample set are converted into text vectors through a word vector model, the cosine similarity matrix and Jaccard similarity matrix between samples are calculated, and the average of the two is taken as the comprehensive similarity matrix. The same greedy algorithm as for numeric fields is used to iteratively select samples to form a representative sample set covering the semantic features of the field.

[0033] Specifically, by parsing the basic metadata information of the data fields to be processed, a set of field metadata is generated to obtain business attribute descriptions. Based on this set, a data query interface is called to connect to the enterprise database, and a structured query statement is executed to filter non-empty data records, remove duplicate values ​​and invalid format data, and form an original sample set containing the actual values ​​of the fields. Combining the original sample set and the field association information of the structured knowledge base, a greedy algorithm is used to sample and generate a representative sample set. Then, based on the field metadata set and the representative sample set, a retrieval query text is constructed, input into the M3E-base vector model to generate a query vector, and semantic similarity retrieval is performed in the vector database of the structured knowledge base. The cosine similarity between the query vector and each rule vector is calculated and sorted by score. The top-ranked rules are selected as the set of the most relevant rules, achieving accurate matching between samples and rules.

[0034] In some embodiments, step S30 specifically includes the following steps: converting field attribution relationships, call relationships, sample data, and retrieval rules into structured text and integrating them in logical order to obtain thought chain prompts; inputting the thought chain prompts into a fine-tuned large language model, which, based on the contextual information and reasoning guidance in the thought chain prompts, performs field semantic analysis, sample feature matching, and classification label mapping step by step to generate preliminary labeling and classification results containing field meanings, data types, business associations, and classification labels, wherein the field meanings describe the semantic connotations of the fields, and the business association information clarifies the core application scenarios of the fields in the call relationships; constructing a multi-model evaluation framework to verify the relevance and consistency of the preliminary labeling and classification results, and the relevance... The validation process includes concatenating the preliminary results with field attribution and calling relationships into an evaluation text, inputting at least two different pre-trained large language models, generating relevance scores, and measuring the semantic consistency between the preliminary annotation results and field association information. Consistency validation involves collecting scores from different evaluation models, calculating the standard deviation of the scores, and measuring the similarity of the evaluation results. If the mean relevance score is greater than a preset threshold and the standard deviation of the scores is less than a preset threshold, the preliminary annotation classification result is considered to have passed the validation. Otherwise, the thought chain prompts are readjusted, the preliminary annotation classification result is regenerated, and validated again. The validated preliminary annotation classification result is then structured and encapsulated, and the output is a data processing result containing field names, annotation meanings, data types, classification labels, and business association information.

[0035] To quantify the semantic consistency between the initial annotation classification results and field association information, this embodiment uses BERTS score as the relevance verification metric. The specific steps are as follows: Evaluation text construction: Concatenate the initial annotation results with field attribution and calling relationships to form the evaluation text; BERTS score calculation: Input at least two pre-trained large language models (such as GPT-4 and GLM-4), and the models output relevance scores. BERTS score is calculated using the following formula:

[0036] Where c represents the labeled result generated by the model, and r represents the true label. The cosine similarity of the BERT word embedding vectors is used, and the harmonic mean of precision and recall is taken as the F1 score.

[0037] To ensure the stability of the evaluation results, this embodiment achieves consistency verification by calculating the standard deviation of scores from different models: at least two evaluation models (such as...) are obtained. Relevance score for the same annotation result For natural numbers, the rating variability is calculated using the following formula: ; in, The average score of k models, The standard deviation is taken as the threshold. If the standard deviation does not exceed the threshold, the evaluation results of different models are considered to be consistent.

[0038] Specifically, by integrating field attribution relationships, calling relationships, sample data, and retrieval rules into structured text and generating thought chain prompts in logical order, the data is input into a fine-tuned large language model. Based on the contextual information of the prompts and inference guidance, the model performs field semantic analysis, sample feature matching, and classification label mapping step by step, generating preliminary labeled classification results containing field meanings, data types, business associations, and classification labels. A multi-model evaluation framework is constructed to verify the relevance and consistency of the preliminary results. The relevance verification concatenates the preliminary results with field attribution and calling relationships into evaluation text, inputs at least two pre-trained models to generate scores to measure semantic consistency, and the consistency verification calculates the standard deviation of the scores to measure evaluation similarity. After verification, the structured encapsulated output contains data processing results containing field names, label meanings, data types, classification labels, and business association information. If the verification fails, the prompts are readjusted and the generated results are verified again, achieving accurate and reliable output of data processing results.

[0039] In some embodiments, step S40 specifically includes the following steps: performing semantic parsing and structured extraction on unstructured industry standard documents, encapsulating the extracted elements into structured data standard entries according to a fixed structure; converting the structured data standard entries into text strings, using the M3E-base vector model to vectorize the text strings, generating dense vectors of fixed dimensions to capture semantic features, storing the text strings and their dense vectors in a vector database, and establishing a mapping relationship between entries and vector indexes; selecting fields that need to be formatted from the data processing results, determining the metadata information of the fields, and obtaining the fields to be standardized; integrating the metadata information of the fields to be standardized into retrieval query text, converting the retrieval query text into query vectors using the M3E-base vector model, and then... The database calculates the cosine similarity between the query vector and the dense vector of the text string, and sorts them from high to low similarity scores. The top-ranked data standards are selected as the set of data standard rules with the highest relevance. Metadata information of the field to be standardized is integrated with the set of data standard rules to form prompt words. These prompt words include field data type, target format requirements, and business constraints, and are input into the large language model. The large language model generates regular expressions that match the standard format data based on the format constraints and business rules in the prompt words. These regular expressions include key format features such as field character length, character type, and the position of special symbols. The regular expressions are used to perform format validation on each line of the field data. Data lines that do not match the regular expressions are marked as non-standard data lines. All non-standard data lines are aggregated to form a non-standard dataset.

[0040] Specifically, unstructured industry standard documents undergo semantic parsing and structured extraction. Elements are encapsulated into structured data standard entries with a fixed structure. After being converted into text strings, fixed-dimensional dense vectors are generated using the M3E-base vector model to capture semantic features and stored in a vector database to establish an entry-vector index mapping. Fields requiring standardized formats are selected from the data processing results to determine metadata information and obtain fields to be standardized. The metadata is integrated into retrieval query text and quantified. The cosine similarity between the data and entry vectors is calculated in the vector database, and the top few data standards are selected as the rule set based on the scores. The metadata and rule set are integrated into prompt words containing field data types, format requirements, and business constraints, which are input into a large language model to generate regular expressions containing character length, type, and special symbol position features. The regular expressions are used to validate field data line by line, marking non-standard data lines that do not match and summarizing them to form a non-standard dataset, thus achieving the pre-processing standardization of multi-source heterogeneous data.

[0041] In some embodiments, step S50 specifically includes the following steps: Based on the non-standard dataset, multiple non-standard samples are selected using a stratified random sampling method, and the sampled samples are deduplicated; the non-standard samples, data standard rules, and task description are integrated to convert the non-standard data into a format that conforms to the data standard rules, and a thought chain prompt is constructed to generate SQL conversion code; the thought chain prompt is input into a fine-tuned large language model, and the large language model generates SQL conversion code for data standardization based on the non-standard sample features, data standard rules, and conversion logic reasoning in the thought chain prompt; the SQL conversion code includes data format conversion, outlier handling, and business rule adaptation logic; the SQL conversion code is executed to standardize the non-standard dataset, and regular expressions are used to perform secondary filtering on the updated data, marking non-standard data rows that still do not meet the standards, and calculating the current non-standard data rate; if the non-standard data rate exceeds a preset threshold, the process returns to resample and convert the non-standard data until the non-standard data rate does not exceed the preset threshold.

[0042] Furthermore, step S50 also includes the following steps: classifying and labeling the sampled non-standard samples, and displaying the non-standard samples in a fixed format to form a non-standard data list that the model can understand; extracting relevant rules related to the fields to be standardized from the data standard knowledge base, extracting data type constraints, format requirements and verification logic, organizing the rules in a preset order to form the target benchmark for model inference; determining the step-by-step derivation template based on the preset complete logical chain of error analysis, rule matching and correction scheme; forming complete thought chain prompts based on the non-standard data list, target benchmark and step-by-step derivation template, forcing the model to output step by step according to error analysis, rule matching and correction scheme, and constraining the SQL code format.

[0043] Specifically, based on the non-standard dataset, multiple non-standard samples are selected through stratified random sampling and deduplication. After the sampled samples are classified, organized, and labeled with features, they are displayed in a fixed format to form a non-standard data list. Relevant rules for the fields to be standardized are extracted from the data standard knowledge base. Data type constraints, format requirements, and verification logic are extracted and organized in a preset order to form the target benchmark for model inference. The step-by-step derivation template is determined based on the complete logical chain of preset error analysis, rule matching, and correction scheme. The non-standard data list, target benchmark, and step-by-step derivation template are integrated to construct the thought chain prompt words (forcing the model to output in steps according to error analysis, rule matching, and correction scheme, and constraining the SQL code format). The fine-tuned large language model is input to generate SQL conversion code containing data format conversion, outlier handling, and business rule adaptation logic. After the code is executed to standardize the non-standard dataset, regular expressions are called to filter out non-standard data rows. The non-standard data rate is calculated. If it exceeds the preset threshold, it returns to resampling and conversion until the non-standard data rate reaches the standard, thus achieving accurate iterative cleaning of non-standard data.

[0044] In some embodiments, step S60 specifically includes the following steps: collecting query logs of standardized data, extracting high-frequency query fields and multi-table join scenarios; inputting the query logs into a fine-tuned large language model, which, combined with table structure information in a structured knowledge base, analyzes field query popularity and data distribution characteristics, generates index optimization suggestions, and generates index creation SQL statements; after executing the index creation SQL statements, the effect is verified by comparing the query response time before and after optimization; based on the verification rules in the data standard knowledge base, an anomaly detection rule base is constructed, and the rule reasoning capabilities of standardized data and the large language model are used to detect abnormal data in real time; when a user's natural language query is received, the large language model converts the user's query text into an executable SQL query statement, executes the SQL query statement, obtains the SQL query result, and generates a visualization script based on the SQL query result, integrating index optimization, anomaly detection, and user SQL query result output visualization report.

[0045] Specifically, by collecting query logs from standardized data, high-frequency query fields and multi-table join scenarios are extracted. These are then input into a finely tuned large language model, combined with table structure information from a structured knowledge base. The model analyzes field query popularity and data distribution characteristics to generate index optimization suggestions and index creation SQL statements. After execution, the query response time before and after optimization is compared to verify the effect. An anomaly detection rule base is built based on the validation rules in the data standard knowledge base. The model's rule reasoning capabilities are used to detect abnormal data in real time. When receiving natural language queries from users, the model converts the query text into executable SQL query statements. After execution, a visualization script is generated based on the results. The system integrates index optimization, anomaly detection, and user query results to output a visual report.

[0046] The implementation principle of an enterprise data analysis method in this application is as follows: A large language model is fine-tuned using a low-rank adaptation technique for Text2SQL tasks. A structured knowledge base containing an enterprise data dictionary, field call relationships, and national standards is constructed and stored using an M3E-base vector model. Based on the data fields to be processed, a data query interface is called. A greedy algorithm is used to sample representative samples from the structured knowledge base and retrieve the most relevant rules. Field attribution relationships, call relationships, sample data, and retrieval rules are integrated as thought chain prompts, which are input into the fine-tuning model to generate preliminary labeled classification results. After multi-model evaluation and verification, the data processing results are output. Unstructured industry standards are converted into a structured data standard knowledge base. Based on the fields to be standardized in the data processing results, relevant data standards are retrieved. The model is guided to generate regular expressions to filter non-standard data rows to obtain a non-standard dataset. SQL conversion code is generated through sampling and iterative cleaning until the non-standard data rate meets the standard, and standardized data is output. Based on the SQL generation capability of the fine-tuning model and the standardized data, database index optimization and anomaly detection are performed. User natural language queries are received to generate SQL and visualization scripts, and visualization reports are output. This approach employs low-rank adaptation techniques to fine-tune large language models for Text2SQL tasks, requiring only a small amount of local enterprise data (combined with publicly available datasets) to adapt to business scenarios, avoiding the reliance of traditional machine learning on large-scale labeled samples. A structured knowledge base containing enterprise data dictionaries, field call relationships, and national standards is constructed. Cross-domain dynamic knowledge retrieval is achieved through the M3E-base vector model, and pre-existing information such as attribution and call relationships is integrated using thought chain prompts. This allows the model to understand complex business logic without manual rule definition, significantly improving generalization ability. By integrating the entire data processing, standardization, and analysis process end-to-end, and through automated labeling and classification, iterative regular expression generation and cleaning, the traditionally manual pre-processing steps are transformed into automated model processing, greatly reducing the workload of data engineers in rule writing and maintenance.

[0047] Figure 1 This is a flowchart illustrating an enterprise data analysis method in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0048] Based on the same technical concept, referring to Figure 2 This application also provides an enterprise data analysis device, which adopts the following technical solution: the device includes: The model fine-tuning module is used to fine-tune the large language model for Text2SQL tasks using low-rank adaptation technology, and to build a structured knowledge base that includes enterprise data dictionary, field call relationship and national standards. The structured knowledge base is vectorized and stored through the M3E-base vector model. The data sampling module is used to call the data query interface based on the data fields to be processed, use a greedy algorithm to sample representative samples from the structured knowledge base, and retrieve the most relevant rules in the structured knowledge base. The data processing module is used to linearly integrate field attribution relationships, calling relationships, sample data and retrieval rules into mind chain prompts, input them into a fine-tuned large language model, generate preliminary labeling and classification results, and verify the preliminary labeling and classification results through a multi-model evaluation enhancement mechanism, and output the data processing results. The data filtering module is used to convert unstructured industry standard documents into structured formats, encode them using the M3E-base vector model, and store them in a vector database to form a data standard knowledge base. Based on the fields to be standardized in the data processing results, it retrieves the most relevant data standards from the data standard knowledge base, guides the large language model to generate regular expressions corresponding to the fields to be standardized, and filters non-standard data rows using regular expressions to obtain a non-standard dataset. The data conversion module is used to randomly sample multiple samples from a non-standard dataset, construct a large language model for inputting thought chain prompt words, generate SQL conversion code to perform data standardization, iteratively filter non-standard data rows until the non-standard data rate does not exceed a preset threshold, and output standardized data. The data query module is used to perform database index optimization and anomaly detection based on the fine-tuned model's SQL generation capabilities and standardized data. It also receives user natural language queries, generates SQL query statements and visualization scripts, and outputs visualization reports.

[0049] In some embodiments, the data fine-tuning module is specifically used to construct a fine-tuning sample set, collect the CSpider Chinese dataset and enterprise local data; A pre-trained large language model was selected as the base model and all pre-training parameters were frozen. Dimensionality reduction and dimension increase matrices were inserted into the QKV attention unit of the Transformer layer and fine-tuned. Samples were input into the model in batches and concatenated into a query format of question and objective. The cross-entropy loss function was adopted. The pre-training parameters were fixed and only the matrix dimensionality reduction and dimension increase matrices were updated. Iterative training was carried out until the loss on the validation set no longer decreased after a certain number of consecutive preset periods. The model was then fused and saved. The dimensionality reduction matrix and the dimensionality increase matrix are fused with the pre-trained parameters, and the output layer weights are updated to the sum of the products of the pre-trained weights and the dimensionality reduction matrix and the dimensionality increase matrix, thus saving the fine-tuned model adapted to the enterprise task. Integrating knowledge sources: Extracting an enterprise data dictionary containing field names, data types, corresponding database tables, and business descriptions from database metadata; mining field call relationships of four-tuples from IT system operation logs; The knowledge source is processed in a structured manner to form entries in a unified format, thus obtaining structured knowledge entries; The M3E-base Chinese vector model is used to convert structured knowledge entries into text strings and generate high-dimensional dense vectors to capture the semantic features of structured knowledge entries. The knowledge entry text and corresponding vectors are stored in a vector database to establish a mapping relationship between knowledge text and vector index.

[0050] In some embodiments, the data sampling module is specifically used to obtain the business attribute description of the field by parsing the structured metadata of the basic metadata information of the data field to be processed, and generate a field metadata set. Based on the field metadata set, the data query interface is called to connect to the enterprise database, execute structured query statements, and filter out non-empty data records of the fields to be processed; duplicate values ​​and invalid format data are removed from the non-empty data records to form an original sample set containing the actual values ​​of the fields; Based on the field association information of the original sample set and the structured knowledge base, a greedy algorithm is used to sample representative samples to generate a representative sample set. Based on the field metadata set and representative sample set, a retrieval query text is constructed. The retrieval query text is input into the M3E-base vector model to generate a query vector. Semantic similarity retrieval is performed in the vector database of the structured knowledge base. The cosine similarity between the query vector and each rule vector in the knowledge base is calculated. The rules are sorted from high to low according to the cosine similarity and the top-ranked rules are selected as the set of relevant rules with the highest relevance.

[0051] In some embodiments, the data sampling module is also used for fields whose data types include numeric fields and string fields. For numeric fields, the original sample set is standardized, the Euclidean distance similarity matrix between samples is calculated, and samples are iteratively selected using a greedy algorithm: first, an initial sample is randomly selected, then the farthest distance between the remaining samples and the existing sample set is calculated in a loop, and the sample with the farthest distance is selected to be added to the representative sample set until the number of samples reaches a preset threshold. For string-type fields, the string values ​​of the original sample set are converted into text vectors through a word vector model. The cosine similarity matrix and Jaccard similarity matrix between samples are calculated, and the average of the two is taken as the comprehensive similarity matrix. The same greedy algorithm as for numerical fields is used to iteratively select samples to form a representative sample set covering the semantic features of the field.

[0052] In some embodiments, the data processing module is specifically used to convert field attribution relationships, calling relationships, sample data and retrieval rules into structured text and integrate them in a logical order to obtain mind chain prompt words; The thought chain prompts are input into the fine-tuned large language model. Based on the contextual information and reasoning guidance in the thought chain prompts, the large language model performs field semantic analysis, sample feature matching, and classification label mapping step by step to generate preliminary labeling and classification results containing field meaning, data type, business association, and classification labels. The field meaning describes the semantic connotation of the field, and the business association information clarifies the core application scenario of the field in the call relationship. A multi-model evaluation framework is constructed to verify the relevance and consistency of the preliminary annotation and classification results. The relevance verification includes concatenating the preliminary results with the field attribution and calling relationships into the evaluation text, inputting at least two different pre-trained large language models, generating a relevance score, and measuring the semantic consistency between the preliminary annotation results and the field association information. The consistency verification includes collecting the scores of different evaluation models, calculating the standard deviation of the scores, and measuring the similarity of the evaluation results. If the mean relevance score is greater than the preset threshold and the standard deviation of the score is less than the preset threshold, the preliminary labeling and classification result is deemed to have passed the verification; otherwise, the thought chain prompts are readjusted, the preliminary labeling and classification result is regenerated and verified. The preliminary labeling and classification results that pass the verification are encapsulated in a structured manner, and the data processing results are output, which include field names, label meanings, data types, classification labels, and business-related information.

[0053] In some embodiments, the data conversion module is specifically used to perform semantic parsing and structured extraction on unstructured industry standard documents, and to encapsulate the extracted elements into structured data standard entries according to a fixed structure. The standard entries of structured data are converted into text strings. The text strings are vectorized using the M3E-base vector model to generate dense vectors of fixed dimensions to capture semantic features. The text strings and their dense vectors are stored in a vector database, and a mapping relationship between entries and vector indexes is established. From the data processing results, select the fields that need to be formatted, determine the metadata information of the fields, and obtain the fields to be standardized; The metadata information of the fields to be standardized is integrated into the retrieval query text. The retrieval query text is converted into query vectors through the M3E-base vector model. The cosine similarity between the query vector and the dense vector of the text string is calculated in the vector database. The similarity scores are sorted from high to low, and the top few data standards are selected as the set of data standard rules with the highest relevance. The metadata information of the field to be standardized is integrated with the set of data standard rules into prompt words. The prompt words include the field data type, target format requirements, and business constraints, which are then input into the large language model. Based on the format constraints and business rules in the prompt words, the big language generates regular expressions that can match standard format data. These regular expressions contain key format features such as the character length, character type, and position of special symbols in the fields. The format of each field data is validated by regular expressions, and data rows that do not match the regular expressions are marked as non-standard data rows. All non-standard data rows are then aggregated to form a non-standard dataset.

[0054] In some embodiments, the data conversion module is specifically used to select multiple non-standard samples based on a non-standard dataset using a hierarchical random sampling method, and to perform deduplication on the sampled samples; Integrate non-standard samples, data standard rules, and task descriptions to convert non-standard data into a format that conforms to data standard rules, and construct mind chain prompts for generating SQL conversion code; The thought chain prompts are input into the fine-tuned large language model. The large language model generates SQL conversion code for data standardization based on the non-standard sample features, data standard rules and conversion logic reasoning in the thought chain prompts. The SQL conversion code includes data format conversion, outlier handling and business rule adaptation logic. The SQL transformation code is executed to standardize the non-standard dataset. Regular expressions are used to perform secondary filtering on the updated data, marking non-standard data rows that still do not meet the standards, and calculating the current non-standard data rate. If the non-standard data rate exceeds the preset threshold, the non-standard data is resampled and transformed until the non-standard data rate does not exceed the preset threshold.

[0055] In some embodiments, the data conversion module is specifically used to classify and organize the sampled non-standard samples and label their features, and to display the non-standard samples in a formatted manner according to a fixed structure to form a non-standard data list that the model can understand. Extract relevant rules related to the field to be standardized from the data standard knowledge base, extract data type constraints, format requirements and validation logic, organize the rules in a preset order to form the target benchmark for model reasoning; Based on the complete logical chain of the preset error analysis, rule matching, and correction scheme, determine the step-by-step derivation template; Based on a non-standard data list, target benchmark, and step-by-step derivation template, a complete thought chain prompt is formed, which forces the model to output step by step according to error analysis, rule matching, and correction scheme, and constrains the SQL code format.

[0056] In some embodiments, the data query module is specifically used to collect query logs of standardized data, extract frequently queried fields and multi-table join scenarios; input the query logs into the fine-tuned large language model, the large language model combines the table structure information in the structured knowledge base to analyze the field query popularity and data distribution characteristics, generate index optimization suggestions, generate index creation SQL statements, and verify the effect by comparing the query response time before and after the index creation SQL statements are executed; Based on the verification rules in the data standard knowledge base, an anomaly detection rule base is constructed, and the rule reasoning capabilities of standardized data and large language models are used to detect abnormal data in real time. Upon receiving a user's natural language query, the system converts the user's query text into an executable SQL query statement using a large language model, executes the SQL query statement, obtains the SQL query results, and generates a visualization script based on the SQL query results. This integrates index optimization, anomaly detection, and the output of a visualization report of the user's SQL query results.

[0057] This application also discloses a control device.

[0058] Specifically, the control device includes a memory and a processor, the memory storing computer programs that can be loaded by the processor and executed using the aforementioned enterprise data analysis methods.

[0059] This application also discloses a computer-readable storage medium.

[0060] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed as described above in the enterprise data analysis method. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0061] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for enterprise data analysis, characterized in that, include: We employ low-rank adaptation techniques to fine-tune a large language model for Text2SQL tasks, constructing a structured knowledge base that includes an enterprise data dictionary, field call relationships, and standards. The structured knowledge base is then stored in vector form using the M3E-base vector model. Based on the data fields to be processed, the data query interface is called, and a greedy algorithm is used to sample representative samples from the structured knowledge base, and to retrieve the most relevant rules in the structured knowledge base. The field attribution relationship, calling relationship, sample data and retrieval rules are linearly integrated into the mind chain prompt words, which are input into the fine-tuned large language model to generate preliminary labeling and classification results. The preliminary labeling and classification results are verified through a multi-model evaluation enhancement mechanism, and the data processing results are output. Unstructured industry standard documents are converted into a structured format, encoded using the M3E-base vector model, and stored in a vector database to form a data standard knowledge base. Based on the fields to be standardized in the data processing results, several data standards with the highest relevance in the data standard knowledge base are retrieved, and a large language model is guided to generate regular expressions corresponding to the fields to be standardized. Non-standard data rows are then filtered using regular expressions to obtain a non-standard dataset. Multiple samples are randomly sampled from the non-standard dataset, a large language model is constructed to input thought chain prompt words, SQL conversion code is generated to perform data standardization, non-standard data rows are iteratively filtered until the non-standard data rate does not exceed a preset threshold, and standardized data is output. Based on the fine-tuned model's SQL generation capability and the standardized data, database index optimization and anomaly detection are performed. The system also receives user natural language queries, generates SQL query statements and visualization scripts, and outputs visualization reports. Specifically, the process involves converting unstructured industry standard documents into a structured format, encoding them using the M3E-base vector model, and storing them in a vector database to form a data standard knowledge base. Based on the fields to be standardized in the data processing results, the process retrieves several data standards with the highest relevance from the data standard knowledge base, guides a large language model to generate regular expressions corresponding to the fields to be standardized, and uses these regular expressions to filter non-standard data rows, resulting in a non-standard dataset, including: Semantic parsing and structured extraction are performed on unstructured industry standard documents, and the extracted elements are encapsulated into structured data standard entries according to a fixed structure. The structured data standard entries are converted into text strings, and the text strings are vectorized using the M3E-base vector model to generate dense vectors of fixed dimensions to capture semantic features. The text strings and their dense vectors are stored in a vector database, and a mapping relationship between entries and vector indexes is established. From the data processing results, fields that need to be formatted are selected, the metadata information of the fields is determined, and the fields to be standardized are obtained. The metadata information of the fields to be standardized is integrated into the retrieval query text. The retrieval query text is converted into a query vector through the M3E-base vector model. The cosine similarity between the query vector and the dense vector of the text string is calculated in the vector database. The similarity scores are sorted from high to low, and the top few data standards are selected as the set of data standard rules with the highest relevance. The metadata information of the field to be standardized is integrated with the set of data standard rules into prompt words. The prompt words include the field data type, target format requirements, and business constraints, and are input into the large language model. Based on the format constraints and business rules in the prompt words, the big language generates regular expressions that can match standard format data. These regular expressions contain key format features such as the character length, character type, and position of special symbols in the fields. The format of each field data is validated by regular expressions, and data rows that do not match the regular expressions are marked as non-standard data rows. All non-standard data rows are aggregated to form the non-standard dataset.

2. The enterprise data analysis method according to claim 1, characterized in that, The text describes a process of fine-tuning a large language model for Text2SQL tasks using low-rank adaptation techniques, constructing a structured knowledge base containing an enterprise data dictionary, field call relationships, and standards. This structured knowledge base is then vectorized and stored using the M3E-base vector model. Build a fine-tuning sample set by collecting the Chinese CSpider dataset and local enterprise data; A pre-trained large language model was selected as the base model and all pre-training parameters were frozen. Dimensionality reduction and dimension increase matrices were inserted into the QKV attention unit of the Transformer layer and fine-tuned. Samples were input into the model in batches and concatenated into a query format of question and objective. The cross-entropy loss function was adopted. The pre-training parameters were fixed and only the matrix dimensionality reduction and dimension increase matrices were updated. Iterative training was carried out until the loss on the validation set no longer decreased after a certain number of consecutive preset periods. The model was then fused and saved. The dimensionality reduction matrix and the dimensionality increase matrix are fused with the pre-trained parameters, and the output layer weights are updated to the sum of the products of the pre-trained weights and the dimensionality reduction matrix and the dimensionality increase matrix, thus saving the fine-tuned model adapted to the enterprise task. Integrating knowledge sources: Extracting an enterprise data dictionary containing field names, data types, corresponding database tables, and business descriptions from database metadata; mining field call relationships of four-tuples from IT system operation logs; The knowledge source is processed in a structured manner to form entries in a unified format, thus obtaining structured knowledge entries; The M3E-base Chinese vector model is used to convert structured knowledge items into text strings and generate high-dimensional dense vectors to capture the semantic features of structured knowledge items. The knowledge item text and corresponding vectors are stored in a vector database to establish a mapping relationship between knowledge text and vector index.

3. The enterprise data analysis method according to claim 2, characterized in that, Based on the data fields to be processed, a data query interface is called, and a greedy algorithm is used to sample representative samples from the structured knowledge base. Then, several of the most relevant rules in the structured knowledge base are retrieved, including: Based on the basic metadata information of the data fields to be processed, the business attribute descriptions of the fields are obtained by parsing the structured metadata of the basic metadata information, and a set of field metadata is generated. Based on the aforementioned field metadata set, a data query interface is invoked to connect to the enterprise database, a structured query statement is executed, and non-empty data records of the fields to be processed are filtered out; duplicate values ​​and invalid format data are removed from the non-empty data records to form an original sample set containing the actual values ​​of the fields; Based on the field association information of the original sample set and the structured knowledge base, a greedy algorithm is used to sample representative samples to generate a representative sample set. Based on the aforementioned field metadata set and representative sample set, a retrieval query text is constructed. The retrieval query text is input into the M3E-base vector model to generate a query vector, and semantic similarity retrieval is performed in the vector database of the structured knowledge base. The cosine similarity between the query vector and each rule vector in the knowledge base is calculated, and the rules are sorted from high to low according to the cosine similarity. The top-ranked rules are selected as the set of relevant rules with the highest relevance.

4. The enterprise data analysis method according to claim 3, characterized in that, The representative sample set is generated by using a greedy algorithm to sample representative samples based on the field association information of the original sample set and the structured knowledge base, including: The field data types include numeric fields and string fields. For numeric fields, the original sample set is standardized, the Euclidean distance similarity matrix between samples is calculated, and samples are selected iteratively through a greedy algorithm: first, an initial sample is randomly selected, then the farthest distance between the remaining samples and the existing sample set is calculated in a loop, and the sample with the farthest distance is selected to be added to the representative sample set until the number of samples reaches a preset threshold. For string-type fields, the string values ​​of the original sample set are converted into text vectors through a word vector model. The cosine similarity matrix and Jaccard similarity matrix between samples are calculated, and the average of the two is taken as the comprehensive similarity matrix. The same greedy algorithm as for numerical fields is used to iteratively select samples to form a representative sample set covering the semantic features of the field.

5. The enterprise data analysis method according to claim 1, characterized in that, The process involves linearly integrating field attribution relationships, calling relationships, sample data, and retrieval rules into thought chain prompts, inputting them into a finely tuned large language model, generating preliminary labeling and classification results, and validating these results through a multi-model evaluation and enhancement mechanism. The output data processing results include: The field attribution relationships, calling relationships, sample data, and retrieval rules are converted into structured text and integrated in a logical order to obtain the mind chain prompt words; The thought chain prompts are input into the fine-tuned large language model. Based on the contextual information and reasoning guidance in the thought chain prompts, the large language model performs field semantic analysis, sample feature matching, and classification label mapping step by step to generate the preliminary labeling and classification results containing field meaning, data type, business association, and classification label. The field meaning describes the semantic connotation of the field, and the business association information clarifies the core application scenario of the field in the call relationship. A multi-model evaluation framework is constructed to verify the relevance and consistency of the preliminary annotation and classification results. The relevance verification includes concatenating the preliminary results with the field attribution and calling relationships into evaluation text, inputting at least two different pre-trained large language models, generating a relevance score, and measuring the semantic consistency between the preliminary annotation and classification results and the field association information. The consistency verification includes collecting the scores of different evaluation models, calculating the standard deviation of the scores, and measuring the similarity of the evaluation results. If the mean relevance score is greater than a preset threshold and the standard deviation of the score is less than a preset threshold, the preliminary labeling and classification result is determined to pass the verification; otherwise, the thought chain prompts are readjusted, the preliminary labeling and classification result is regenerated and verified. The preliminary labeling and classification results that have passed the verification are encapsulated in a structured manner, and the data processing results containing field names, label meanings, data types, classification labels, and business association information are output.

6. The enterprise data analysis method according to claim 1, characterized in that, The process involves randomly sampling multiple samples from the non-standard dataset, constructing a large language model for inputting thought chain prompts, generating SQL conversion code for data standardization, iteratively filtering non-standard data rows until the non-standard data rate does not exceed a preset threshold, and outputting standardized data, including: Based on the aforementioned non-standard dataset, a stratified random sampling method was used to select multiple non-standard samples, and the sampled samples were deduplicated. Integrate non-standard samples, data standard rules, and task descriptions to convert non-standard data into a format that conforms to data standard rules, and construct mind chain prompts for generating the SQL conversion code; The thought chain prompts are input into the fine-tuned large language model. The large language model generates the SQL conversion code for data standardization based on the non-standard sample features, data standard rules and conversion logic in the thought chain prompts. The SQL conversion code includes data format conversion, outlier handling and business rule adaptation logic. The SQL transformation code is executed to standardize the non-standard dataset, and regular expressions are used to perform secondary filtering on the updated data. Non-standard data rows that still do not meet the standard are marked, and the current non-standard data rate is calculated. If the non-standard data rate exceeds the preset threshold, the non-standard data is resampled and transformed until the non-standard data rate does not exceed the preset threshold.

7. The enterprise data analysis method according to claim 6, characterized in that, The process integrates non-standard samples, data standard rules, and task descriptions, converts non-standard data into a format conforming to data standard rules, and constructs thought chain prompts for generating the SQL conversion code, including: The non-standard samples are classified, organized, and labeled with features. The non-standard samples are then displayed in a formatted manner according to a fixed structure to form a list of non-standard data that the model can understand. Relevant rules related to the field to be standardized are extracted from the data standard knowledge base. Data type constraints, format requirements and verification logic are extracted and the rules are organized in a preset order to form the target benchmark for model reasoning. Based on the complete logical chain of the preset error analysis, rule matching, and correction scheme, determine the step-by-step derivation template; Based on a non-standard data list, target benchmark, and step-by-step derivation template, a complete thought chain prompt is formed, which forces the model to output step by step according to error analysis, rule matching, and correction scheme, and constrains the SQL code format.

8. The enterprise data analysis method according to claim 1, characterized in that, The SQL generation capability based on the fine-tuned model, along with the standardized data, performs database index optimization and anomaly detection, receives user natural language queries, generates SQL query statements and visualization scripts, and outputs visualization reports, including: Collect query logs of standardized data, extract frequently queried fields and multi-table join scenarios; input the query logs into a fine-tuned large language model, and combine the large language model with the table structure information in the structured knowledge base to analyze the query popularity and data distribution characteristics of fields, generate index optimization suggestions, generate index creation SQL statements, and verify the effect by comparing the query response time before and after the index creation SQL statements are executed; Based on the verification rules in the data standard knowledge base, an anomaly detection rule base is constructed, and the rule reasoning capabilities of the standardized data and the large language model are used to detect abnormal data in real time. Upon receiving a user's natural language query, the system converts the user's query text into an executable SQL query statement using a large language model, executes the SQL query statement, obtains the SQL query results, and generates a visualization script based on the SQL query results. This integrates index optimization, anomaly detection, and user-generated SQL query results visualization reports.

9. An enterprise data analysis device, characterized in that, The device includes: The model fine-tuning module is used to fine-tune the large language model for Text2SQL tasks using low-rank adaptation technology, and to build a structured knowledge base that includes an enterprise data dictionary, field call relationships and standards. The structured knowledge base is then stored in vector form using the M3E-base vector model. The data sampling module is used to call the data query interface based on the data fields to be processed, use a greedy algorithm to sample representative samples from the structured knowledge base, and retrieve the most relevant rules in the structured knowledge base. The data processing module is used to linearly integrate field attribution relationships, calling relationships, sample data and retrieval rules into mind chain prompts, input them into a fine-tuned large language model, generate preliminary labeling and classification results, and verify the preliminary labeling and classification results through a multi-model evaluation enhancement mechanism, and output the data processing results. The data filtering module is used to convert unstructured industry standard documents into a structured format, encode them using the M3E-base vector model, and store them in a vector database to form a data standard knowledge base. Based on the fields to be standardized in the data processing results, the module retrieves the most relevant data standards from the data standard knowledge base, guides the large language model to generate regular expressions corresponding to the fields to be standardized, and filters non-standard data rows using regular expressions to obtain a non-standard dataset. The data conversion module is used to randomly sample multiple samples from the non-standard dataset, construct a large language model for inputting thought chain prompt words, generate SQL conversion code to perform data standardization, iteratively filter non-standard data rows until the non-standard data rate does not exceed a preset threshold, and output standardized data. The data query module is used to perform database index optimization and anomaly detection based on the SQL generation capability of the fine-tuned model and the standardized data, and to receive user natural language queries, generate SQL query statements and visualization scripts, and output visualization reports. Specifically, the data filtering module is used to perform semantic parsing and structured extraction on unstructured industry standard documents, and to encapsulate the extracted elements into structured data standard entries according to a fixed structure. The structured data standard entries are converted into text strings, and the text strings are vectorized using the M3E-base vector model to generate dense vectors of fixed dimensions to capture semantic features. The text strings and their dense vectors are stored in a vector database, and a mapping relationship between entries and vector indexes is established. From the data processing results, fields that need to be formatted are selected, the metadata information of the fields is determined, and the fields to be standardized are obtained. The metadata information of the fields to be standardized is integrated into the retrieval query text. The retrieval query text is converted into a query vector through the M3E-base vector model. The cosine similarity between the query vector and the dense vector of the text string is calculated in the vector database. The similarity scores are sorted from high to low, and the top few data standards are selected as the set of data standard rules with the highest relevance. The metadata information of the field to be standardized is integrated with the set of data standard rules into prompt words. The prompt words include the field data type, target format requirements, and business constraints, and are input into the large language model. Based on the format constraints and business rules in the prompt words, the big language generates regular expressions that can match standard format data. These regular expressions contain key format features such as the character length, character type, and position of special symbols in the fields. The format of each field data is validated by regular expressions, and data rows that do not match the regular expressions are marked as non-standard data rows. All non-standard data rows are aggregated to form the non-standard dataset.

Images