Data storage method, device and equipment based on large model and storage medium

Abstract

The invention provides a data storage method, device and equipment based on a large model and a storage medium, and relates to the technical field of data processing, in particular to the technical field of artificial intelligence, large models and data storage. According to the specific implementation scheme, a heterogeneous data source of a data storage task is analyzed based on a large language model, so that an executable logic code used for processing the heterogeneous data source is generated; executing the logical code to process the heterogeneous data source in a controlled execution environment; monitoring the execution process of the logic code to obtain feedback information representing the state of the execution process; and according to the feedback information, optimizing the generation process of the logic code to obtain the logic code which meets the preset requirement and is used for data storage. According to the scheme provided by the invention, the warehousing efficiency can be remarkably improved, and the computing power resource consumption is reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and in particular to the technical field of artificial intelligence, large models, and data warehousing. BACKGROUND

[0002] In the digital era, data centers, as the core hub of information processing and storage, need to access and integrate data from various sources and in different formats. These data sources often exhibit heterogeneous forms, such as batch data and streaming data from files or object storage, incremental change logs obtained from business databases through change data capture technology, and event data received through application programming interfaces or message queues. How to efficiently and accurately automatically and structurally store the data from these heterogeneous data sources into data warehouses or data lakes, i.e., data warehousing, is one of the core challenges in the field of data engineering. SUMMARY

[0003] The present disclosure provides a large model-based data warehousing method, device, equipment, and storage medium.

[0004] According to an aspect of the present disclosure, a large model-based data warehousing method is provided, comprising:

[0005] analyzing a heterogeneous data source of a data warehousing task based on a large language model to generate executable logic code for processing the heterogeneous data source;

[0006] executing the logic code in a controlled execution environment to process the heterogeneous data source;

[0007] monitoring the execution process of the logic code to obtain feedback information representing the state of the execution process;

[0008] optimizing the generation process of the logic code according to the feedback information to obtain logic code for data warehousing that meets preset requirements.

[0009] According to another aspect of the present disclosure, a large model-based data warehousing device is provided, comprising:

[0010] a code generation module configured to analyze a heterogeneous data source of a data warehousing task based on a large language model to generate executable logic code for processing the heterogeneous data source;

[0011] a code execution module configured to execute the logic code in a controlled execution environment to process the heterogeneous data source;

[0012] a feedback acquisition module configured to monitor the execution process of the logic code to obtain feedback information representing the state of the execution process;

[0013] A closed-loop optimization module is configured to optimize the generation process of the logic code according to the feedback information to obtain the logic code for data warehousing that meets the preset requirements.

[0014] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0015] at least one processor; and

[0016] a memory in communication with the at least one processor; wherein

[0017] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any of the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to make the computer perform the method according to any of the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to any of the embodiments of the present disclosure.

[0020] According to the scheme provided by the present disclosure, the warehousing efficiency can be significantly improved, and the computing resource consumption can be reduced.

[0021] It should be understood that the contents described in this part are not intended to identify the key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0023] Figure 1 is a flowchart of a data warehousing method based on a large model according to an embodiment of the present disclosure;

[0024] Figure 2 is a flowchart of a data warehousing method based on a large model according to another embodiment of the present disclosure;

[0025] Figure 3 is a flowchart of a data warehousing method based on a large model according to another embodiment of the present disclosure;

[0026] Figure 4 is a workflow diagram of data warehousing according to an embodiment of the present disclosure;

[0027] Figure 5 is a schematic diagram of a system architecture for implementing data intelligent warehousing according to an embodiment of the present disclosure;

[0028] Figure 6 is a sequence diagram of a data warehousing method based on a large model according to an embodiment of the present disclosure;

[0029] Figure 7 is a structural schematic diagram of a data warehousing device based on a large model according to an embodiment of the present disclosure;

[0030] Figure 8 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Thus, those of ordinary skill in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Also, descriptions of known functions and constructions are omitted in the following description for clarity and conciseness.

[0032] With the development of artificial intelligence technology, especially large language models (LLM), using models to automatically generate code or instructions has become a new technical direction. In this direction, one approach is to use a large language model as an auxiliary tool to help developers generate code. For example, there are schemes that convert users' natural language requirements into structured query language (SQL) query statements through models, or automatically translate the stored procedure code of one database into another, which to some extent improves development efficiency. Another further approach is to directly embed a large language model into a real-time data processing pipeline. In this mode, each piece or batch of incoming data is sent to the large language model for online analysis, conversion, and structured processing. Although this method can use the intelligence of the model to deal with the variability of data formats, its inherent technical defects are also very significant: first, the inference process of the large language model itself requires intensive computation, resulting in a large delay in processing a single data request; second, due to the high latency, the system can process a limited amount of data per unit of time, i.e., the throughput is low, making it difficult to meet the data warehousing scenarios of high concurrency and large flow in data centers; finally, the valuable and expensive large model inference computing power is consumed in a large number of repetitive and procedural data format processing tasks, resulting in a great waste of resources, making the overall scheme inefficient and costly.

[0033] The related art has the following problems. When a large model is used to process a data warehousing task, if it is used as an offline auxiliary tool, the automation degree is limited and manual intervention is still required. If it is directly used for online real-time processing, it will encounter huge bottlenecks in performance and cost.

[0034] To at least partially solve one or more of the above problems and other potential problems, embodiments of the present disclosure provide a large model-based data warehousing method. The technical solutions of the embodiments of the present disclosure can significantly improve the warehousing efficiency, reduce the consumption of computing resources, and improve the controllability and interpretability of abnormalities.

[0035] Figure 1 is a flowchart of a large model-based data warehousing method according to an embodiment of the present disclosure. As shown in Figure 1 , the method at least includes the following steps:

[0036] S110, analyzing the heterogeneous data source of the data warehousing task based on a large language model to generate executable logic code for processing the heterogeneous data source.

[0037] In this step, the large language model can be a deep learning model pre-trained on a large amount of text and code data, which has the ability to understand, reason and generate code. Heterogeneous data sources can be understood as multiple data collections with different formats, structures or sources, such as JSON files, comma-separated value (CSV) tables, tables in PDF documents, database snapshots or real-time log streams. Executable logic code refers to a program script that can be directly run by a computer to complete a specific data processing task, such as a Python script or an SQL query statement.

[0038] In specific implementation, first, a data warehousing task is received, which specifies the location (such as file path or API endpoint) of the heterogeneous data source to be processed. The system extracts the sample or metadata of the data source and submits it to the large language model together with the task instructions. The large language model analyzes these information, understands the internal structure of the data source and the goal of the data warehousing task, and generates one or more codes to execute the subsequent data processing flow.

[0039] For example, a data warehousing task requires processing a JSON file named product_reviews.json. The system will submit the first few records of the file as a sample to the large language model and indicate the task goal as "parse data and prepare for warehousing". After analysis, the large language model will generate a Python script that uses the json library to read and parse the structure of the file.

[0040] S120, executing the logic code in a controlled execution environment to process the heterogeneous data sources.

[0041] The controlled execution environment can be a standardized running environment isolated from the production system, such as a Docker container, a virtual machine, or a dedicated code execution server (also known as a sandbox). The purpose is to ensure that the execution of the code is safe, reproducible, and does not cause unintended impact on external systems.

[0042] In implementation, the logic code generated in the previous step is automatically distributed to the controlled execution environment. The environment is pre-installed with all necessary dependencies, such as a specific version of the programming language interpreter, data processing libraries, etc. After receiving the logic code, the environment loads the specified data source for processing.

[0043] For example, the aforementioned Python script generated for processing product_reviews.json is sent to a Docker container pre-configured with Python 3.9 and Pandas library. The execution engine inside the container starts the script, and the script begins to read and process the JSON file.

[0044] S130, monitoring the execution process of the logic code to obtain feedback information representing the state of the execution process.

[0045] The feedback information can be understood as structured data collected during code execution, reflecting its running status and results.

[0046] In implementation, within the controlled execution environment, there is a monitoring agent running in parallel with the code execution process. The agent captures standard output, standard error, program logs, and records key performance indicators such as execution time, CPU usage, memory consumption peak, etc. After execution, the monitoring agent also checks whether the expected output (such as generated intermediate files or new data in the database) exists and meets the format requirements. All this information is summarized into a structured report, i.e. feedback information.

[0047] For example, when executing the script for processing the JSON file, the monitoring agent records that the script runs for 15 seconds, the memory usage peak is 512MB, and finds an error message "KeyError" in the log, indicating that the code fails when accessing a non-existent key. These information are packaged into feedback information.

[0048] S140, according to the feedback information, optimizing the generation process of the logic code to obtain logic code for data warehousing that meets the preset requirements.

[0049] The preset requirements can be a set of standards for judging whether the data warehousing process is successful, such as error-free code execution, execution time less than 30 seconds, score on execution result reaching a threshold, number of output data consistent with source data, etc.

[0050] In a specific implementation, a central scheduling or optimization module receives the feedback information. If the feedback information indicates that the execution is successful and all indicators are within the preset requirements, the current generated logical code is considered qualified. If the feedback information contains errors or performance does not meet the standards, the optimization module will re-submit these negative feedbacks, together with the original code and task requirements, to the large language model and give correction instructions, such as "code fails due to KeyError, please check the JSON structure and correct the code". The large language model will adjust according to the new instructions to generate a new version of the logical code, which will again enter the execution and monitoring process, forming a closed loop until the generated code meets all the preset requirements.

[0051] According to the scheme of the embodiments of the present disclosure, by constructing a closed-loop process of "code generation-execution-monitoring-optimization", the intelligent generation capability of the large language model is combined with the engineered execution verification, solving the problem of low efficiency and easy error of traditional manual script writing, and avoiding the defects of uncontrollable performance and difficult tracking of exceptions when pure large model directly processes data, thereby realizing an automatic, self-correcting and efficient and reliable data warehousing method.

[0052] In a possible implementation, S110 analyzes the heterogeneous data sources of the data warehousing task based on the large language model to generate executable logical code for processing the heterogeneous data sources, further comprising the steps of:

[0053] S111, using the large language model, analyzing the heterogeneous data sources involved in the data warehousing task to identify the data patterns of different data sources.

[0054] The data pattern (Data Schema) can be understood as a formalized description of the data structure, which defines the way of data organization, including the name of data field, data type, relationship between fields and constraints that the data should follow, which can also be referred to as Schema.

[0055] In the embodiments of the present disclosure, the core task of the large language model is to identify or infer a schema. The large language model can identify fields, types, semantics (time / amount / ID / place name / indicator), and inter-table relationships (primary-foreign key, entity ownership) from sample data. For example, for a CSV file, it analyzes whether the first line is a column title and infers the type (integer, floating point, string, or date) according to the values in the column; for a JSON file, it recursively parses all key-value pairs, identifies nested objects and arrays, and thus constructs a complete tree structure.

[0056] For example, for a JSON object containing "user_id": 101, "is_active": true, the large language model infers that the user_id field is of integer (Integer) type and the is_active field is of Boolean type.

[0057] S112, based on the data mode, generating extraction code for structured data extraction from the data source.

[0058] The extraction code refers to program code specially used for reading data from the original data source and converting it into a structured representation in memory (such as a data frame or a list of objects) according to the known data mode.

[0059] In specific implementation, the data mode inferred in the previous step is used as context information, which is sent to the large language model together with the instruction for generating the extraction code. The model selects the most suitable library and method to write the code according to the accurate schema.

[0060] S113, generating filtering code for conditionally filtering the data extraction according to the business requirements of the data loading task.

[0061] The business requirement can be a specific processing requirement for data proposed by the user in the form of natural language, etc. The filtering code is program code used to perform filtering, conversion, or aggregation operations on the extracted data.

[0062] In specific implementation, the user's business requirement (for example, "only keep records with sales exceeding 1000 yuan") and the data mode can be provided to the large language model together. The model understands the semantics of the business requirement and converts it into code for operating on the data.

[0063] For example, for data containing an Amount column and receiving the business requirement "sales exceeding 1000 yuan", the large language model adds a line of code in the Python script, such as filtered_data = extracted_data[extracted_data['Amount'] > 1000].

[0064] S114, in combination with the type of target database, generating a database building code for storing the screened data into the target database.

[0065] The database building code refers to the data definition language (DDL) statements such as SQL used to create table structures in the target database, and the program code such as DML statements used to write data into these tables.

[0066] In specific implementation, the system provides information of the target database and the final schema of the screened data, and instructs the large language model to generate the corresponding database building code. The model will generate compatible database building and data insertion statements according to the SQL dialect differences of different databases.

[0067] For example, if the target database is PostgreSQL, the large language model will generate a CREATE TABLE... statement, use the data types specific to PostgreSQL (such as SERIAL PRIMARY KEY), and generate an INSERT INTO... statement or COPY command to batch insert data.

[0068] According to the extraction code, the screening code and the database building code, an executable logic code for processing heterogeneous data sources is obtained.

[0069] In this embodiment, the codes generated in the above steps S112, S113 and S114 are logically combined or concatenated to form an end-to-end, complete and executable logic code. The logic code can independently complete the whole process from reading data from the data source, on-demand processing, to finally storing into the target database.

[0070] According to the scheme of the embodiments of the present disclosure, by dividing the code generation process into four clear stages of schema recognition, extraction, screening and database building, the large language model has clear and focused task targets and sufficient context information at each step. This structured generation method greatly improves the accuracy, completeness and adaptability to complex business requirements of the final generated code compared to generating all codes at once, and solves the problem that a single large model instruction cannot cover complex multi-stage processes.

[0071] In one possible implementation, S111 uses a large language model to analyze the heterogeneous data sources involved in the data loading task to identify the data schema of different data sources, specifically including:

[0072] The multi-modal large model is used to infer data patterns from the heterogeneous data source in combination with a preset structure perception prompt word, to obtain candidate pattern metadata.

[0073] In the embodiments of the present disclosure, the multi-modal large model refers to a large language model capable of simultaneously processing and understanding multiple different types of data (such as text, images, and audio). The structure perception prompt word (Prompt) can be a specially designed instruction text for guiding the large language model to focus on and extract structured information in the data. The candidate pattern metadata is an intermediate data format generated by the model according to the inference, which is structured and describes the data pattern, and is usually in JSON or XML format.

[0074] In specific implementation, when the system encounters a heterogeneous data source in a non-pure text format, such as a PDF financial report containing a table or a web page screenshot, the system calls a multi-modal large model. At the same time, a structure perception prompt word is used, for example: "You are a data analysis expert. Please analyze the following input (which can be a picture or a document) and identify all the data presented in the form of a table. For each table, output its data pattern in JSON format, including the name of each column (name), the inferred data type, and a data sample of the column." After receiving the input and the prompt word, the model uses its visual understanding ability to locate and parse the table, and generates the candidate pattern metadata in JSON format according to the instructions.

[0075] According to the scheme of the embodiments of the present disclosure, by using a multi-modal large model, the types of data sources that the method can handle are greatly expanded, from traditional structured and semi-structured text files to images and complex documents containing structured information, solving the problem that the prior art cannot automatically process non-structured data sources such as PDF and screenshots. At the same time, in combination with the structure perception prompt word, the model can be guided to perform more accurate and more standardized pattern inference, improving the accuracy and automation level of pattern recognition.

[0076] In one possible implementation, S112 generates extraction code for structured data extraction from the data source based on the data pattern, specifically including:

[0077] According to the candidate pattern metadata, an instruction fine-tuning or retrieval enhancement generation technology is used to generate a data extraction script for extracting data from the heterogeneous data source.

[0078] In the embodiments of the present disclosure, instruction fine-tuning is a technique of training a pre-trained large language model on a large number of "instruction-response" format data to make it better follow user instructions to complete specific tasks. Retrieval-augmented generation (RAG) is a technique of combining the generation capability of a large language model with the retrieval capability of an external knowledge base. The model will retrieve relevant information from the knowledge base as a reference before generating a response.

[0079] In a specific implementation, to improve the quality and pertinence of the extracted code, the large language model can be pre-trained through instruction fine-tuning. The training data can include a large number of instruction pairs of "given a data mode and data sample, write a Python code to parse it". When generating the extraction script, the candidate mode metadata is taken as the context, and an explicit instruction is given, for example: "According to the provided mode metadata, write a Python script to read the source file using the Pandas library and ensure that the data type of each column matches the mode."

[0080] Alternatively, the RAG technique can be used to retrieve the most relevant code snippets from a code library containing code examples according to the characteristics of the candidate mode metadata (such as "nested JSON" and "complex date format"), and then provide these snippets together with the mode metadata to the large language model to generate the final data extraction script by referring to these best practices.

[0081] According to the scheme of the embodiments of the present disclosure, by using instruction fine-tuning or retrieval-augmented generation, the large language model can generate more professional, reliable, and efficient data extraction code. Instruction fine-tuning enhances the model's ability to follow specific formats and requirements, while RAG introduces external, verified knowledge and best practices to the model, solving the problem of suboptimal or non-robust code generated by general large models when facing specific and complex data extraction scenarios.

[0082] In a possible implementation, S113 generates screening code for conditionally screening data extraction according to the business requirements of the data warehousing task, specifically including:

[0083] In combination with the business knowledge graph, the natural language requirements containing fuzzy business semantics in the business requirements of the data warehousing task are mapped to precise query statements.

[0084] In the embodiments of the present disclosure, the business knowledge graph can be a knowledge base in a graph structure, containing entities, concepts and their mutual relationships in a specific business field. The fuzzy business semantics refers to business terms appearing in natural language without accurate calculation definition, such as "high-value customers", "recently active users", etc.

[0085] In specific implementation, when the received business requirement contains fuzzy business semantics, directly submitting it to the large language model may lead to inaccurate or inconsistent results. Therefore, the business knowledge graph can be queried first. The knowledge graph has pre-defined accurate rules for these fuzzy terms. For example, the knowledge graph can store a rule that the definition of the entity "high-value customer" is "a customer with cumulative consumption amount greater than 10000 yuan and purchase times greater than or equal to 5 times in the past year". This accurate rule is used as key context information, which is provided to the large language model together with the user's natural language requirement and data pattern. After receiving this explicit and quantitative rule, the model can accurately translate it into code.

[0086] According to the scheme of the embodiments of the present disclosure, by introducing the business knowledge graph, the large language model is provided with the domain knowledge and accurate definition required for processing fuzzy business concepts, solving the problem that the general large language model lacks specific business background and cannot accurately understand and quantify fuzzy terms such as "high-value customers". This ensures that the business rules are consistently and accurately implemented at the code level, greatly improving the reliability of business logic processing.

[0087] In a possible implementation, S114 generates, in combination with the type of the target database, a database building code for storing the filtered data into the target database, specifically including:

[0088] S114a, in combination with the type of the target database, generates a data definition language or data manipulation language script adapted to the target database.

[0089] In the embodiments of the present disclosure, the data definition language (DDL) refers to SQL commands for defining and managing database objects (such as tables, indexes), such as CREATE TABLE, ALTER TABLE. The data manipulation language (DML) refers to SQL commands for operating data in the database, such as INSERT, UPDATE, DELETE.

[0090] In specific implementation, when generating the database building code, the large language model considers the differences in SQL syntax and data types of different database systems (such as MySQL, PostgreSQL, Oracle). According to the type of the target database, the model generates DDL and DML scripts that conform to the specific database.

[0091] S114b, optimize the success rate of the script using reinforcement learning, and / or verify the stability of the script using automated regression testing.

[0092] Reinforcement learning can be understood as a training mechanism, in which the model learns how to make optimal decisions (generate optimal scripts) by interacting with the environment (executing scripts and returning results). Automated regression testing is a set of pre-written test cases used to systematically check whether code changes have introduced new bugs or caused existing functionality to degrade.

[0093] In specific implementation, to improve the quality of generated scripts, one of the following two methods or a combination thereof can be used:

[0094] Method one (optimization): use reinforcement learning. Use a large language model as an agent to generate DDL / DML scripts. Execute the script in a controlled execution environment, and if it is successfully executed, give the model a positive reward; if the execution fails (e.g., due to syntax errors or constraint conflicts), give a negative reward. Through a large number of iterative training, the model will gradually learn to generate scripts with higher success rates.

[0095] Method two (verification): use automated regression testing. That is, verify it through a test set containing various boundary conditions and historical failure cases. Each time a new database building script is generated, the test set will be automatically run, for example, to test whether the script can correctly handle null value insertion, data type overflow, foreign key constraints, etc. Only scripts that pass all regression tests are considered stable and qualified.

[0096] For example, a certain INSERT script generated by the model fails to handle strings with special characters. Through reinforcement learning, the model will receive a negative reward and tend to add string escaping logic in subsequent generation. Or, through automated regression testing, a use case that tests special characters will fail, marking the script as unstable and requiring regeneration.

[0097] According to the scheme of the embodiments of the present disclosure, a mechanism for ensuring the quality of database building code is established by using reinforcement learning or automated regression testing. This solves the problem that a general large model may generate scripts that are syntactically correct but cannot run stably under specific data or constraint conditions, significantly improving the success rate and robustness of the last link of data warehousing.

[0098] In one possible implementation, S130 monitors the execution process of the logic code to obtain feedback information representing the state of the execution process, further comprising the following steps:

[0099] S131, automatically check the execution result of the logic code to determine whether the logic code meets the preset correctness or performance requirements.

[0100] In the embodiments of the present disclosure, the automated verification can be a checking procedure automatically executed by the system without human intervention, for comparing the actual running result of the code with a set of expected standards.

[0101] In actual implementation, a series of checks are performed by a verification module after the execution of the logic code. The correctness check can include checking whether the number of output data is within the expected range, whether the primary key is unique, whether the foreign key relationship is satisfied, whether the numerical sum of the key field is consistent with the source data, etc. The performance requirement check can include judging whether the total execution time exceeds the set threshold (such as 60 seconds), whether the memory peak value exceeds the resource limit, etc.

[0102] For example, the source data of a data warehousing task has 1000 rows. After the execution of the logic code, the automated verification module queries the new table in the target database, and if it is found that the table only has 950 rows of data, it is judged that the verification fails, because it does not meet the correctness requirement of consistent data row number.

[0103] S132, taking the result of the automated verification as feedback information.

[0104] In actual implementation, the automated verification module formats all the check items and their results (success, failure, specific difference value) into a structured report. This report will be the core part of the feedback information and be sent to the optimization module.

[0105] According to the scheme of the embodiments of the present disclosure, by introducing the automated verification mechanism, the evaluation of the code execution result is improved from simple success / failure to deep verification of data content correctness and performance indicators. This makes the feedback information more accurate and valuable, and can reveal logic errors (such as data loss) that cannot be found by running logs, thereby providing more accurate guidance for subsequent closed-loop optimization, and solving the problem that the code can run to completion but the data warehousing result is incorrect.

[0106] In a possible implementation, S131 performs automated verification on the execution result of the logic code, specifically including:

[0107] S131a, testing the logic code by a preset test suite.

[0108] The test suite can be a set of pre-defined and repeatable test cases, aiming to systematically verify various aspects of the software. The test suite includes unit testing, performance testing, and regression testing.

[0109] When a logic code is generated, the automated validation module will invoke one or more of these test suites to conduct a comprehensive check on the code and its output.

[0110] Unit test suite to verify the core transformation logic in the logic code.

[0111] In this context, unit tests can be independent tests on key functions or code blocks in the logic code that are responsible for data cleaning, transformation, calculation, etc.

[0112] For example, if the logic code contains a function to convert a currency string (e.g. "$1,234.56") to a floating point number, unit tests will invoke this function with various inputs (including normal values, empty values, format incorrect strings) and assert its return values to ensure that this core transformation logic is correct.

[0113] Performance test suite to evaluate the execution latency or resource usage of the logic code on sample data.

[0114] Performance tests can be running the entire logic code on a subset of representative data (sample data) and measuring its performance metrics.

[0115] For example, the system will randomly select 10,000 rows from a million rows of source data as sample data, then run the complete logic code, and record the time it takes to process these 10,000 rows of data and the memory it consumes. If the time exceeds the pre-set performance baseline, the performance test fails.

[0116] Regression test suite to compare the execution results with historical versions to verify consistency.

[0117] Regression tests can be running a new version of the logic code on a fixed input data set and comparing its output with a known, correct standard output.

[0118] For example, the system has a standard 100-line test data set. When a new logic code is generated, it will process this data set and compare the final table data it generates with the previously verified, correct table data line by line and field by field. If there is any difference, the regression test fails, indicating that the new code may have introduced unintended changes.

[0119] According to the scheme of the embodiment of the present disclosure, a systematic and multi-dimensional automatic verification method is established by defining a comprehensive test suite including unit testing, performance testing and regression testing. This not only verifies the logical correctness of the code, but also evaluates the performance and consistency between versions, solves the problem that simple result checking may miss deep logical errors, performance degradation or unintended side effects, and greatly improves the depth and breadth of automatic verification.

[0120] In a possible implementation, the feedback information includes an execution exception detected during execution of the logic code.

[0121] The execution exception can be defined as any event that occurs during code execution, which causes the program to fail to complete normally or produce incorrect results. The execution exception includes but is not limited to:

[0122] 1. Syntax error or runtime exception.

[0123] This refers to the problem of the code itself. Syntax error is that the code does not conform to the programming language specification and will fail before execution. Runtime exception is an error that occurs during execution, such as division by zero, type mismatch, null pointer reference, etc.

[0124] For example, the monitoring agent captures the ZeroDivisionError or TypeError thrown by the Python interpreter and records it as an execution exception in the feedback information.

[0125] 2. Data inconsistency of execution result.

[0126] This refers to the fact that the code has successfully run, but the data it produces is logically contradictory or does not meet expectations.

[0127] For example, the automated verification module finds that in the data table after warehousing, the total order amount is not equal to the total amount of all order line items, which is an exception of data inconsistency.

[0128] 3. Performance bottleneck of execution process.

[0129] This refers to the fact that the code runs inefficiently, exceeding the preset resource or time limit.

[0130] For example, performance testing shows that processing 10,000 rows of sample data takes more than 5 minutes, which is identified as a performance bottleneck exception.

[0131] 4. Execution result does not meet the preset business constraints.

[0132] This refers to the fact that the execution result violates the pre-defined business rules.

[0133] For example, one business constraint specifies that the user's age field must be between 0 and 120. Automated checks find records with age -1 or 200 in the ingested data, which are exceptions that do not meet the preset business constraints.

[0134] According to the scheme of the embodiments of the present disclosure, by classifying the execution exceptions in detail, the content of the feedback information is more rich and structured. Such classification enables the optimization module to more accurately diagnose the root cause of the problem: is the code itself defective (syntax / runtime exception), or is the data processing logic incorrect (data inconsistency / business constraint inconsistency), or is the execution efficiency insufficient (performance bottleneck). Such refined feedback provides a more explicit guidance direction for subsequent automated optimization, solving the problem that the general "failure" feedback information is insufficient to support effective automatic repair.

[0135] In one possible implementation, S140 optimizes the generation process of the logic code according to the feedback information, specifically including at least one of the following ways:

[0136] S141, when the feedback information indicates that the execution fails, adjusting the prompt word used to generate the logic code, and regenerating the logic code.

[0137] Adjusting the prompt word is a lightweight and fast optimization method.

[0138] In specific implementation, when the feedback information received by the optimization module is an error that can be directly understood by the model (such as a syntax error or a simple runtime exception), it will combine the original generation instruction (prompt word) with the error information to form a new and more instructive prompt word.

[0139] For example, the original prompt word is "write Python code to read CSV", and the feedback error after execution is FileNotFoundError. The optimization module will generate a new prompt word: "write Python code to read CSV, note: file path may be incorrect, please add file existence check logic." Then send the new prompt word to the large language model to generate the corrected code.

[0140] S142, taking the feedback information as a training sample, periodically incrementally training or reinforcement learning the large language model to optimize the code generation capability of the large language model.

[0141] This is a more in-depth and systematic optimization method.

[0142] In a specific implementation, all cases of failed execution (including failed logic code, task requirements, feedback information) and subsequent corrected successful code can be stored as a high-quality "problem-solution" data pair in a special training sample library. When the number of samples in this library reaches a certain number or at a preset period (such as once a week), the training process is started, and these new sample pairs are used to fine-tune or reinforce the large language model. This process can be understood as a "data flywheel", the system continuously accumulates experience in solving problems, and uses these experiences to improve the core capabilities of the model.

[0143] For example, the system encounters cases of failure in processing a specific date format (such as MM-DD-YY) multiple times, and stores these cases as training samples. After periodic training, the large language model learns how to better recognize and process this unusual date format, and in the future, it can generate correct code in one go when encountering similar situations.

[0144] According to the scheme of the embodiments of the present disclosure, by providing two optimization methods of adjusting prompt words and incremental training, a closed-loop optimization mechanism with both rapid response and long-term capability evolution is established. Adjusting prompt words can immediately fix simple and explicit errors, while periodic incremental training can fundamentally enable the model to learn and master the ability to solve complex or repeatedly occurring problems, solving the contradiction between immediate repair efficiency and long-term performance improvement of a single optimization method.

[0145] In a possible implementation, the data warehousing method provided by the embodiments of the present disclosure further includes the following steps:

[0146] The manual intervention step is triggered when at least one of the following preset conditions is met:

[0147] 1. The optimization step fails continuously for a preset number of times and fails to generate logic code that meets the requirements.

[0148] This condition is used to handle difficult problems that the machine cannot solve by itself.

[0149] In a specific implementation, the system sets a counter for each automated optimization cycle of a data warehousing task, for example, the maximum number of attempts is 5. If the system still cannot output logic code that passes all checks after 5 consecutive attempts (generate new code → execute → fail → optimize → generate again...), the system determines that it is in a dilemma and automatically triggers manual intervention.

[0150] For example, a logical code repeatedly fails due to a complex data consistency problem involving multiple table joins, and the large language model tries multiple fixes without success. After the 5th failure, the system marks the task as “needs human handling” and sends a notification to the data engineer’s ticket system with all the history of attempts.

[0151] 2. The complexity or risk level of the generated logical code is assessed and determined to be higher than a preset threshold.

[0152] This condition is used to handle cases where the code can successfully execute, but there is potential risk in it itself.

[0153] In implementation, whenever a new logical code is generated, its complexity indicators and risk features (such as performing data deletion operations or data modification operations without transaction wrapping) are calculated. If these indicators exceed the preset safety threshold, even if the code is successful in testing, the system will trigger human intervention and require a human engineer to conduct a safety review.

[0154] According to the scheme of the embodiments of the present disclosure, by introducing an explicit human intervention triggering condition, a key safety and reliability guarantee is added to the fully automated process. This solves the problem that a purely automated system may waste resources in an infinite loop on difficult problems, or may deploy code with potential risks, and achieves an effective balance between automation efficiency and human supervision reliability.

[0155] In a possible implementation, the data warehousing method provided by the embodiments of the present disclosure further includes the steps of:

[0156] S210. Calculate a unique state signature for each version of the generated logical code and record it.

[0157] The unique state signature can be a string that uniquely identifies a piece of code content, such as a value obtained by hashing the code text (such as MD5 or SHA-256).

[0158] In implementation, whenever a new version of logical code is generated by the closed-loop optimization step, the system calculates the hash value of the code text and stores this hash value together with the version number in the history record of the task.

[0159] S220. In the case where the state signature of the newly generated logical code is detected to be repeated with the state signature in the history record, it is determined that optimization shock occurs and the human intervention step is triggered.

[0160] Optimization shock refers to a phenomenon in which the optimization system switches back and forth between several different error states and cannot converge to the correct solution.

[0161] In a specific implementation, after calculating the state signature of the new version of the code, the system checks whether the signature already exists in the history. If it does, it means that the optimization process has fallen into a loop.

[0162] For example, the system generates three versions of the code with signatures A, B, and C in sequence, all of which fail to execute. After the fourth optimization, the signature of the generated code becomes B again. At this time, the system detects that the signature B already exists in the history, and determines that an optimization shock has occurred (for example, the model switches between two imperfect solutions), and immediately triggers the manual intervention step to report this loop to the engineer.

[0163] According to the scheme of the embodiments of the present disclosure, by calculating and comparing the state signatures of the code, an accurate and efficient optimization shock detection mechanism is realized. This solves the problem that the automated optimization process may fall into an infinite loop and cannot be self-aware, and can actively identify the state of the model being stuck and introduce human intelligence in a timely manner to break the deadlock, improving the robustness and problem solving efficiency of the entire system.

[0164] In a possible implementation, the data warehousing method provided by the embodiments of the present disclosure further includes the steps of:

[0165] S310, after the logical code passes the automated verification, the logical code is automatically deployed to an online execution flow for processing actual business data.

[0166] The online execution flow can be a stable and efficient data processing pipeline or scheduling system designed for processing real and massive data in a production environment.

[0167] In a specific implementation, when a logical code successfully passes all the verifications of the test suite, the system will mark it as "production ready". Then, the fully verified logical code and its dependent configurations will be packaged and pushed to a production-level task scheduling system. The scheduling system will automatically execute the code in the online environment according to the preset plan or trigger condition to process real business data.

[0168] According to the scheme of the embodiments of the present disclosure, by seamlessly connecting the automatic deployment step after the automated verification is successful, an end-to-end automated (CI / CD) process from code generation, verification to online is constructed. This solves the problem that the development, testing and deployment links in the traditional process are artificially fragmented and need to be manually handed over and deployed, greatly shortening the period from requirement submission to function online, and ensuring that the code deployed to the online environment is systematically verified, improving the stability of the production environment and the timeliness of data processing.

[0169] The embodiment provides a complete data intelligent warehousing process combined with the foregoing embodiments, which can be referred to Figure 4 for understanding. The embodiment describes an end-to-end workflow from receiving raw data to finally completing data warehousing. As shown in Figure 4 the warehouse process includes:

[0170] S410, pattern recognition and requirement modeling.

[0171] This step is at the initial stage of the whole process. Pattern recognition and requirement modeling can be understood as the process of comprehensive analysis and understanding of the system to the initial input raw data and Schema, aiming to convert unstructured data and fuzzy requirements into structured models that can be processed by machines.

[0172] In specific implementation, the system first receives a data warehousing task, which contains raw data, such as a PDF file named daily_sales_report.pdf, and a business requirement, such as "extract table data in daily sales report and store sales amount by region after aggregation". The system will call the multi-modal large language model described in embodiment 3 to parse the PDF file, use its visual and text understanding ability to identify the table structure in it, and infer its data pattern. At the same time, the system will parse the business requirement in natural language form and decompose it into a series of structured requirement specifications such as "data extraction", "group aggregation" and "data storage".

[0173] S420, execute intelligent code generation.

[0174] The data pattern and requirement specification modeled in the previous step are submitted to the large language model as context and instructions. The large language model will integrate these information to generate a complete executable logic code that can execute end-to-end tasks.

[0175] S430, perform verification in CodeServer.

[0176] CodeServer can be understood as the controlled execution environment described in the foregoing embodiments. The purpose of this step is to verify the correctness, performance and stability of the generated code before deployment to the production environment.

[0177] In specific implementation, the Python script generated in S420 is sent to CodeServer. The system will automatically run the test suite described in the foregoing embodiments. Unit testing and performance testing are performed, and regression testing is performed with the results of the similar report processed by the last stable version to check the consistency of the results.

[0178] S440, judge the result of execution verification and execute branch process.

[0179] This step is a critical decision point in the process, determining whether to enter the optimization loop or deploy the process.

[0180] S441. When the results do not meet expectations, perform feedback and adaptive optimization.

[0181] In practice, if the S430 validation fails—for example, due to a script error or inconsistent calculation results—the system will send back feedback information. The optimization module will adjust the generation strategy based on this feedback, for example, by modifying prompt words or fine-tuning the model using accumulated failure samples. Then, it will instruct the large language model to re-execute the S420 intelligent code generation steps. This "generation-validation-feedback-regeneration" cycle will continue until the generated code passes validation.

[0182] For example, the initial version of the script encountered an error while processing a table containing merged cells. After this error was reported, the system optimized the prompt, instructing the model to "pay attention to handling merged cells in the PDF," and the model subsequently generated a new version of the script that used more robust parsing logic.

[0183] S442. When the results meet expectations, perform automated deployment and manual code review (CR).

[0184] In practice, once the S430 verification is successful, the code is considered functionally correct. At this point, the system enters the deployment phase. Simultaneously, the system performs a risk assessment on the code. If the code has low complexity and does not involve high-risk operations, it can be automatically approved and proceed to the next step. If the code is complex or contains sensitive operations, the system will automatically create a code review (CR) task and notify the relevant engineers for manual review and approval.

[0185] S450: Deploy the verified code to the code execution flow.

[0186] In practice, code approved via S442 is automatically deployed to the production environment's task scheduling system. Thereafter, this code execution flow processes the daily new data stably and efficiently. This execution flow has the following characteristics:

[0187] Version control: Each deployed code version is recorded in a version control system (such as Git). If a problem occurs online, it can be rolled back to the previous stable version.

[0188] Anomaly detection and fault tolerance: Production-grade monitoring systems monitor the health of the code execution flow in real time. If a task fails due to transient issues such as network fluctuations, the system will automatically retry. If a serious anomaly such as a sudden change in the data source format is detected, the system will immediately pause the task and issue an alert.

[0189] Manual intervention capability: Operations engineers can view the status of the execution flow at any time through the console, and manually pause, rerun or terminate any data entry task as needed.

[0190] Efficient execution and database construction completion: The code execution flow runs on computing resources optimized for large-scale data processing, ensuring that the data entry task can be completed efficiently and on time.

[0191] According to the solution of this disclosure, by organically integrating steps such as pattern recognition, intelligent generation, execution verification, feedback optimization, and automatic deployment into an end-to-end automated workflow, and equipping the final production execution flow with engineering capabilities such as versioning, fault tolerance, and manual intervention, this solution not only realizes the intelligent processing of a single data entry task, but also builds a complete intelligent data entry system with self-evolution capabilities and production-grade reliability, systematically solving the challenges of efficiency, accuracy, and stability when processing large-scale, heterogeneous data sources.

[0192] Figure 5 This is a schematic diagram of a system architecture for intelligent data entry according to this embodiment. This architecture can be understood as a specific systematic implementation of the methods described in the foregoing embodiments. Figure 5 As shown, the various functional modules and the data and control flows between them include:

[0193] S510: Receives data from multiple sources and stores it in a landing area.

[0194] In this step, the system's entry point is multi-source data. These data sources can take various forms, such as file or object storage data received via batch or streaming processing. They can also be incremental change logs obtained from the business database using Change Data Capture (CDC) technology. Furthermore, they can be event data pushed from external systems via API interfaces or message queues.

[0195] A landing zone can be a centralized storage area used to temporarily store raw, unprocessed data, such as object storage in the cloud or a distributed file system. Its function is to aggregate data from all sources as a unified data source for all subsequent processing flows.

[0196] In practice, the system configures multiple data acquisition agents, each connecting to different data sources. For example, a stream processing job continuously consumes event data from a Kafka message queue and writes it to the landing zone of the data lake in Raw format; meanwhile, another batch processing job synchronizes incremental files from an FTP server daily.

[0197] S520, Schema recognition, structured extraction, and conditional filtering extraction are performed on the raw data, and the analysis results are input into the prompt center.

[0198] This step is the core link of data understanding and requirement transformation.

[0199] The prompt center (Prompt Hub) can be a central module dedicated to managing, building, and versioning prompts. It does not directly generate code, but is responsible for "preparing" high-quality, context-rich instructions for large language models.

[0200] In specific implementation, the first three stages of the data processing pipeline: original Schema recognition, structured extraction, and conditional and filtering extraction, are executed in sequence. They analyze the raw data in the landing area and send the analysis results, i.e. the inferred data patterns, the logic of fields to be extracted, and the filtering conditions to be applied, as structured information to the prompt center. After receiving this information, the prompt center will assemble it into a complete and detailed prompt according to the preset template. At the same time, the prompt center also interacts with the model registry (Model Registry) and the supervised fine-tuning (Supervised Fine-Tuning, SFT) example library. The model registry is used to store and manage different versions or fine-tuned large language models, and the prompt center selects the most suitable model according to the task type. The SFT example library stores a large number of high-quality "instruction-code" examples, and the prompt center can select a few examples most similar to the current task and add them to the prompt to further improve the accuracy of code generation.

[0201] S530, In the intelligent control plane, the generated code package is executed, evaluated, reviewed, and optimized.

[0202] The intelligent control plane is the core part of the system, which is a closed-loop control system for code generation, verification, and optimization.

[0203] In specific implementation, the process is iterated in the intelligent control plane until qualified code is produced.

[0204] S531, Build a library package and send it to the execution sandbox.

[0205] The large language model generates logical code containing DDL / DML operations based on the prompt provided by the prompt center, and these codes are packaged into a library package. The package is then sent to the CodeServer execution sandbox for trial running of the code on sample data.

[0206] S532, Perform automated evaluation and store feedback in the warehouse.

[0207] S533, model optimization using feedback information.

[0208] The data in the feedback warehouse has two utilization paths: first, high-quality failed cases and corrected successful cases are filtered out and sent to the SFT example library for future supervised fine-tuning of the model. Second, the execution result (success or failure) can be quantified as a reward signal (e.g., +1 for success and -1 for failure) and sent to a CodeServer-based reinforcement learning (RL) module to adjust and optimize the code generation strategy of the model.

[0209] S534, automatic or manual review through CR Gate.

[0210] Code packages that pass the automatic evaluation enter the CR Gate (Code Review Gate). The CR Gate first performs a series of automated checks, such as code specifications, security scans, etc. If the automated checks pass and the code complexity is low, it can be directly passed. If the automated checks fail or the code logic is complex, the system will trigger a human-machine co-review process and push the code to relevant engineers for manual review.

[0211] S535, decision-making based on review results.

[0212] If the code package is rejected in the CR Gate (whether automatically or manually), the reasons for rejection will be sent back to the prompt center as new feedback information to guide the next round of code generation, forming a cycle. If the code package is approved, it will leave the intelligent control plane and enter the final production execution stage.

[0213] S540, execute the reviewed DDL / DML to complete data loading.

[0214] This step is the final output link of the process.

[0215] In specific implementation, the qualified build package that passes all the checkpoints of the intelligent control plane will be sent to the DDL / DML execution engine in the production environment. The engine will parse the code in the package and execute it on the target data warehouse (Data Warehouse) or lake warehouse (Lakehouse) to complete the final table creation, data modeling, and data loading operations.

[0216] According to the scheme of the embodiment of the present disclosure, by constructing a system architecture composed of a prompt center, a model warehouse, an intelligent control plane and other professionalized modules, the intelligent process of data warehousing is improved from a pure methodology to an engineering system that can be implemented. The architecture realizes the automation of the code "generation-verification-optimization" closed loop through the intelligent control plane, and endows the system with the ability of continuous self-learning and evolution through the SFT example library and reinforcement learning mechanism, while the CR Gate ensures the quality and safety of the finally deployed code. This modular and closed-loop system design systematically solves the challenges of realizing reliable, efficient and evolving intelligent data warehousing in complex production environments.

[0217] Figure 6 is a sequence diagram of a large model-based data warehousing method according to an embodiment of the present disclosure. The embodiment takes a large model orchestrator (LLM Orchestrator) as the mainstay and shows the process of interaction with other modules.

[0218] In this embodiment, the system can contain a core large model orchestrator (LLM Orchestrator). The orchestrator can be understood as a central controller responsible for receiving tasks, scheduling resources, distributing instructions, collecting feedback, and driving the entire data warehousing process according to the preset logic.

[0219] The specific interaction steps of the process are as follows:

[0220] 1. Receive the trigger signal of new data or new demand.

[0221] When a new data warehousing task starts, for example, a new file arrives at the landing area or a user submits a new demand, a trigger signal is sent to the large model orchestrator to start the entire process.

[0222] 2. Select the stage-specific prompt words and examples.

[0223] The large model orchestrator first interacts with the prompt center (Prompt Hub). According to the stage of the current task (for example, is it the initial Schema identification stage, or the subsequent library building code generation stage), it will obtain the optimal prompt word template and related examples from the prompt center for constructing the instructions sent to the large language model.

[0224] 3. Select or switch the base model.

[0225] Next, the large model orchestrator queries the Model Registry. Depending on the complexity and type of the task, it selects the most suitable base model. For example, for a PDF file containing charts, it selects a multi-modal large model; for a regular database extraction task, it might select a model fine-tuned on SQL and Python.

[0226] (4) Generate candidate code and N variants.

[0227] The large model orchestrator uses the prompt and model obtained in the previous two steps to generate the logic code for processing the data. To conduct comparative testing and optimization, the orchestrator can instruct the model to generate a main candidate code and N code variants that are functionally identical but have slightly different implementations.

[0228] (5) Execute in execution sandbox and obtain results.

[0229] The large model orchestrator sends the generated candidate code (and its variants) to the CodeServer execution sandbox, instructs it to execute on the sample data, and waits to receive the execution results of the code, including the output data, logs, and performance metrics.

[0230] (6) Submit evaluation and scoring.

[0231] The large model orchestrator submits the execution results from the CodeServer to the evaluation and feedback warehouse. This module quantitatively scores the results, and the scoring dimensions can include accuracy (comparison with expected results), code coverage (test case coverage of code branches), stability (performance under boundary conditions), and cost (estimated computing resource consumption).

[0232] (7) Perform optimization iteration based on scoring results.

[0233] This step is an optional (alt) alternative process triggered when the score is below the preset threshold. After receiving the substandard score, the large model orchestrator starts an optimization cycle:

[0234] (8) Optimize prompts or replace models. The orchestrator will first try to interact with the prompt center to rewrite or optimize the prompts based on the failure reasons provided by the feedback warehouse. For example, add more specific constraints or introduce failed cases as negative examples. If the adjustment of the prompts is not effective, it can also decide to return to step (4) to replace a different model from the model warehouse or attach a specific tool chain (such as a professional database connection library) to the current model.

[0235] ⑼Carry error object generation and retry. When initiating a new round of code generation, the large model arranger will include the specific error information of the previous failure (which can be understood as an error object) in the new prompt, explicitly instructing the large language model to repair the error. Thereafter, the process returns to step ⑸, forming a retry loop with error information feedback.

[0236] ⑽Submit changes to CR Gate for review.

[0237] When the score result reaches the preset threshold, the process enters the optional (opt) deployment phase.

[0238] The large model arranger will package the verified logical code, evaluation report, and use cases that can be used to reproduce the verification process as review evidence and submit a formal change request to the CR Gate module.

[0239] ⑾Process the review results of CR Gate.

[0240] CR Gate will perform automated review and decide whether manual review is needed according to preset rules. Finally, CR Gate will return a "pass" or "reject" result. If rejected, the rejection opinion will drive the process back to the optimization iteration step.

[0241] ⑿Trigger automated deployment.

[0242] After receiving the "pass" instruction from CR Gate, the large model arranger will issue a deployment instruction to the CI / CD (Continuous Integration / Continuous Deployment) system. The instruction can include specific deployment strategies, such as gray release, blue-green deployment, to ensure smooth and safe online process.

[0243] ⒀Confirm online completion and monitoring.

[0244] The CI / CD system performs deployment operations and reports the final state (e.g., "online completion") to the large model arranger. After deployment, the system will start the preset rollback strategy, which can quickly roll back to the previous stable version if unexpected behavior is found in the production environment.

[0245] According to the scheme of the embodiments of the present disclosure, by introducing a centralized large model arranger, the intelligent method of data warehousing is implemented as a highly structured, event-driven sequential process. This process not only clearly defines the dynamic interaction between modules, but also generates code variants, multi-dimensional quantitative scoring, carries error objects for retry, submits review evidence, and uses gray release and other specific engineering methods to greatly improve the automation level, reliability, and traceability of the entire system, solving the core technical problems of process control and quality assurance in complex intelligent systems.

[0246] Figure 7 is a structural schematic diagram of a large model-based data warehousing device according to an embodiment of the present disclosure. As shown in Figure 7 , the data warehousing device 700 includes:

[0247] a code generation module 701 configured to analyze a heterogeneous data source of a data warehousing task based on a large language model to generate executable logic code for processing the heterogeneous data source;

[0248] a code execution module 702 configured to execute the logic code in a controlled execution environment to process the heterogeneous data source;

[0249] a feedback acquisition module 703 configured to monitor an execution process of the logic code to acquire feedback information representing a state of the execution process;

[0250] a closed-loop optimization module 704 configured to optimize a generation process of the logic code according to the feedback information to obtain logic code for data warehousing that meets preset requirements.

[0251] In a possible implementation, the code generation module 701 is configured to:

[0252] analyze the heterogeneous data source involved in the data warehousing task using a large language model to identify data patterns of different data sources;

[0253] generate extraction code for structured data extraction from the data source based on the data patterns;

[0254] generate filtering code for conditional filtering of the data extraction according to business requirements of the data warehousing task;

[0255] generate build database code for storing the filtered data into a target database in combination with a type of the target database;

[0256] obtain executable logic code for processing the heterogeneous data source according to the extraction code, the filtering code, and the build database code.

[0257] In a possible implementation, the code generation module 701 is configured to:

[0258] use a multi-modal large model to infer data patterns of the heterogeneous data source in combination with a preset structure perception prompt word to obtain candidate pattern metadata.

[0259] In a possible implementation, the code generation module 701 is configured to:

[0260] According to the candidate schema metadata, a data extraction script for extracting data from the heterogeneous data sources is generated by using instruction fine-tuning or retrieval enhancement generation technology.

[0261] In a possible implementation, the code generation module 701 is configured to:

[0262] In combination with the business knowledge graph, the natural language requirement containing fuzzy business semantics in the business requirement of the data warehousing task is mapped to an accurate query statement.

[0263] In a possible implementation, the code generation module 701 is configured to:

[0264] In combination with the type of the target database, a data definition language or data operation language script that is adapted to the target database is generated.

[0265] The success rate of the script is optimized by using reinforcement learning, and / or the stability of the script is verified by using automated regression testing.

[0266] In a possible implementation, the feedback acquisition module 703 is configured to:

[0267] The execution result of the logic code is automatically checked to determine whether the logic code meets a preset correctness or performance requirement.

[0268] The result of the automatic checking is taken as the feedback information.

[0269] In a possible implementation, the feedback acquisition module 703 is configured to:

[0270] The logic code is tested by using a preset test suite, and the test suite includes at least one of the following:

[0271] A unit test suite is configured to verify a core conversion logic in the logic code.

[0272] A performance test suite is configured to evaluate an execution delay or resource usage of the logic code on sample data.

[0273] A regression test suite is configured to compare the execution result with a result of a historical version to verify consistency.

[0274] In a possible implementation, the feedback information includes an execution exception detected in the execution process of the logic code, and the execution exception includes at least one of the following types:

[0275] A syntax error or a runtime exception;

[0276] Data inconsistency of the execution result;

[0277] Performance bottleneck of the execution process

[0278] The execution result does not conform to the preset business constraint.

[0279] In a possible implementation, the closed-loop optimization module 704 is configured to at least one of the following manners:

[0280] When the feedback information indicates that the execution fails, adjusting the prompt word used for generating the logic code, and re-generating the logic code;

[0281] Taking the feedback information as a training sample, periodically incrementally training or reinforcement learning the large language model to optimize the code generation capability of the large language model.

[0282] In a possible implementation, the apparatus further includes a trigger judgment module configured to:

[0283] Triggering the manual intervention step when at least one of the following preset conditions is met:

[0284] The step of optimizing the generation process of the logic code fails continuously for a preset number of times and fails to generate the logic code meeting the requirements.

[0285] By evaluation, determining that the complexity or risk level of the generated logic code is higher than a preset threshold.

[0286] In a possible implementation, the apparatus further includes a signature module configured to:

[0287] Calculating and recording a unique state signature for each version of the generated logic code.

[0288] If it is detected that the state signature of the newly generated logic code is repeated with the state signature in the historical record, it is determined that optimization shock occurs and the manual intervention step is triggered.

[0289] In a possible implementation, the apparatus further includes a deployment module configured to:

[0290] After the logic code passes the automatic verification, automatically deploying the logic code to an online execution flow for processing actual business data.

[0291] The specific functions and examples of the modules and sub-modules of the apparatus of the embodiments of the present disclosure are described in the related description of the corresponding steps in the above method embodiments, which will not be described here.

[0292] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0293] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0294] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0295] As shown in Figure 8 The device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0296] Various components in the device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; the storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0297] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the large model based data ingestion method. For example, in some embodiments, the large model based data ingestion method can be implemented as a computer software program tangibly embodied in a machine readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the large model based data ingestion method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the large model based data ingestion method by any other appropriate means, such as by means of firmware.

[0298] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0299] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine or server, or entirely on a remote machine or server.

[0300] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0301] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0302] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0303] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0304] It should be understood that the various forms of flow shown above can be re-ordered, steps added or removed, etc. For example, the steps recited in the present disclosure can be performed in parallel, in series, in a different order, etc., so long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0305] The above detailed description does not constitute a limitation of the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for data import based on a large model, comprising: Based on the analysis of heterogeneous data sources for data ingestion tasks using large language models, executable logic code is generated for processing the heterogeneous data sources. The logic code is executed in a controlled execution environment to process the heterogeneous data source; Monitor the execution process of the logic code to obtain feedback information characterizing the execution process status; Based on the feedback information, the generation process of the logic code is optimized to obtain logic code that meets preset requirements for data entry.

2. The method according to claim 1, wherein, The heterogeneous data source for the data import task based on the large language model analysis is used to generate executable logic code for processing the heterogeneous data source, including: Using a large language model, we analyze the heterogeneous data sources involved in the data import task to identify the data patterns of different data sources. Based on the data pattern, extraction code is generated for extracting structured data from the data source; Based on the business requirements of the data entry task, generate filtering code for conditional filtering of the extracted data; Based on the type of the target database, generate database creation code to store the filtered data into the target database; Based on the extraction code, the filtering code, and the database creation code, executable logic code for processing the heterogeneous data source is obtained.

3. The method according to claim 2, wherein, The method utilizes a large language model to analyze heterogeneous data sources involved in the data import task, in order to identify data patterns from different data sources, including: Using a multimodal large model, combined with preset structure-aware cue words, data pattern inference is performed on heterogeneous data sources to obtain candidate pattern metadata.

4. The method according to claim 2 or 3, wherein, The step of generating extraction code for extracting structured data from the data source based on the data pattern includes: Based on the candidate pattern metadata, a data extraction script is generated using instruction fine-tuning or retrieval enhancement generation techniques to extract data from the heterogeneous data source.

5. The method according to claim 2, wherein, The step of generating filtering code for conditional filtering of the data extraction based on the business requirements of the data import task includes: By combining business knowledge graphs, natural language requirements containing fuzzy business semantics in the business requirements of the data ingestion task are mapped into precise query statements.

6. The method according to claim 2, wherein, The step of generating database creation code to store the filtered data into the target database, based on the target database type, includes: Based on the type of the target database, generate a data definition language or data manipulation language script adapted to the target database; The success rate of the script is optimized using reinforcement learning, and / or the stability of the script is verified using automated regression testing.

7. The method according to claim 1, wherein, The monitoring of the execution process of the logic code to obtain feedback information characterizing the execution process status includes: The execution result of the logic code is automatically verified to determine whether the logic code meets the preset correctness or performance requirements; The result of the automated verification is used as the feedback information.

8. The method according to claim 7, wherein, The automated verification of the execution result of the logic code includes: The logic code is tested using a pre-defined test suite, which includes at least one of the following: A unit test suite is used to verify the core transformation logic in the logic code. A performance test suite is used to evaluate the execution latency or resource usage of the logic code on sampled data; A regression test suite is used to compare the execution results with those of historical versions to verify consistency.

9. The method according to claim 1, wherein, The feedback information includes execution exceptions detected during the execution of the logic code, and the execution exceptions include at least one of the following types: Syntax error or runtime exception; The execution results are inconsistent; Performance bottlenecks in the execution process; The execution result does not match the preset business constraints.

10. The method according to claim 1, wherein, The optimization of the logic code generation process based on the feedback information includes at least one of the following methods: When the feedback information indicates that the execution has failed, adjust the prompt words used to generate the logic code and regenerate the logic code; The feedback information is used as training samples to perform periodic incremental training or reinforcement learning on the large language model in order to optimize the code generation capability of the large language model.

11. The method according to claim 1 or 10, further comprising: The manual intervention step is triggered when at least one of the following preset conditions is met: The step of optimizing the generation process of the logic code fails consecutively within a preset number of times and fails to generate logic code that meets the requirements. The assessment determines that the complexity or risk level of the generated logic code exceeds a preset threshold.

12. The method according to claim 1 or 11, further comprising: Calculate and record a unique state signature for each version of the generated logic code; If the state signature of the newly generated logic code is found to be duplicated with the state signature in the history, an optimization oscillation is determined and a manual intervention step is triggered.

13. The method of claim 8, further comprising: After the logic code passes the automated verification, it is automatically deployed to the online execution flow to process actual business data.

14. A data import device based on a large model, comprising: The code generation module is used to analyze the heterogeneous data sources of the data entry task based on the large language model, so as to generate executable logic code for processing the heterogeneous data sources. A code execution module is used to execute the logic code in a controlled execution environment to process the heterogeneous data source; The feedback acquisition module is used to monitor the execution process of the logic code in order to obtain feedback information that characterizes the state of the execution process; The closed-loop optimization module is used to optimize the generation process of the logic code based on the feedback information, so as to obtain logic code that meets preset requirements for data entry.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-13.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-13.

Images