Power service data script automatic generation method, system, computer device, readable storage medium and program product

Abstract

The application relates to a power service data script automatic generation method, system, computer equipment, readable storage medium and program product. The method comprises the following steps: obtaining input power service data warehousing requirements and corresponding data samples; performing semantic analysis on the data warehousing requirements and the data samples, and identifying a parameter list for script generation; wherein the parameter list comprises a data ownership type, a writing mode and a business parameter of the data samples; determining a target processing mode according to the data ownership type and the writing mode; and generating a script file according to the target processing mode and the business parameter. The method can adapt to the processing of power data in different scenes and improve flexibility.

Description

Technical Field

[0001] This application relates to the field of power data processing technology, and in particular to a method, system, computer equipment, computer-readable storage medium, and computer program product for automatically generating power business data scripts. Background Technology

[0002] With the deepening of power market reform, power transaction data in various provinces are characterized by multiple sources, heterogeneity, and high frequency. To support unified data governance, data platforms typically use ETL code generation tools or SQL template engines to assist in development.

[0003] In traditional technologies, these tools typically generate fixed template table creation statements and extraction logic based on the field mapping and table name information input by the user, and the generated code often adopts a uniform structure.

[0004] However, traditional single templates cannot automatically adapt to the differences in different business scenarios, which means that developers still need to manually modify a lot of code, which is not only inefficient, but also cannot adapt to different scenarios. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, system, computer equipment, computer-readable storage medium, and computer program product for automatically generating power business data scripts that can be adapted to different scenarios, addressing the aforementioned technical issues.

[0006] Firstly, this application provides a method for automatically generating power business data scripts, applied to a large language model intelligent agent, the method comprising:

[0007] Obtain the input requirements for power business data entry and the corresponding data samples;

[0008] Semantic parsing is performed on the data entry requirements and the data samples to identify the parameter list used for script generation; wherein, the parameter list includes the data ownership type, writing method and business parameters of the data samples;

[0009] The target processing mode is determined based on the data ownership type and the writing method.

[0010] Based on the target processing mode and the business parameters, a script file is generated; wherein, the script file includes a first script for creating a temporary table, a second script for creating a target data storage table, and a third script for reading data to be processed from the temporary table, transforming it, and writing it into the target data storage table; the temporary table is used to read data to be processed from the data source storage system, and the target data storage table is used to store the transformed data to be processed.

[0011] In one embodiment, obtaining a data sample includes:

[0012] When the data entry request described in natural language does not include data samples, the potential storage path in the data source storage system is determined based on the geographic information and feature name information in the data entry request.

[0013] If the scan of the potential storage path is successful, the header and at least one row of data of the corresponding original data file are read from the partition directory under the potential storage path as the data sample.

[0014] When scanning the potential storage path fails, fuzzy matching is performed to determine candidate feature name information related to the feature name information corresponding to the regional information; based on the selection instruction and the candidate feature name information, the target storage path is determined; the header and at least one row of data of the corresponding original data file are read from the partition directory under the target storage path as the data sample.

[0015] In one embodiment, the data ownership type includes public type and private type; the writing method includes full write and incremental write; determining the target processing mode based on the data ownership type and the writing method includes:

[0016] If the data belongs to a public type and the writing method is full write, then the target processing mode is determined to be the first processing mode;

[0017] If the data belongs to a public type and the writing method is incremental writing, then the target processing mode is determined to be the second processing mode.

[0018] If the data belongs to a private type and the writing method is full write, then the target processing mode is determined to be the third processing mode.

[0019] If the data belongs to a private type and the writing method is incremental writing, then the target processing mode is determined to be the fourth processing mode.

[0020] The step of generating a script based on the target processing mode and the business parameters includes:

[0021] Based on one of the first processing mode, the second processing mode, the third processing mode, or the fourth processing mode, determine the script template corresponding to each processing mode;

[0022] The business parameters are filled into the script template to generate the script file.

[0023] In one embodiment, the script templates corresponding to the third processing mode and the fourth processing mode include a first sub-template; wherein, the first sub-template is a configuration template for partition attributes; the partition attributes are used to partition the target data storage table;

[0024] The templates corresponding to the second processing mode and the fourth processing mode include a second sub-template; wherein, the second sub-template is a configuration template for the data lake aggregation engine attribute; the data lake aggregation engine attribute configuration is used to deduplicate and merge the target data storage table.

[0025] In one embodiment, the business parameters include business fields; the semantic parsing of the data entry requirements and the data samples to identify the parameter list used for script generation includes:

[0026] Extract the first keyword related to the business domain and the second keyword related to the writing method from the data entry requirements;

[0027] The data attribution type is determined based on whether the value of the first keyword belongs to the target information domain;

[0028] The writing method is determined based on the writing category corresponding to the value of the second keyword;

[0029] The business fields are extracted from the data sample and matched with a preset field dictionary to determine the business fields.

[0030] In one embodiment, the business parameter further includes time granularity; the method further includes:

[0031] When the data sample contains a time point field, the time interval between adjacent data samples is calculated; based on the time interval, the time granularity is determined; the time granularity is used to ensure that the time format and range of the target data storage table conform to the electricity business settlement cycle.

[0032] In one embodiment, the field dictionary is updated in the following ways:

[0033] When the business field does not exist in the preset field dictionary, a second standard business field is determined according to the preset root abbreviation rules.

[0034] The business field and the corresponding second standard business field are stored in the field dictionary.

[0035] In one embodiment, after generating the script file, the method further includes:

[0036] The target data storage table is validated according to the data quality validation configuration file; wherein, the validation configuration file contains field-level validation rules, table-level validation rules, and special validation for the time type in the power industry; the field-level validation rules include: the field name, data type, whether null values ​​are allowed, and whether the field must exist for each field; the table-level validation rules include: whether additional columns are allowed, the column order validation range, and whether column order is ignored.

[0037] In one embodiment, after the target data storage table passes verification, the method further includes:

[0038] The workflow orchestration tool is invoked to process the script file, generating a workflow identifier and task node identifiers within the workflow identifier.

[0039] Secondly, this application also provides an automatic power business data script generation system, the system comprising:

[0040] The acquisition module is used to acquire the input power business data entry requirements and corresponding data samples;

[0041] The parsing module is used to perform semantic parsing on the data entry requirements and the data samples, and to identify the parameter list used for script generation; wherein, the parameter list includes the data ownership type, writing method and business parameters of the data samples;

[0042] The determination module is used to determine the target processing mode based on the data ownership type and the writing method;

[0043] The generation module is used to generate a script file based on the target processing mode and the business parameters; wherein, the script file includes a first script for creating a temporary table, a second script for creating a target data storage table, and a third script for reading data to be processed from the temporary table, transforming it, and writing it into the target data storage table; the temporary table is used to read data to be processed from the data source storage system, and the target data storage table is used to store the transformed data to be processed.

[0044] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0045] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0046] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0047] The aforementioned method, system, computer equipment, computer-readable storage medium, and computer program product for automatically generating power business data scripts first acquire the input power business data entry requirements and corresponding data samples; then, they perform semantic parsing on the data entry requirements and data samples to identify a parameter list for script generation; this parameter list includes the data ownership type, writing method, and business parameters of the data samples; secondly, based on the data ownership type and writing method, they determine the target processing mode; finally, based on the target processing mode and business parameters, they generate a script file; this script file includes a first script for creating a temporary table, a second script for creating a target data storage table, and a third script for reading data to be processed from the temporary table, transforming it, and writing it to the target data storage table; the temporary table is used to read data to be processed from the data source storage system, and the target data storage table is used to store the transformed data to be processed. By analyzing user input to obtain the data ownership type and writing method, and determining different modes corresponding to different power business scenarios based on the combination of data ownership type and writing method, different script files are generated according to different modes, and the script files are run to realize the data sample access requirements. This approach can adapt to the processing of power data in different scenarios and significantly improves efficiency compared to manual development. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating a method for automatically generating power business data scripts in one embodiment;

[0050] Figure 2 This is a flowchart illustrating the process of obtaining a data sample in one embodiment;

[0051] Figure 3 This is a flowchart illustrating the process of determining the target processing mode based on the data ownership type and writing method in one embodiment.

[0052] Figure 4 This is a flowchart illustrating the process of semantic parsing of data entry requirements and data samples to identify a list of parameters in one embodiment.

[0053] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] In one embodiment, such as Figure 1 As shown, a method for automatically generating power business data scripts is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps S102 to S108. Wherein:

[0056] Step S102: Obtain the input data entry requirements for the power business and the corresponding data samples.

[0057] The data samples can be data samples entered by the user when inputting the data entry requirements; or data samples obtained by analyzing the user's input data entry requirements.

[0058] Optionally, business personnel submit data entry requests to the AI ​​agent through an Integrated Development Environment (IDE). The terminal then receives the data entry request through the AI ​​agent, such as "{Business Domain} {Province} {Feature Name} {Full | Incremental} [Primary Key Field][Scheduled | Triggered][Scheduled Period], along with CSV-formatted field descriptions and sample data." Alternatively, a single sentence in natural language can describe the request: "I want to access Anhui's day-ahead clearing electricity price data," without providing sample data. The AI ​​agent matches the data entry request with the corresponding data entry requirement. Another example is listing multiple feature names at once, such as "Batch access Anhui [day-ahead clearing electricity price, real-time clearing electricity price, power generation, total output] full data."

[0059] Step S104: Perform semantic parsing on the data entry requirements and data samples to identify the parameter list used for script generation.

[0060] The parameter list includes the data ownership type, writing method, and business parameters of the data sample.

[0061] Optionally, the terminal uses an AI agent to perform semantic analysis on the data entry requirements and data samples to identify the following key information: data ownership type, writing method, and business parameters. The business parameters include province and pinyin conversion, data granularity, business fields, and conflict handling.

[0062] Data Attribution Type: Identify business domain attribution from natural language processing to determine the public / private attributes of data samples. When uncertain, the AI ​​agent proactively initiates interactive queries with the user to avoid erroneous assumptions.

[0063] Write method: Based on the full / incremental identifier, automatically select the INSERT OVERWRITE (full overwrite) or INSERTINTO (incremental append) write strategy and configure the corresponding data lake table attributes.

[0064] Province to Pinyin Conversion: Recognizes Chinese province names and converts them into standard Pinyin for use in data table naming and directory path generation.

[0065] Data granularity: Based on the time interval of the time field (hm) in the first sample data, the number of data points is automatically calculated. For example, after observing two sample values, "00:15" and "00:30", the calculated interval is 15 minutes, inferring a 96-point table (corresponding to a 15-minute settlement cycle in the electricity spot market), and automatically adding the "d96" mark to the table name. It also supports numeric time formats (such as 1, 2, 3, ..., 96), using the maximum numerical value to infer the number of points and automatically generating the corresponding time conversion SQL logic.

[0066] Business fields: In incremental data scenarios, the AI ​​agent automatically infers the primary key combination (time anchor + business identifier field) based on the field semantics. At the same time, it supports users to explicitly specify the primary key with Chinese field names (such as "date + time + transaction unit name"). The system automatically completes the mapping from Chinese to English field names through the field dictionary.

[0067] Conflict handling: When the type of the first sample data is inconsistent with that in the field dictionary (e.g., the dictionary is marked DECIMAL but the first sample looks like an integer, or the dictionary is marked STRING but the samples are all numbers), the system will follow the "dictionary first" principle and use the dictionary as the standard to maintain cross-province consistency, so as to avoid type deviation caused by accidental characteristics of the sample data.

[0068] Step S106: Determine the target processing mode based on the data ownership type and writing method.

[0069] Optionally, the terminal uses an AI agent to call a multi-mode code generation engine to select a processing mode as the target processing mode from standardized processing modes based on a two-dimensional combination of data ownership type and writing method.

[0070] Step S108: Generate a script file based on the target processing mode and business parameters.

[0071] The script file includes a first script for creating a temporary table, a second script for creating a target data storage table, and a third script for reading data to be processed from the temporary table, transforming it, and then writing it to the target data storage table. The temporary table is used to read the data to be processed from the data source storage system, and the target data storage table is used to store the transformed data to be processed. The data sample can be the same as or different from the data to be processed.

[0072] Optionally, the terminal uses an AI agent to invoke a multi-mode code generation engine to generate a script file based on the selected target processing mode. The script file consists of three parts. Running the script file will correspondingly create a temporary table DDL (external table, for reading OSS data), a target table DDL (which can be in Paimon format), and data processing SQL (that is, the SQL that reads the data to be processed from the temporary table, transforms it, and writes it to the target data storage table).

[0073] Temporary Table DDL: An external table definition based on Object Storage Service (OSS), including CSV parsing options (header recognition, schema enforcement, recursive directory lookup, file time filtering, etc.). The OSS path is constructed according to the specification as oss: / / ${input_bucket} / {province in Chinese} / {filename in Chinese} / .

[0074] Target Table (ODS Table) DDL: Table definition based on the Apache Paimon data lake format, including field type conversion, appending standard audit fields (dw_created_at, dw_updated_at), primary key configuration, partition settings, and data lake engine attributes.

[0075] Data processing SQL: Includes complete ETL logic such as file-level deduplication (RANK window function based on filename sorting), market entity code parsing (extracting from OSS filenames using substring_index + regexp_extract regular expression), time point standardization (00:00 / 24:00 cross-day processing), and business time calculation.

[0076] The aforementioned method for automatically generating power business data scripts first obtains the input power business data entry requirements and corresponding data samples; then, it performs semantic parsing on the data entry requirements and data samples to identify the parameter list used for script generation. This parameter list includes the data ownership type, writing method, and business parameters of the data samples. Next, based on the data ownership type and writing method, it determines the target processing mode. Finally, based on the target processing mode and business parameters, it generates a script file. This script file includes a first script for creating a temporary table, a second script for creating a target data storage table, and a third script for reading data to be processed from the temporary table, transforming it, and writing it to the target data storage table. The temporary table is used to read data to be processed from the data source storage system, and the target data storage table is used to store the transformed data to be processed. By analyzing user input to obtain the data ownership type and writing method, and determining different modes corresponding to different power business scenarios based on the combination of data ownership type and writing method, it generates script files corresponding to different modes and runs the script files to fulfill the data sample access requirements. This method can adapt to the processing of power data in different scenarios and significantly improves efficiency compared to manual development.

[0077] In one exemplary embodiment, such as Figure 2 As shown, obtaining a data sample includes the following steps S202 to S206. Wherein:

[0078] Step S202: When the data entry requirements described in natural language do not include data samples, determine the potential storage path in the data source storage system based on the geographic information and feature name information in the data entry requirements.

[0079] Optionally, when the data import request described in natural language does not include data samples, i.e., when the user does not provide sample data, the terminal automatically calls the OssSampleDataFetcherJob tool in the IDE through an AI agent. Based on the geographical information in the user description, such as province and feature name keywords, and combined with the OSS path specification in the domain knowledge base (oss: / / {bucket name} / {province in Chinese} / {feature name in Chinese} / ), it automatically infers the potential storage path of the target OSS. Automatic switching between production buckets and development buckets is supported; when the user description contains keywords such as "dev", "development", or "test", the development bucket is automatically used.

[0080] Step S204: If the scan of the potential storage path is successful, read the header and at least one row of data of the corresponding original data file from the partition directory under the potential storage path as a data sample.

[0081] Optionally, if the inferred potential storage path is successful, the terminal automatically calls the OssSampleDataFetcherJob tool in the IDE via an AI agent to automatically connect to Alibaba Cloud OSS, scans CSV files under the inferred potential storage path, and reads the header and the first 10 rows of data of the latest file in the latest partition as samples. At the same time, it automatically identifies the partition directory structure in OSS (such as dt=yyyy-MM-dd, month=yyyy-MM) and derives the partition fields and frequency suffixes.

[0082] Step S206: When scanning potential storage paths fails, perform fuzzy matching to determine candidate feature name information related to the feature name information corresponding to the regional information; determine the target storage path based on the selection instruction and the candidate feature name information; read the header and at least one row of data of the corresponding original data file from the partition directory under the target storage path as a data sample.

[0083] Optionally, when scanning for potential storage paths fails—that is, when the inferred potential storage path does not exist in OSS—the OssSampleDataFetcherJob tool automatically performs fuzzy matching, listing all similar feature names containing the keyword under that province (e.g., user input: clearing electricity price, return: day-ahead clearing electricity price, real-time clearing electricity price, etc.). The user can then select the correct candidate feature name and retry, avoiding process interruption due to inaccurate names. The obtained partition information, field names, and sample data are automatically passed to the AI ​​agent as input for subsequent intent parsing and code generation.

[0084] Optionally, after obtaining the data sample, the inferred data sample is analyzed to obtain the corresponding data parameters (such as business domain, full / incremental, scheduling method, number of points, etc.), which are then displayed to the user for confirmation at once. When using default values ​​(such as the business domain defaulting to market domain, the scheduling method defaulting to scheduling mode, etc.), the user must be explicitly informed. Generation only begins after user confirmation to avoid overturning and starting over after generating the script file.

[0085] In this embodiment, fuzzy inference enables the generation of script files with minimal input information.

[0086] In one exemplary embodiment, such as Figure 3 As shown, data ownership types include public and private types; writing methods include full write and incremental write; based on the data ownership type and writing method, the target processing mode is determined, including steps S302 to S308. Wherein:

[0087] Step S302: If the data belongs to a public type and the writing method is full write, then the target processing mode is determined to be the first processing mode.

[0088] Step S304: If the data ownership type is public and the writing method is incremental writing, then the target processing mode is determined to be the second processing mode.

[0089] Step S306: If the data belongs to a private type and the writing method is full write, then the target processing mode is determined to be the third processing mode.

[0090] Step S308: If the data ownership type is private and the writing method is incremental writing, then the target processing mode is determined to be the fourth processing mode.

[0091] Optionally, the terminal uses an AI agent to determine the target processing model based on the preset mapping relationship between data ownership type, writing method and processing model. The mapping relationship is shown in Table 1.

[0092] Table 1. Mapping relationship between data ownership type, writing method, and processing model:

[0093]

[0094] Optionally, if the data belongs to a public type and the write method is full write, then the target processing mode is determined to be the first processing mode, the primary key strategy is not to specify a primary key, use a bucket key, and the partitioning strategy is no partitioning, which is suitable for scenarios such as market clearing information, non-stop unit public small tables, etc.

[0095] Optionally, if the data belongs to a public type and the writing method is incremental writing, then the target processing mode is determined to be the second processing mode, with a specified composite primary key strategy and no partitioning strategy, which is suitable for scenarios such as publicly available large tables such as day-ahead / real-time clearing electricity prices.

[0096] Optionally, if the data belongs to a private type and the write method is full write, then the target processing mode is determined to be the third processing mode. Its primary key strategy is to specify a composite primary key, and its partitioning strategy is to partition by market entity code, suitable for scenarios such as medium- and long-term settlement and contract information.

[0097] Optionally, if the data belongs to a private type and the writing method is incremental writing, then the target processing mode is determined to be the fourth processing mode. Its primary key strategy is to specify a composite primary key containing the market entity code, and the partitioning strategy is to partition by the market entity code, which is suitable for high-frequency data scenarios such as spot market declarations and electricity confirmations.

[0098] In an exemplary embodiment, generating a script based on a target processing mode and business parameters includes: determining a script template corresponding to each processing mode according to one of a first processing mode, a second processing mode, a third processing mode, or a fourth processing mode; filling the business parameters into the script template; and generating a script file.

[0099] Optionally, the terminal uses an AI agent to determine the script template corresponding to each processing mode based on one of the first, second, third, or fourth processing modes; fills the business parameters into the script template, and generates a script file.

[0100] In this embodiment, by flexibly corresponding to the corresponding application scenarios according to four different modes, both efficiency and flexibility can be improved.

[0101] In an exemplary embodiment, the script templates corresponding to the third and fourth processing modes include a first sub-template; the templates corresponding to the second and fourth processing modes include a second sub-template.

[0102] The first sub-template is the configuration template for partition attributes; partition attributes are used to partition the target data storage table. The second sub-template is the configuration template for data lake aggregation engine attributes; data lake aggregation engine attribute configuration is used to deduplicate and merge data in the target data storage table.

[0103] Optionally, the data ownership type corresponding to the third and fourth processing modes is a private type, and the corresponding script template includes a first sub-template; the first sub-template is a configuration template for partition attributes; the partition attributes are used to partition the target data storage table.

[0104] For private data samples, market entity code parsing logic and partitioning settings are automatically injected. This means the script file includes code for partitioning by market entity code. This is because the processing logic for public and private data types is complex. Electricity trading data is divided into public data types (such as publicly available data like market operation information and clearing prices) and private data (such as transaction declarations and settlement data from specific electricity sales companies and power generation enterprises). Private data needs to have its market entity code (mkt_ent_code) parsed from the object storage (OSS) filename and used as the partition key for data isolation and storage. This logical difference permeates multiple stages, including table creation, partitioning, primary key design, and write methods, making manual maintenance highly prone to errors.

[0105] For trigger-based private data, market entity filtering conditions are automatically injected. The generated SQL conforms to coding standards: keywords are uppercase, commas are placed after the text, field pairs are aligned to the left, and TBLPROPERTIES key-value pairs are left-aligned at the beginning.

[0106] Optionally, the write method corresponding to the second and fourth processing modes is incremental write, and the corresponding template includes a second sub-template, which is a configuration template for the data lake aggregation engine attribute; the data lake aggregation engine attribute configuration is used to deduplicate and merge the target data storage table.

[0107] Configure four aggregation engine properties of Apache Paimon for incremental write scenarios.

[0108] (1) merge-engine=aggregation: Enables the streaming aggregation and merging engine to achieve automatic deduplication and merging based on the primary key.

[0109] (2) sequence.field=dw_updated_at: Set the sequence field to the data entry update timestamp to solve the problem of data disorder in a distributed environment.

[0110] (3) fields.default-aggregate-function=last_value: The default aggregation strategy is to take the latest value and supports null value overriding semantics.

[0111] (4) fields.dw_created_at.aggregate-function=min: The earliest value of the creation time of the data entering the database is retained to ensure accurate data traceability.

[0112] In traditional methods, incremental data is written using the MERGE INTO (merge-into) mode, which matches the source data with the primary key of the target table: if the primary key exists, an update is performed; if the primary key does not exist, an insert is performed. Regardless of how many times this is executed, ultimately each primary key retains only the latest record (or retains it according to specified rules).

[0113] By simplifying incremental data writing from the traditional, complex MERGE INTO mode to INSERT INTO mode through four aggregation engine properties, the data lake engine automatically performs data merging at the storage layer, significantly reducing ETL logic complexity and development threshold. INSERT INTO mode directly appends data to the target table without checking for rows with the same primary key. Multiple executions will result in duplicate primary key data in the table (unless the table itself defines a deduplication mechanism). The target table has an underlying automatic merging mechanism (such as Apache Paimon configured with an aggregation engine), allowing for appending followed by automatic deduplication by the engine.

[0114] When executing a script file configured with four aggregation engine attributes, a direct append operation is performed on the incremental data in the target data storage table to obtain the populated target data storage table; the populated target data storage table is then deduplicated and merged according to the merge configuration parameters and the primary key configuration parameters.

[0115] In this embodiment, by configuring the data lake aggregation engine in the second sub-template, the complex merging logic is pushed down to the storage layer. The upper-layer application only needs to execute a simple INSERT INTO, which simplifies the process from "complex merging" to "simple appending" and improves the efficiency of power data processing.

[0116] In one exemplary embodiment, such as Figure 4 As shown, the business parameters include standard business fields; semantic parsing is performed on the data entry requirements and data samples to identify the parameter list used for script generation, including steps S402 to S408. Among them:

[0117] Step S402: Extract the first keyword related to the business domain and the second keyword related to the writing method from the data entry requirements.

[0118] Optionally, the terminal uses an AI agent to extract a primary keyword related to the business domain from the data entry requirements. This primary keyword can be a business domain keyword, dividing the power trading business into ten major business domains: market domain (MKT), trading domain (TRD), generation domain (GEN), demand domain (DMD), user domain (CUST), product domain (PDM), external data domain (EXT), virtual power plant domain (VPP), general domain (NML), and power sales mini-program domain (ESMP). Each domain defines a standard abbreviation and the nature of the data ownership (public / private). The market domain contains public data, covering publicly available information such as power market operation information, clearing prices, and transmission sections; the other domains contain private data, involving sensitive information such as transaction declarations and settlement details of specific market participants.

[0119] Optionally, the terminal uses an AI agent to extract a second keyword related to the writing method from the data entry requirements. The second keyword can be an incremental keyword or a keyword from the entire data set.

[0120] Step S404: Determine the data attribution type based on whether the value of the first keyword belongs to the target information domain.

[0121] The target information domain can be the market domain.

[0122] Optionally, if the value of the first keyword belongs to the market domain, then the data type is determined to be a public type. If the value of the first keyword does not belong to the market domain, but to one of the other nine categories outside the market domain, then the data type is determined to be a private type.

[0123] Step S406: Determine the writing method based on the writing category corresponding to the value of the second keyword.

[0124] Optionally, if the write category corresponding to the value of the second keyword is an incremental category, the write method is determined to be incremental write, and the incremental write adopts the INSERT INTO (incremental append) write strategy.

[0125] Optionally, if the write category corresponding to the value of the second keyword is the full write category, the write method is determined to be full write, and the incremental write adopts the INSERT OVERWRITE (full overwrite) write strategy.

[0126] Step S408: Extract business fields from the data sample and match the business fields with the preset field dictionary to determine the standard business fields.

[0127] The field dictionary includes a Chinese-English mapping dictionary of standardized fields in the power trading field, as well as root abbreviation rules. For example, the field dictionary contains a Chinese-English mapping dictionary (word_dict) of more than 370 standardized fields in the power trading field, and 22 commonly used root abbreviation rules (stems), covering core concepts such as price, energy, unit, transmission, section (intf), clearing, ancillary services (aux_serv), nodal marginal price (loc_marginal_price), rolling trade (roll_trade), medium- and long-term (mlt), and spot unit (spot_unit).

[0128] Optionally, the terminal uses an AI agent to extract specific business fields from data samples and matches these business fields with a pre-defined field dictionary to determine the standard sub-segment mapped to each specific business field. For example, if the technical term "node marginal price" is not mapped using pre-defined rules, different engineers may write various different names such as "node_price", "margin_price", and "lmp", resulting in multiple copies of the same concept appearing in the system.

[0129] In this embodiment, by matching business fields with a preset field dictionary, the inconsistency problem of standard execution is fundamentally eliminated, ensuring that the naming, structure, and logic of data across all provinces in the country are unified.

[0130] In an exemplary embodiment, the business parameters further include time granularity; the method further includes: when the data sample contains a time point field, calculating adjacent time intervals in the data sample; determining the time granularity based on the time intervals; the time granularity is used to ensure that the time format and range of the target data storage table conform to the electricity business settlement cycle.

[0131] Optionally, the terminal reads the first two rows of values ​​in the "Time" field of the data sample (e.g., "00:15" and "00:30"), calculates the time interval between adjacent time points (i.e., 15 minutes), and determines that the data is a 96-point table (corresponding to a 15-minute settlement cycle in the electricity spot market); if the interval is 30 minutes, it is determined to be a 48-point table; if the interval is 1 hour, it is a 24-point table; if the interval is 2 hours, it is a 12-point table; if the time field is in numerical number format (e.g., 1, 2, 3, ..., 96), the maximum value is used to deduce the number of points. The terminal stores the determined time granularity (e.g., 96 points) as a business parameter in the parameter object, and automatically encodes the point mark (e.g., "d96") into the table name when generating the target table definition. At the same time, it generates the corresponding time standardization logic in the data processing structured query language (e.g., converting numerical numbers to standard time and handling the cross-day boundary from "24:00" to "00:00" of the next day), and maps a dedicated time point verification type (e.g., "time_96") to the field of this time in the data quality verification configuration to ensure that the time format and range of the data entering the database strictly comply with the requirements of the corresponding settlement cycle.

[0132] In this embodiment, by automatically inferring the time granularity of power trading data, users do not need to manually specify the location or write time conversion logic. It can intelligently adapt to the differences in settlement cycles of different provinces and different power markets, avoiding problems such as data entry failure, time series analysis deviation, or cross-provincial data misalignment caused by incorrect location settings.

[0133] In an exemplary embodiment, the field dictionary update method includes: when the business field does not exist in the preset field dictionary, determining a second standard business field for the business field according to preset root abbreviation rules; and storing the business field and the corresponding second standard business field in the field dictionary.

[0134] Optionally, the large language model agent loads a pre-built field dictionary and a root abbreviation rule base. When the large language model agent reads a business field (e.g., "new energy output") from the list of Chinese field names in the data sample, it performs an exact match search in the field dictionary. If no match is found, the dictionary self-evolution process is triggered. The business field is semantically decomposed, key semantic units (e.g., "new energy" and "output") are extracted, and the corresponding standard abbreviations are matched sequentially according to the root abbreviation rules (e.g., "new energy" corresponds to "new_energy", and "output" corresponds to "output"). The abbreviations are then combined in sequence to generate a second standard business field (i.e., "new_energy_output").

[0135] Perform type inference based on the actual data value of the corresponding column of the business field in the data sample to determine its data type (such as string, numeric, or date type). Append and store the Chinese name of the business field ("New Energy Output"), the generated second standard business field ("new_energy_output"), the inferred data type, etc. (such as the comma-separated value format of "Chinese name, English field name, data type") to the field dictionary to complete the real-time update of the dictionary. After the update is completed, actively feedback a prompt message to the user through the dialogue interface, informing the user of the newly added field content that has been automatically supplemented for the user to confirm or perform necessary manual review.

[0136] In this embodiment, by automatically supplementing the newly discovered business fields to the preset field dictionary, the field dictionary can continuously precipitate and accumulate new field knowledge in the power trading field during use, avoiding the repetitive work of manually intervening to maintain the dictionary every time a new field is encountered. At the same time, through the root abbreviation rule, it is ensured that even new fields can follow the unified naming specification, avoiding the problem of inconsistent field names caused by differences in manual naming habits.

[0137] In an exemplary embodiment, after generating the script file, it further includes: verifying the target data storage table according to the data quality verification configuration file.

[0138] Among them, the verification configuration file includes field-level verification rules, table-level verification rules, and special verifications for power industry time types; the field-level verification rules include: the field name, data type, whether null values are allowed, and whether it must exist for each field; the table-level verification rules include: whether additional columns are allowed, the column order verification range, and whether to ignore the column order.

[0139] The special verification for power industry time types is to map the time point field to the time point verification type dedicated to the power business settlement cycle according to the corresponding time granularity. The time point verification type includes the first time point type corresponding to a 15-minute interval, the second time point type corresponding to a 30-minute interval, the third time point type corresponding to a 1-hour interval, and the fourth time point type corresponding to a 2-hour interval, which is used to ensure that the format and range of the time point field meet the requirements of the corresponding power market settlement cycle.

[0140] Optionally, the terminal, through an AI agent, invokes the data quality verification configuration automatic generation module to output a verification configuration file in YAML format based on the target table DDL. This file includes field type verification (including power industry-specific types such as time_96 / time_48 / time_24 / time_12), null value constraints, and column order verification rules. Based on the generated DDL definition, a data quality verification configuration file (YAML format) is exported, containing: Field-level verification rules: field name (using business Chinese names), data type, whether nullable is allowed, and whether it is required. The nullable rule is linked to the DDL's NOT NULL constraint and primary key definition—primary key fields and fields explicitly marked NOT NULL are automatically set to non-nullable. Table-level verification rules: whether additional columns are allowed (allow_additional_columns), column order verification range (column_order_validation_limit), and whether column order is ignored (ignore_column_order). Dedicated verification for time types in the power industry: The time field is mapped to custom verification types such as time_96, time_48, time_24, and time_12 according to the location table type to ensure that the time format and range conform to the corresponding power market settlement cycle.

[0141] Optionally, after outputting the YAML format validation configuration file, the terminal automatically saves the generated SQL script to the path dw_model / ods / {province pinyin} through the AI ​​agent, and saves the YAML validation configuration to the path config / data_validation / {province pinyin} / . If the province directory does not exist, it is automatically created.

[0142] In this embodiment, by automatically validating the target table, it is ensured that each row of data entering the target table conforms to the specifications defined by the business. At the same time, problems can be detected in time before the data is entered into the database, ensuring the accuracy of the written data.

[0143] In an exemplary embodiment, after the target data storage table passes verification, the method further includes: calling a workflow orchestration tool to process the script file and generating a workflow identifier and task node identifiers in the workflow identifier.

[0144] Optionally, after the target data storage table passes verification and the generated SQL script is saved, the terminal uses an AI agent to call the OdsWorkflowGeneratorJob tool. Through the DataWorks OpenAPI, it creates the corresponding workflow on the Alibaba Cloud DataWorks platform (scheduled creation of CycleWorkflow, triggered creation of TriggerWorkflow), automatically orchestrates the DIDE_SHELL+EMR_SPARK_SQL dual-node architecture, configures workflow parameters (input_bucket, biz_date, etc.) and scheduling rules (supporting natural language scheduling cycles such as "8 AM daily" automatically converted to cron), and automatically submits it for deployment. Workflow creation failure does not affect the already generated SQL and YAML files. Afterward, the AI ​​agent automatically executes a self-check checklist, verifying table name integrity, public / private consistency, primary key coverage uniqueness, incremental aggregation engine attribute integrity, point table rules, YAML field consistency, file path correctness, etc., to ensure the reliability of the generated script files.

[0145] Furthermore, based on the scheduled / triggered identifier, corresponding workflows are automatically created. Scheduled workflows create CycleWorkflows, and triggered workflows create Event TriggeredWorkflows, which are then automatically archived to the corresponding DataWorks directories (scheduled workflows are categorized under ods / {province}, and triggered workflows are categorized under triggered_ods / {province}). Each workflow automatically creates two cascading nodes: the DIDE_SHELL node (environment preparation) and the EMR_SPARK_SQL node (ETL execution). The SQL content generated by the AI ​​agent is automatically populated into the Spark SQL node. All workflow parameters are automatically derived from user input, requiring no additional user input. These include the OSS input bucket path (input_bucket), business date (biz_date), partition type (day / month), and raw data path. Users can specify the scheduling time in natural language (e.g., "8:00 AM daily", "10:30 AM daily", "1st of each month"), which the system automatically converts into a cron expression; if not specified, the default scheduling time is 10:00 AM daily. Once the workflow is created, it will be automatically submitted and uploaded. Upon successful execution, the workflow ID and node ID will be output for user viewing. Workflow creation failures (such as AK / SK authentication failure, name conflicts, network timeouts, etc.) will not affect the generated SQL and YAML files; previous outputs will remain valid. Users can repair and retry or manually create the workflow in DataWorks.

[0146] Optionally, the terminal, through an AI agent, automatically creates workflows corresponding to the scheduling mode based on the scheduling mode identifier (scheduled / triggered identifier). When the scheduling mode identifier is batch scheduling, a periodic scheduling workflow is created; when the scheduling mode identifier is event-triggered, an event-triggered workflow is created. The scheduling mode automatically orchestrates environment preparation nodes and structured query language execution nodes in the workflow and fills the content of the data processing script into the structured query language execution nodes. Workflow configuration parameters are automatically derived from the user's input natural language requirements. Configuration parameters include object storage input path, business date, and partition type, without requiring additional information from the user. When the user specifies a scheduling period in the natural language requirements, the time expression described in the natural language is automatically converted into a timing expression. When the user does not specify a scheduling period, a preset default scheduling period is used. The workflow is automatically submitted and deployed online, and workflow identifiers and node identifiers are output for user viewing. When the workflow creation fails, the generated script files and data quality verification configuration files remain valid, and the user can fix the problem and retry or create them manually.

[0147] In this embodiment, the generated script file is automatically deployed to the cloud big data development platform by calling the workflow orchestration tool. Users do not need to log in to the platform to manually create workflows and configure nodes, which automates the entire solution from natural language requirement input to script file online execution on the cloud platform.

[0148] In an exemplary embodiment, the engineer who performs full access to public data (market domain - generator set non-stop information) enters the following in the IDE: Market domain, Anhui, generator set non-stop status, full data, as follows: "Date", "Release time", "Unit name", "Capacity", "Non-stop time", "Recovery time"; "2025-07-23", "2025-07-22", "#1", "600", "2025-07-23 08:00", "2025-07-23 16:00".

[0149] The terminal automatically performs the following processing via an AI agent: Business domain identification: Identifies the "market domain" and determines it to be public data, abbreviated as mkt. Province conversion: Converts "Anhui" to the pinyin "anhui". Write strategy: "Full", suffix "df", write method "INSERT OVERWRITE". Field mapping: Queries the field dictionary to map "Date" to dt DATE, "Publication time" to pub_dt DATE, "Unit name" to unit_name STRING, "Capacity" to capacity DECIMAL(30, 10), "Non-stop time" to outage_time TIMESTAMP, and "Recovery time" to restore_timeTIMESTAMP. The first processing model is obtained based on the public type + full write. Code generation based on the first processing model: Generates a table named ods_mkt_anhui_gen_unit_forced_outage_df, using a bucket key instead of a primary key in the DDL (public full data mode), and the INSERT OVERWRITE processing SQL includes the RANK window function for deduplication. Validation configuration: Generate YAML, annotating the type and nullable rules for all 6 business fields. File saving: Save the SQL to dw_model / ods / anhui / , and the YAML to config / data_validation / anhui / .

[0150] In an exemplary embodiment, the public data incremental 96-point table is accessed (market domain, clearing price). The engineer inputs the following into the IDE: Market Domain, Anhui Province, Actual Operating Transmission Section Constraint Status, Increment, with the data as follows: "Date", "Time", "Section Name", "Section Constraint Status"; "2025-07-23", "00:15", "500kV#3 Main Transformer + 500kV#2 Main Transformer of Changmen Substation", "No Blocking"; "2025-07-23", "00:30", "500kV#3 Main Transformer + 500kV#2 Main Transformer of Changmen Substation", "No Blocking".

[0151] The terminal automatically performs the following processing via an AI agent: Automatic location inference: Detects the existence of the "Time" field, calculates the interval from 00:15 to 00:30 as 15 minutes, identifies it as a 96-location table (standard settlement cycle in the electricity spot market), and adds "d96" to the table name. Primary key inference: The AI ​​infers the primary key as dt, hm, and intf_name (date + time + section name) based on field semantics. Incremental configuration: Automatically configures Paimon aggregation engine attributes (merge-engine, sequence.field, etc.). Business time calculation: Automatically adds the biz_time field, including logic for handling the 24:00 / 00:00 cross-day boundary. Temporary table configuration: Automatically adds a file time filter with modifiedAfter='${biz_date}T00:00:00' in incremental mode.

[0152] In an exemplary embodiment, private data is accessed incrementally and triggered (trading domain, spot declaration). The engineer enters the following in the IDE: Trading domain, Shanghai, spot default declaration, incremental, date + time + trading unit name, triggered, with the data as follows: "Date", "Time", "Trading unit name", "Electricity"; "2025-07-23", "00:15", "Trading unit A", "100.5"; "2025-07-23", "00:30", "Trading unit A", "98.3".

[0153] The terminal automatically performs the following processing via an AI agent: The data belongs to the "trading domain," therefore it is determined to be private data, abbreviated as trd, requiring mkt_ent_code partitioning. Chinese primary key mapping: The user specifies "date + time + trading unit name," which is mapped to dt, hm, trd_unit_name via a dictionary, and mkt_ent_code is automatically appended to the end of the primary key. Market entity code parsing: Regular expression logic for extracting mkt_ent_code from OSS filenames is automatically injected into the ETL SQL. Triggered adaptation: Upon detecting the "triggered" flag, the WHERE mkt_ent_code='${trading_unit_id}' filter condition is automatically injected into the SELECT subquery to ensure that only data from the specified market entity triggered by the OpenAPI is processed. YAML validation: In the generated validation configuration, the hm field type is automatically mapped to time_96, and the nullable primary key field is set to false.

[0154] In one exemplary embodiment, a simplified end-to-end one-click library entry (full-link automation) is used. Engineers need to access Heilongjiang's battery confirmation data. The entire process is as follows: In the IDE, simply enter the sentence: "Access Heilongjiang's battery confirmation data, incremental, triggered." No sample data is required.

[0155] The terminal automatically invokes the OssSampleDataFetcherJob tool via an AI agent. Based on "Heilongjiang" and "battery confirmation," it infers the OSS path to be oss: / / oss-luminai-data-crawler / Heilongjiang / battery confirmation / . It automatically connects to OSS, reads the header and the first 10 rows of sample data from the latest CSV file in the latest partition under that directory, and identifies the partition structure (e.g., dt=2026-03-25), injecting this information into the AI ​​agent. The AI ​​agent displays the acquired sample data and inference parameters (transaction domain = private data, incremental, triggered) to the user. After user confirmation, generation begins. Within seconds, the AI ​​agent automatically generates complete DDL, ETL SQL scripts (saved to dw_model / ods / heilongjiang / ), and YAML validation configuration (saved to config / data_validation / heilongjiang / ), appending the new fields to word_dict.csv. The AI ​​agent automatically invokes the OdsWorkflowGeneratorJob tool to create a TriggerWorkflow named ods_trd_heilongjiang_energy_confirmation_di via the DataWorks OpenAPI. It automatically orchestrates a two-node workflow (DIDE_SHELL + EMR_SPARK_SQL), populates the generated SQL into the nodes, configures workflow parameters (input_bucket, biz_date, trading_unit_id), and sets up API event triggering. The workflow is automatically submitted and deployed online, outputting the workflow ID and node ID. Data begins to be automatically stored in the database according to the triggering rules. After generation, the AI ​​agent performs a self-check, verifying all items such as table name integrity, primary key coverage and uniqueness, and aggregation engine attributes.

[0156] The entire process involved only one sentence from the engineer: no need to find sample data, write any code, log into the DataWorks console, or understand Paimon table attribute configuration, OSS path specifications, or Spark SQL syntax. From a single line of natural language to the data ingestion workflow going live, it achieved truly one-click data ingestion.

[0157] In an exemplary embodiment, an engineer needs to access multiple public data tables in Shandong Province at once. The process is as follows: The engineer enters in the IDE: Batch Access, Shandong, [Daily Clearing Price, Real-time Clearing Price, Total Power Generation Forecast], Incremental, 8 AM daily. The terminal uses an AI agent to call OssSampleDataFetcherJob for each feature to obtain sample data. For "Daily Clearing Price," the tool returns 96 sample data points including the time field; for "Total Power Generation Forecast," it returns 48 data points. Output is generated after independently confirming the parameters for each feature. DDL, SQL, and YAML files are generated for the three tables respectively, and a DataWorks workflow is automatically created, with the scheduling cycle uniformly set to 08:00 daily (cron: 00 00 08 * *?). Newly added fields are summarized into a final list of entries added to word_dict.csv. If the engineer is unsure of the structure of a table, they can first use the preview mode: Preview, Shandong, Daily Clearing Price, Incremental. Only the DDL and data dictionary are output; no files are saved, and no workflow is created. Once the engineer confirms, the generation process can continue to produce the complete output.

[0158] In the above embodiments, the AI ​​agent only needs tens of seconds to receive natural language input and output script code, significantly improving development efficiency. Over 370 rules, including naming conventions, field dictionaries, and hierarchical architecture, are hard-coded into the AI ​​agent's knowledge base, fundamentally eliminating inconsistencies in rule execution and ensuring unified naming, structure, and logic for data across all provinces nationwide. Domain knowledge accumulated in the power trading industry is structured and automatically updated through a root word library, field dictionary, and processing pattern library. Verification configurations are generated synchronously with ETL code, shifting data quality assurance from traditional "post-event repair" to "development synchronization," achieving the co-construction of quality rules and data pipelines. The entire process from data entry into OSS to writing to the Paimon data warehouse adheres to quality rules. This lowers the technical barrier to power trading data entry, allowing even non-professional business personnel to independently complete data access. After generation, the self-checking engine automatically performs multi-dimensional verification, covering checks such as table name integrity, public / private consistency, primary key uniqueness, aggregation engine attribute integrity, point table rules, and YAML field number matching. This effectively prevents AI from missing key configuration items in complex scenarios and ensures the engineering reliability of each generated product.

[0159] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0160] Based on the same inventive concept, this application also provides an automatic power business data script generation system for implementing the above-mentioned automatic power business data script generation method. The solution provided by this system is similar to the implementation described in the above method; therefore, the specific limitations of one or more embodiments of the automatic power business data script generation system provided below can be found in the limitations of the automatic power business data script generation method described above, and will not be repeated here.

[0161] In one exemplary embodiment, an automatic power business data script generation system is provided, comprising:

[0162] The acquisition module is used to acquire the input power business data entry requirements and corresponding data samples.

[0163] The parsing module is used to perform semantic parsing on data entry requirements and data samples to identify the parameter list used for script generation; the parameter list includes the data type, writing method, and business parameters of the data samples.

[0164] The determination module is used to determine the target processing mode based on the data ownership type and writing method.

[0165] The generation module is used to generate script files based on the target processing mode and business parameters. The script files include a first script for creating a temporary table, a second script for creating a target data storage table, and a third script for reading data to be processed from the temporary table, transforming it, and writing it into the target data storage table. The temporary table is used to read data to be processed from the data source storage system, and the target data storage table is used to store the transformed data to be processed.

[0166] In an exemplary embodiment, the acquisition module is further configured to: determine a potential storage path in the data source storage system based on the geographic information and feature name information in the data entry requirement when the acquired data entry requirement described in natural language does not include data samples; if scanning the potential storage path is successful, read the header and at least one row of data of the corresponding original data file from the partition directory under the potential storage path as a data sample; if scanning the potential storage path fails, perform fuzzy matching to determine candidate feature name information related to the feature name information corresponding to the geographic information; determine the target storage path based on the selection instruction and the candidate feature name information; and read the header and at least one row of data of the corresponding original data file from the partition directory under the target storage path as a data sample.

[0167] In an exemplary embodiment, the data ownership type includes public type and private type; the writing method includes full write and incremental write; the determining module is further configured to determine the target processing mode as a first processing mode if the data ownership type is public type and the writing method is full write; determine the target processing mode as a second processing mode if the data ownership type is public type and the writing method is incremental write; determine the target processing mode as a third processing mode if the data ownership type is private type and the writing method is full write; and determine the target processing mode as a fourth processing mode if the data ownership type is private type and the writing method is incremental write; the generating module is further configured to determine the script template corresponding to each processing mode according to one of the first processing mode, the second processing mode, the third processing mode, or the fourth processing mode; fill the business parameters into the script template, and generate a script file.

[0168] In an exemplary embodiment, the script templates corresponding to the third and fourth processing modes include a first sub-template; wherein, the first sub-template is a configuration template for partition attributes; the partition attributes are used to partition the target data storage table; the templates corresponding to the second and fourth processing modes include a second sub-template; wherein, the second sub-template is a configuration template for data lake aggregation engine attributes; the data lake aggregation engine attribute configuration is used to deduplicate and merge the target data storage table.

[0169] In an exemplary embodiment, the business parameters include standard business fields; the parsing module is further configured to extract a first keyword related to the business domain and a second keyword related to the writing method from the data entry requirements; determine the data attribution type based on whether the value of the first keyword belongs to the target information domain; determine the writing method based on the writing category corresponding to the value of the second keyword; extract business fields from the data sample and match the business fields with a preset field dictionary to determine standard business fields.

[0170] In an exemplary embodiment, the business parameters further include time granularity; the parsing module is also configured to calculate adjacent time intervals in the data sample when the data sample contains a time point field; determine the time granularity based on the time interval; the time granularity is used to ensure that the time format and range of the target data storage table conform to the electricity business settlement cycle.

[0171] In an exemplary embodiment, the system further includes an update module, configured to determine a second standard business field for the business field according to a preset root abbreviation rule when the business field does not exist in a preset field dictionary; and store the business field and the corresponding second standard business field in the field dictionary.

[0172] In an exemplary embodiment, the system further includes a verification module for verifying the target data storage table according to a data quality verification configuration file; wherein, the verification configuration file includes field-level verification rules, table-level verification rules, and special verification for the power industry time type; the field-level verification rules include: the field name, data type, whether null values ​​are allowed, and whether the field must exist for each field; the table-level verification rules include: whether additional columns are allowed, the column order verification range, and whether column order is ignored.

[0173] In one exemplary embodiment, the system further includes a deployment module for invoking a workflow orchestration tool to process the script file and generate a workflow identifier and task node identifiers within the workflow identifier.

[0174] The modules in the aforementioned automatic power business data script generation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0175] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores power business data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for automatically generating power business data scripts.

[0176] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0177] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0178] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0179] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0180] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0181] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0182] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for automatically generating power business data scripts, characterized in that, The method, applied to large language model agents, includes: Obtain the input data entry requirements for power business and the corresponding data samples; Semantic parsing is performed on the data entry requirements and the data samples to identify the parameter list used for script generation; wherein, the parameter list includes the data ownership type, writing method and business parameters of the data samples; The target processing mode is determined based on the data ownership type and the writing method. Based on the target processing mode and the business parameters, a script file is generated; wherein, the script file includes a first script for creating a temporary table, a second script for creating a target data storage table, and a third script for reading data to be processed from the temporary table, transforming it, and writing it into the target data storage table; the temporary table is used to read data to be processed from the data source storage system, and the target data storage table is used to store the transformed data to be processed.

2. The method according to claim 1, characterized in that, Obtain data samples, including: When the data entry request described in natural language does not include data samples, the potential storage path in the data source storage system is determined based on the geographic information and feature name information in the data entry request. If the scan of the potential storage path is successful, the header and at least one row of data of the corresponding original data file are read from the partition directory under the potential storage path as the data sample. When scanning the potential storage path fails, fuzzy matching is performed to determine candidate feature name information related to the feature name information corresponding to the regional information; based on the selection instruction and the candidate feature name information, the target storage path is determined; the header and at least one row of data of the corresponding original data file are read from the partition directory under the target storage path as the data sample.

3. The method according to claim 1, characterized in that, The data ownership type includes public type and private type; the writing method includes full write and incremental write; based on the data ownership type and the writing method, the target processing mode is determined, including: If the data belongs to a public type and the writing method is full write, then the target processing mode is determined to be the first processing mode; If the data belongs to a public type and the writing method is incremental writing, then the target processing mode is determined to be the second processing mode. If the data belongs to a private type and the writing method is full write, then the target processing mode is determined to be the third processing mode. If the data belongs to a private type and the writing method is incremental writing, then the target processing mode is determined to be the fourth processing mode. The step of generating a script based on the target processing mode and the business parameters includes: Based on one of the first processing mode, the second processing mode, the third processing mode, or the fourth processing mode, determine the script template corresponding to each processing mode; The business parameters are filled into the script template to generate the script file.

4. The method according to claim 3, characterized in that, The script templates corresponding to the third processing mode and the fourth processing mode include a first sub-template; wherein, the first sub-template is a configuration template for partition attributes; the partition attributes are used to partition the target data storage table; The templates corresponding to the second processing mode and the fourth processing mode include a second sub-template; wherein, the second sub-template is a configuration template for the data lake aggregation engine attribute; the data lake aggregation engine attribute configuration is used to deduplicate and merge the target data storage table.

5. The method according to claim 1, characterized in that, The business parameters include a first standard business field; the semantic parsing of the data entry requirements and the data sample to identify the parameter list used for script generation includes: Extract the first keyword related to the business domain and the second keyword related to the writing method from the data entry requirements; The data attribution type is determined based on whether the value of the first keyword belongs to the target information domain; The writing method is determined based on the writing category corresponding to the value of the second keyword; The business fields are extracted from the data sample and matched with a preset field dictionary to determine the first standard business field.

6. The method according to claim 5, characterized in that, The business parameters also include time granularity; the method further includes: When the data sample contains a time point field, the time interval between adjacent data samples is calculated; based on the time interval, the time granularity is determined; the time granularity is used to ensure that the time format and range of the target data storage table conform to the electricity business settlement cycle.

7. The method according to claim 5, characterized in that, The update methods for the field dictionary include: When the business field does not exist in the preset field dictionary, a second standard business field is determined according to the preset root abbreviation rules. The business field and the corresponding second standard business field are stored in the field dictionary.

8. The method according to claim 1, characterized in that, Following the generation of the script file, the following is also included: The target data storage table is validated according to the data quality validation configuration file; wherein, the validation configuration file contains field-level validation rules, table-level validation rules, and special validation for the time type in the power industry; the field-level validation rules include: the field name, data type, whether null values ​​are allowed, and whether the field must exist for each field; the table-level validation rules include: whether additional columns are allowed, the column order validation range, and whether column order is ignored.

9. The method according to claim 8, characterized in that, After the target data storage table passes verification, the following is also included: The workflow orchestration tool is invoked to process the script file, generating a workflow identifier and task node identifiers within the workflow identifier.

10. A system for automatically generating power business data scripts, characterized in that, The system includes: The acquisition module is used to acquire the input power business data entry requirements and corresponding data samples; The parsing module is used to perform semantic parsing on the data entry requirements and the data samples, and to identify the parameter list used for script generation; wherein, the parameter list includes the data ownership type, writing method and business parameters of the data samples; The determination module is used to determine the target processing mode based on the data ownership type and the writing method; The generation module is used to generate a script file based on the target processing mode and the business parameters; wherein, the script file includes a first script for creating a temporary table, a second script for creating a target data storage table, and a third script for reading data to be processed from the temporary table, transforming it, and writing it into the target data storage table; the temporary table is used to read data to be processed from the data source storage system, and the target data storage table is used to store the transformed data to be processed.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images