Data processing task script processing method and device, equipment and medium
By collecting operational information and using large language models to automatically diagnose and update the scripts for data warehouse ETL tasks, the problems of low efficiency and poor reliability caused by reliance on human experience in existing technologies are solved, and continuous optimization and automated updates of data warehouse performance are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BAOTOU BAOYIN CONSUMER FINANCE CO LTD
- Filing Date
- 2026-01-16
- Publication Date
- 2026-04-28
AI Technical Summary
In existing technologies, data warehouse ETL tasks rely on human experience, resulting in low update processing efficiency and poor reliability, making it difficult to efficiently solve performance bottlenecks. Furthermore, traditional rule engines have limited coverage, cannot identify complex problems, and lack automated verification mechanisms, thus hindering the evolution of optimization capabilities.
By collecting operational information, triggering conditions to be adjusted, obtaining initial script data and converting it into tree-structured data, combining it with a preset large language model to determine fault information, automatically updating script data, and performing canary releases, and combining monitoring information to perform self-feedback optimization of the training sample set.
It achieves end-to-end automation of data processing tasks, improves the accuracy and efficiency of updates, breaks through the coverage limitations of traditional rule engines, and ensures the continuous iteration of data warehouse performance optimization capabilities.
Smart Images
Figure CN121934874A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data warehouse technology, and in particular to a data processing task script processing method, a data processing task script processing device, an electronic device, and a computer-readable storage medium. Background Technology
[0002] With the explosive growth of enterprise data volume, the number and complexity of data processing tasks in data warehouses, such as ETL (Extract-Transform-Load) tasks, have increased significantly. In related technologies, performance optimization, correction, and updates for data processing tasks such as data warehouse ETL tasks rely on human experience, resulting in low efficiency and reliability in update processing. This makes it difficult to efficiently resolve the performance bottlenecks of data warehouses. Summary of the Invention
[0003] In view of the above problems, embodiments of the present invention are proposed to provide a data processing task script processing method, a data processing task script processing apparatus, an electronic device, and a computer-readable storage medium that overcome or at least partially solve the above problems.
[0004] To address the aforementioned problems, in a first aspect of this invention, an embodiment of the invention discloses a data processing task script processing method, comprising: Collect runtime information while performing data processing tasks; In response to the running information triggering the adjustment condition, the initial script data corresponding to the data processing task is obtained; Convert the initial script data into tree structure data; Based on the tree structure data, the operational information, and the preset large language model, fault information is determined; The initial script data is updated based on the fault information, and the target script data is determined. Canary release is performed based on the target script data.
[0005] Optionally, the method further includes: After the gray-scale release, collect monitoring information; By combining the monitoring information, the target script data, and the initial script data, a training sample set is determined; The preset large language model is trained based on the training sample set to update the preset large language model.
[0006] Optionally, the step of determining the training sample set by combining the monitoring information, the target script data, and the initial script data includes: Based on the monitoring information, the performance status is determined; When the performance status is in an upward trend, the target script data and the initial script data are associated to determine the training sample set.
[0007] Optionally, the method further includes: If the performance status is in a degraded state, a version rollback is performed based on the initial script data.
[0008] Optionally, the step of determining fault information based on the tree structure data, the operational information, and the preset large language model includes: Input information is generated by combining the tree structure data and the runtime information; The input information is input into a preset large language model; Obtain the fault location and the impact score corresponding to the fault location output by the preset large language model; By combining the fault location and the impact score, the fault information is determined.
[0009] Optionally, the step of updating the initial script data based on the fault information and determining the target script data includes: Based on the impact score, the fault locations are sorted in descending order to determine the set of fault locations; The first few fault locations in the fault location set are determined as the fault locations to be corrected. For any of the aforementioned fault locations to be corrected, the fault location to be corrected is mapped to the initial script data to determine the script configuration data; Determine the data type of the script configuration data; Based on the data type, the updated configuration data is determined by matching from a preset rule base; The script configuration data is replaced based on the updated configuration data to determine the target script data.
[0010] Optionally, the step of performing canary release based on the target script data includes: Perform a consistency test on the target script data to determine the verification status; If the verification status is "verification passed", a canary release will be performed based on the target script data.
[0011] In a second aspect, embodiments of the present invention disclose a data processing task script processing apparatus, comprising: The first acquisition module is used to collect operational information when performing data processing tasks; The acquisition module is used to acquire the initial script data corresponding to the data processing task in response to the running information triggering the adjustment condition; The conversion module is used to convert the initial script data into tree structure data; The fault module is used to determine fault information based on the tree structure data, the running information, and the preset large language model; The update module is used to update the initial script data based on the fault information and determine the target script data; The publishing module is used to perform canary releases based on the target script data.
[0012] In a third aspect, an embodiment of the present invention discloses an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the data processing task script processing method as described above.
[0013] In a fourth aspect, embodiments of the present invention disclose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data processing task script processing method described above.
[0014] The embodiments of the present invention have the following advantages: This invention, in its embodiments, collects operational information during data processing tasks; in response to the operational information triggering adjustment conditions, obtains the initial script data corresponding to the data processing task; converts the initial script data into tree-structured data; determines fault information based on the tree-structured data, the operational information, and a preset large language model; updates the initial script data based on the fault information, and determines target script data; and performs canary release based on the target script data. By collecting operational information and autonomously triggering the determination of initial script data requiring updates, updates can be detected and triggered in a timely manner, allowing for processing before performance bottlenecks occur. Combining the large language model to correct and update the initial script data, with automatic parsing and problem diagnosis, overcomes the coverage limitations of traditional rule engines, improving the accuracy and efficiency of updates; finally, canary release achieves end-to-end automation, improving the efficiency and reliability of data processing task script updates, thereby enhancing the performance of data warehouse data processing tasks. Attached Figure Description
[0015] Figure 1 This is a flowchart illustrating the steps of an embodiment of a data processing task script processing method according to the present invention; Figure 2 This is a flowchart illustrating the steps of another embodiment of the data processing task script processing method of the present invention; Figure 3 This is a flowchart of the first part of an example of a data processing task script processing method of the present invention; Figure 4 This is a flowchart of the second part of an example of a data processing task script processing method of the present invention; Figure 5 This is a structural block diagram of an embodiment of a data processing task script processing device according to the present invention; Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention; Figure 7 This is a structural block diagram of a computer-readable storage medium provided in an embodiment of the present invention. Detailed Implementation
[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0017] In related technologies, optimizing data processing tasks such as ETL tasks mainly relies on the experience of data engineers. Engineers must manually review task logs, analyze execution plans, write optimization scripts, and then verify the optimization through test scripts before deployment. In this process, the optimization cycle for a single task is typically 1-3 days, and the optimization effect is highly dependent on the individual engineer's ability, making it difficult to establish a standardized optimization system. Some solutions use monitoring tools to collect operational metrics (time consumption, resource utilization, etc.) for data processing tasks such as ETL tasks; a pre-defined rule engine stores fixed optimization rules (e.g., "add an index for full table scan"); when a metric triggers a rule threshold (e.g., time consumption exceeds 1 hour), the system pushes an alert and generates rule-based correction suggestions; the script is then manually modified according to the suggestions, tested, and deployed manually. However, existing solutions rely on pre-defined rule engines, requiring rules to be defined manually in advance, resulting in limited coverage due to the limitations of the optimization rules. Due to the high logical complexity of ETL task scripts (SQL / Python) (such as nested joins between multiple tables and custom function calls), new performance issues (such as "full calculation caused by window functions not specifying partition keys") cannot be identified by predefined rules, resulting in over 70% of complex problems failing to be automatically diagnosed. Existing solutions only generate correction and update suggestions, lacking automated testing and verification mechanisms, leading to low reliability. Manual script modifications, if not adequately tested (e.g., failure to verify data consistency), can result in data errors after deployment; statistics show that such issues account for over 40% of ETL failures. The existing rule base cannot be automatically updated based on historical optimization cases, and the lack of a closed-loop feedback mechanism prevents optimization capabilities from evolving. When the ETL engine version is upgraded (e.g., Spark 2.x → 3.x) or the data model changes, old rules may become invalid, but the system cannot automatically discard invalid rules or generate new ones, causing processing effectiveness to decline over time.
[0018] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of a data processing task script processing method according to the present invention. The data processing task script processing method may specifically include the following steps: Step 101: Collect runtime information while performing data processing tasks; In this embodiment of the invention, the data processing task can be an ETL task. An ETL task refers to a task that extracts data from a data source, transforms and processes it, and then loads it into a target data warehouse. When executing a data processing task, such as an ETL task, runtime information can be collected. The runtime information characterizes various performance states of the data processing task during execution, such as time consumption, resource utilization, and data skew.
[0019] Step 102: In response to the running information triggering the adjustment condition, obtain the initial script data corresponding to the data processing task; The execution status of tasks can be identified through runtime information. When the runtime information triggers the adjustment conditions, the initial script data corresponding to the data processing task can be obtained. The initial script data is the script data for executing the data processing task. The adjustment conditions can be determined according to the actual situation, such as the time taken exceeding a certain duration, resource utilization exceeding a certain utilization rate, data skew exceeding a certain skew, etc., and these conditions can be used individually.
[0020] Step 103: Convert the initial script data into tree structure data; The configurations in the initial script data can be converted into tree structure data, which can then be used to display the parent-child relationships and recursive characteristics of the configurations in the initial script data.
[0021] Step 104: Based on the tree structure data, the running information, and the preset large language model, determine the fault information; The pre-defined large language model refers to a deep learning model trained on massive amounts of text, possessing natural language understanding, code parsing, and generation capabilities. Examples include GPT and DeepSeek. Tree-structured data and runtime information can be used as the task execution status, combined with the pre-defined large language model for analysis, to identify faults in the current initial script data and determine the fault information.
[0022] Step 105: Update the initial script data based on the fault information and determine the target script data; Based on the faults indicated in the fault information, the corresponding faults are corrected, the initial script data is updated, and the target script data is determined. The target script data is the script data generated after correcting the initial script data based on the fault information.
[0023] Step 106: Perform canary release based on the target script data.
[0024] Data can be released in a phased manner based on the target script, gradually going live to ensure the reliability of business operations.
[0025] This invention, in its embodiments, collects operational information during data processing tasks; in response to the operational information triggering adjustment conditions, obtains the initial script data corresponding to the data processing task; converts the initial script data into tree-structured data; determines fault information based on the tree-structured data, the operational information, and a preset large language model; updates the initial script data based on the fault information, and determines target script data; and performs canary release based on the target script data. By collecting operational information and autonomously triggering the determination of initial script data requiring updates, updates can be detected and triggered in a timely manner, allowing for processing before performance bottlenecks occur. Combining the large language model to correct and update the initial script data, with automatic parsing and problem diagnosis, overcomes the coverage limitations of traditional rule engines, improving the accuracy and efficiency of updates; finally, canary release achieves end-to-end automation, improving the efficiency and reliability of data processing task script update processing, thereby enhancing the performance of the data warehouse.
[0026] Reference Figure 2 The diagram illustrates a flowchart of another embodiment of the data processing task script processing method of the present invention. The data processing task script processing method may specifically include the following steps: Step 201: Collect runtime information while performing data processing tasks; When performing data processing tasks, such as ETL tasks, information collection components can be used to collect various status parameters as runtime information. For example, log collection components, performance metric collection components, and code repository synchronization components can be deployed as information collection components to collect ETL task runtime logs, hardware status, task performance metrics, script code, and metadata change records in real time and associate them with a unique task ID (identifier) as runtime information.
[0027] Step 202: In response to the running information triggering the adjustment condition, obtain the initial script data corresponding to the data processing task; Feature extraction can be performed on the collected operational information to determine whether the corresponding status meets the adjustment criteria. The content requiring correction and updates can then be filtered based on these criteria. For example, the collected data can be cleaned and standardized, and logs can be structured into JSON format and filtered for outliers. Performance characteristics, data skew, subquery nesting depth, etc., can then be extracted and adjusted based on preset adjustment criteria, such as "full table scan" or "unreasonable resource configuration."
[0028] When the running information triggers the adjustment condition, the system can respond to the running information triggering the adjustment condition and obtain the initial script data corresponding to the current data processing task.
[0029] Step 203: Convert the initial script data into tree structure data; Configuration items in the initial script data can be parsed, and each configuration item can be converted into tree-structured data based on its relationships. The tree-structured data can be an AST (Abstract Syntax Tree) structure, meaning the initial script data is converted into AST data. For example, the initial script data can be read character by character, identifying keywords, identifiers, operators, literals (such as numbers and strings), delimiters (such as parentheses and semicolons), etc. No type and value are assigned to each lexical unit. The type and value of the lexical units are converted into AST data according to the language's grammatical rules (such as BNF paradigm).
[0030] Step 204: Based on the tree structure data, the running information, and the preset large language model, determine the fault information; The operating status can be represented by tree-structured data and operating information, and combined with a pre-set large language model to locate faults and determine fault information.
[0031] In an optional embodiment of the present invention, the step of determining fault information based on the tree structure data, the runtime information, and the preset large language model includes: Sub-step S2041: Combine the tree structure data and the running information to generate input information; It can combine tree structure data and runtime information to convert them into input information for a pre-defined large language model.
[0032] Sub-step S2042: Input the input information into a preset large language model; The input information is fed into a preset large language model for analysis. The input to the preset large language model can be metadata from the script data and various performance statuses from the runtime information. Based on the script configuration items corresponding to the performance, the model locates the script configuration items corresponding to performance deficiencies or failures, and determines the impact score based on the importance of these script configuration items.
[0033] Sub-step S2043: Obtain the fault location and the impact score corresponding to the fault location output by the preset large language model; It can obtain the fault location corresponding to the configuration item that needs to be modified output by the preset large language model, as well as the impact score corresponding to the fault location.
[0034] Sub-step S2044: Combine the fault location and the impact score to determine the fault information.
[0035] Fault information is determined by combining fault location and impact score.
[0036] Step 205: Update the initial script data based on the fault information and determine the target script data; Based on the fault location and impact score in the fault information, the corresponding content in the initial script data is corrected, the initial script data is updated, and the target script data is determined.
[0037] In an optional embodiment of the present invention, the step of updating the initial script data based on the fault information and determining the target script data includes: Sub-step S2051: Sort the fault locations in descending order based on the impact score to determine the fault location set; The fault locations can be sorted in descending order based on their impact scores, that is, sorted from highest to lowest importance, to determine the set of fault locations.
[0038] Sub-step S2052: Determine the first few fault locations in the fault location set as the fault locations to be corrected; By identifying the top few fault locations in the fault location set—that is, the configuration items corresponding to the more important fault locations—as the content that needs to be corrected, the top few fault locations in the fault location set can be determined as the fault locations to be corrected. The specific number of fault locations in the fault location set can be determined according to the actual situation, and this invention does not limit this.
[0039] Sub-step S2053: For any of the fault locations to be corrected, map the fault location to be corrected to the initial script data and determine the script configuration data; For any fault location to be corrected, map the fault location to be corrected to the initial script data, and match the configuration items in the initial script data to determine the script configuration data.
[0040] Sub-step S2054: Determine the data type of the script configuration data; Determine the data type corresponding to the script configuration data.
[0041] Sub-step S2055: Based on the data type, match and determine the updated configuration data from the preset rule base; Based on this data type, the corresponding correction rules are matched from the preset rule base, such as adding partition filtering conditions, adjusting the Join order, etc. Based on these rules, the data to be updated for the configuration items corresponding to the fault location to be corrected is determined, and the updated configuration data is determined.
[0042] Sub-step S2056: Replace the script configuration data based on the updated configuration data to determine the target script data.
[0043] The updated configuration data replaces the script configuration data in the initial script data, thereby updating the initial script data to determine the target script data.
[0044] Step 206: Perform canary release based on the target script data; Canary releases can be performed based on target script data, and tasks can be launched in batches according to task priority, such as releasing low-priority tasks first, followed by core tasks.
[0045] In an optional embodiment of the present invention, the step of performing canary release based on the target script data includes: Sub-step S2061: Perform a consistency test on the target script data to determine the verification status; Consistency testing can be performed on target script data. The target script data can be executed in a test environment, and consistency testing can be conducted through data sampling and full data verification to determine the verification status. The verification status can include verification passed and verification failed. A verification passed status indicates that the consistency test is error-free; conversely, a verification failed status indicates that the consistency test contains errors.
[0046] Sub-step S2062: If the verification status is verified as passed, perform canary release based on the target script data.
[0047] If the verification status is "verification passed," meaning the consistency test has passed, a canary release can be carried out based on the target script data.
[0048] When the verification status is "verification failed", the initial script data can be corrected and updated again.
[0049] Step 207: After the gray-scale release, collect monitoring information; After a canary release, the information collection component can be used to collect various performance metrics of the target script data during execution to determine monitoring information.
[0050] Step 208: Combine the monitoring information, the target script data, and the initial script data to determine the training sample set; Based on the status identification of the target script data in the monitoring information, whether the faults in the initial script data have been corrected, and thus determine the corresponding training sample set, which can be used for fine-tuning of the large language model.
[0051] In an optional embodiment of the present invention, the step of determining the training sample set by combining the monitoring information, the target script data, and the initial script data includes: Sub-step S2081: Determine the performance status based on the monitoring information; Performance status can be determined by extracting various states from the monitoring information.
[0052] Sub-step S2082: When the performance state is in an upward performance state, associate the target script data and the initial script data to determine the training sample set.
[0053] When the performance is improving—meaning the target script data corrects the errors in the initial script data—the target and initial script data can be correlated to identify the modified rules and designate them as samples in the training set. When the modified rules are frequently identified, they can be solidified to improve processing efficiency.
[0054] Furthermore, to avoid performance degradation caused by target script data or a decrease in overall system reliability due to data inconsistencies or other anomalies, version rollback can be performed. In an optional embodiment of the present invention, the method further includes: Sub-step S1: If the performance status is in a performance degradation state, perform a version rollback based on the initial script data.
[0055] If the performance status is degraded, meaning the target script data has not corrected the faults in the initial script data but has instead amplified the faults, a version rollback can be performed based on the initial script data. For example, if performance drops by more than 10% or data is inconsistent after deployment, an automatic rollback to the initial script data should be performed and the reason for the failure should be recorded.
[0056] Step 209: Train the preset large language model based on the training sample set to update the preset large language model.
[0057] A training sample set can be used to train a pre-set large language model, thereby further updating and optimizing the pre-set large language model, completing a self-feedback loop, and improving the processing accuracy of the pre-set large language model.
[0058] This invention, in its embodiments, collects operational information during data processing tasks; in response to the operational information triggering adjustment conditions, obtains the initial script data corresponding to the data processing task; converts the initial script data into tree-structured data; determines fault information based on the tree-structured data, the operational information, and a preset large language model; updates the initial script data based on the fault information, and determines target script data; performs canary release based on the target script data; after canary release, collects monitoring information; combines the monitoring information, the target script data, and the initial script data to determine a training sample set; and trains the preset large language model based on the training sample set to update the preset large language model. By collecting operational information to autonomously trigger the determination of initial script data requiring updates, updates can be detected and triggered in a timely manner, allowing for processing before performance bottlenecks occur. Combining the large language model to correct and update the initial script data, automatic parsing and problem diagnosis overcome the coverage limitations of traditional rule engines, improving the accuracy and efficiency of updates; and finally, canary release achieves end-to-end automation, which can improve the efficiency and reliability of data processing task script update processing, thereby improving the performance of the data warehouse. Furthermore, after release, the large language model will be trained and adjusted based on the monitoring information, target script data, and initial script data to achieve a self-feedback closed loop, enabling continuous iteration of optimization capabilities and thus improving processing accuracy.
[0059] To enable those skilled in the art to clearly understand the implementation process of the embodiments of the present invention, some examples are provided below: (Refer to...) Figure 3 The first part of the flowchart of an example of a data processing task script processing method of the present invention is shown; refer to Figure 4 The diagram shows a second part of the flowchart of an example of a data processing task script processing method according to the present invention.
[0060] The ETL task executes at the start. It returns to the data acquisition layer for continuous monitoring and optimization. The log acquisition component collects runtime logs; the performance metric acquisition component collects hardware / task metrics. This data is associated with a unique task ID and aggregated. It then enters the data processing layer for further processing. Cleaning and standardization: logs are structured into JSON format + outlier filtering. Feature engineering: data skew / subquery depth features are extracted. Issue labeling: preliminary classification based on preset labels. The processed data + features + issue labels are output. It determines whether optimization is triggered. If not, the process ends. If triggered, it adds the data to the optimization list. Processing is handled by the large language model layer sub-process. The script can be modified by the script parsing and modification modules to obtain the modified script. Subsequent processing is performed by the automated execution layer. Consistency testing engine: data consistency + performance consistency. It determines whether the consistency test passes. If the consistency test fails, it returns and regenerates optimization suggestions. If the consistency test passes, it is released. The deployment effect is monitored in real time. It determines whether the deployment results are abnormal. In case of an anomaly, based on the rollback mechanism: automatically roll back to the original version and notify manual review. Then end the update. In case of normal operation, deployment is complete. The feedback evolution layer handles the process, optimizing the feedback, evaluating its effectiveness, fine-tuning the model, and updating and eliminating rule base elements. Finally, update the LLM model and rule base.
[0061] In practical applications, the above process can be deployed within an operating system, including a data acquisition layer, a data processing layer, a large language model layer, an automated execution layer, and a feedback evolution layer connected sequentially. These layers collaborate to achieve end-to-end self-feedback optimization of the ETL task. A model monitoring report can be used to calculate the ETL task. This task is an MLSQL script executed daily at 7 AM, its function being to statistically analyze the input and output parameters of the model data table to determine if they are normal. Recently, due to a surge in data volume, the task has frequently timed out (originally within 0.5 hours), causing delays in the model monitoring report at 9 AM, impacting business operations. This is used as an example for illustration. The data acquisition layer can collect runtime information such as runtime logs and performance metrics. The runtime logs can display "Data skew in the model data table Join phase (business_no associated data volume percentage 40%)". Performance metrics include the execution status of subtasks; most tasks are completed within minutes, but one task's execution time exceeded one hour, triggering the task protection mechanism and forcibly terminating it. The data processing layer can extract performance features from the cleaned data, with a data skew ratio of 5:1 (maximum partition data size / minimum partition data size), and the issue is initially labeled as "data skew". The large language model layer can use a large language model for script analysis and correction, parsing the AST structure of the SparkSQL script. Combined with performance indicator diagnosis, the model data table is an incremental table, and the related key business_no has hot values (business_no is null in 20%), causing data skew during the Shuffle phase, frequent executor OOM, failure to utilize Spark's distributed computing capabilities, and excessively long single-node computation time. The correction and update are to add a partition filter condition where loan_date between '2025-10-01' and '2025-10-31' (taking data from the past month, within business requirements), reducing the expected scan volume to 50 million rows (originally 10%); and generating a modified SparkSQL script. The automated execution layer includes a consistency testing engine, a canary release controller, and a rollback mechanism. First, consistency testing is performed. The modified script is executed in the test environment, and the results show consistent data. Performance is verified; if the total execution time is within 30 minutes, the target script's performance is considered satisfactory. Finally, a canary release is conducted. On the first day, the script is deployed in parallel, tasks are reviewed, and monitoring shows no data anomalies and stable execution time. On the third day, the official release begins, and the task is completed on time. If the performance does not meet the standards, a version rollback is performed. The feedback evolution layer performs closed-loop optimization based on the updated script's execution status. The effectiveness can be evaluated, including the percentage reduction in execution time, the percentage of resource savings, and whether there is any business impact; such cases are marked as "high-value optimization cases."This case study (original script + performance data + optimization scheme) was incorporated into the fine-tuning dataset to improve the LLM's accuracy in identifying scenarios such as "data skew". The rule base can also be updated to include new rules for "data skew conditions involving incremental tables," which can be directly triggered by subsequent similar tasks, reducing the cost of LLM calls.
[0062] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.
[0063] Reference Figure 5 The diagram illustrates a structural block diagram of an embodiment of a data processing task script processing device according to the present invention. The data processing task script processing device may specifically include the following modules: The first acquisition module 501 is used to acquire running information when performing data processing tasks; The acquisition module 502 is used to acquire the initial script data corresponding to the data processing task in response to the running information triggering the adjustment condition; The conversion module 503 is used to convert the initial script data into tree structure data; Fault module 504 is used to determine fault information based on the tree structure data, the running information and the preset large language model; Update module 505 is used to update the initial script data based on the fault information and determine the target script data; The publishing module 506 is used to perform canary releases based on the target script data.
[0064] In an optional embodiment of the present invention, the device further includes: The second data acquisition module is used to collect monitoring information after the gray-scale release. The training set module is used to determine the training sample set by combining the monitoring information, the target script data, and the initial script data; The training module is used to train the preset large language model based on the training sample set in order to update the preset large language model.
[0065] In an optional embodiment of the present invention, the training set module includes: The performance determination submodule is used to determine the performance status based on the monitoring information. The training set submodule is used to associate the target script data and the initial script data to determine the training sample set when the performance state is in an upward performance state.
[0066] In an optional embodiment of the present invention, the device further includes: The rollback module is used to perform a version rollback based on the initial script data when the performance status is in a performance degradation state.
[0067] In an optional embodiment of the present invention, the fault module 504 includes: The input conversion submodule is used to combine the tree structure data and the running information to generate input information; The input submodule is used to input the input information into a preset large language model; The acquisition submodule is used to acquire the fault location output by the preset large language model and the impact score corresponding to the fault location; The fault submodule is used to determine fault information by combining the fault location and the impact score.
[0068] In an optional embodiment of the present invention, the update module 505 includes: The sorting submodule is used to sort the fault locations in descending order based on the impact score to determine the set of fault locations. The location determination submodule is used to determine the first few fault locations in the fault location set as the fault locations to be corrected. The mapping submodule is used to map any of the fault locations to be corrected to the initial script data, thereby determining the script configuration data. The type submodule is used to determine the data type of the script configuration data; The matching submodule is used to match and determine the updated configuration data from a preset rule base based on the data type. The replacement submodule is used to replace the script configuration data based on the updated configuration data in order to determine the target script data.
[0069] In an optional embodiment of the present invention, the publishing module 506 includes: The testing submodule is used to perform consistency testing on the target script data and determine the verification status; The release submodule is used to perform canary release based on the target script data when the verification status is verified as passed.
[0070] This invention, in its embodiments, collects operational information during data processing tasks; in response to the operational information triggering adjustment conditions, obtains the initial script data corresponding to the data processing task; converts the initial script data into tree-structured data; determines fault information based on the tree-structured data, the operational information, and a preset large language model; updates the initial script data based on the fault information, and determines target script data; and performs canary release based on the target script data. By collecting operational information and autonomously triggering the determination of initial script data requiring updates, updates can be detected and triggered in a timely manner, allowing for processing before performance bottlenecks occur. Combining the large language model to correct and update the initial script data, with automatic parsing and problem diagnosis, overcomes the coverage limitations of traditional rule engines, improving the accuracy and efficiency of updates; finally, canary release achieves end-to-end automation, improving the efficiency and reliability of data processing task script update processing, thereby enhancing the performance of the data warehouse.
[0071] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0072] Reference Figure 6 This invention also provides a vehicle, including a processor 601, a memory 602, and a computer program stored in the memory 602 and capable of running on the processor 601. When executed by the processor, the computer program implements the steps of the data processing task script processing method described above. The data processing task script processing method includes: Collect runtime information while performing data processing tasks; In response to the running information triggering the adjustment condition, the initial script data corresponding to the data processing task is obtained; Convert the initial script data into tree structure data; Based on the tree structure data, the operational information, and the preset large language model, fault information is determined; The initial script data is updated based on the fault information, and the target script data is determined. Canary release is performed based on the target script data.
[0073] Optionally, the method further includes: After the gray-scale release, collect monitoring information; By combining the monitoring information, the target script data, and the initial script data, a training sample set is determined; The preset large language model is trained based on the training sample set to update the preset large language model.
[0074] Optionally, the step of determining the training sample set by combining the monitoring information, the target script data, and the initial script data includes: Based on the monitoring information, the performance status is determined; When the performance status is in an upward trend, the target script data and the initial script data are associated to determine the training sample set.
[0075] Optionally, the method further includes: If the performance status is in a degraded state, a version rollback is performed based on the initial script data.
[0076] Optionally, the step of determining fault information based on the tree structure data, the operational information, and the preset large language model includes: Input information is generated by combining the tree structure data and the runtime information; The input information is input into a preset large language model; Obtain the fault location and the impact score corresponding to the fault location output by the preset large language model; By combining the fault location and the impact score, the fault information is determined.
[0077] Optionally, the step of updating the initial script data based on the fault information and determining the target script data includes: Based on the impact score, the fault locations are sorted in descending order to determine the set of fault locations; The first few fault locations in the fault location set are determined as the fault locations to be corrected. For any of the aforementioned fault locations to be corrected, the fault location to be corrected is mapped to the initial script data to determine the script configuration data; Determine the data type of the script configuration data; Based on the data type, the updated configuration data is determined by matching from a preset rule base; The script configuration data is replaced based on the updated configuration data to determine the target script data.
[0078] Optionally, the step of performing canary release based on the target script data includes: Perform a consistency test on the target script data to determine the verification status; If the verification status is "verification passed", a canary release will be performed based on the target script data.
[0079] This invention, in its embodiments, collects operational information during data processing tasks; in response to the operational information triggering adjustment conditions, obtains the initial script data corresponding to the data processing task; converts the initial script data into tree-structured data; determines fault information based on the tree-structured data, the operational information, and a preset large language model; updates the initial script data based on the fault information, and determines target script data; and performs canary release based on the target script data. By collecting operational information and autonomously triggering the determination of initial script data requiring updates, updates can be detected and triggered in a timely manner, allowing for processing before performance bottlenecks occur. Combining the large language model to correct and update the initial script data, with automatic parsing and problem diagnosis, overcomes the coverage limitations of traditional rule engines, improving the accuracy and efficiency of updates; finally, canary release achieves end-to-end automation, improving the efficiency and reliability of data processing task script update processing, thereby enhancing the performance of the data warehouse.
[0080] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0081] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0082] Reference Figure 7 This invention also provides a computer-readable storage medium 701, on which a computer program is stored. When a processor executes the computer program, it performs the data processing task script processing method as described in any one of the embodiments of this invention. The data processing task script processing method includes: Collect runtime information while performing data processing tasks; In response to the running information triggering the adjustment condition, the initial script data corresponding to the data processing task is obtained; Convert the initial script data into tree structure data; Based on the tree structure data, the operational information, and the preset large language model, fault information is determined; The initial script data is updated based on the fault information, and the target script data is determined. Canary release is performed based on the target script data.
[0083] Optionally, the method further includes: After the gray-scale release, collect monitoring information; By combining the monitoring information, the target script data, and the initial script data, a training sample set is determined; The preset large language model is trained based on the training sample set to update the preset large language model.
[0084] Optionally, the step of determining the training sample set by combining the monitoring information, the target script data, and the initial script data includes: Based on the monitoring information, the performance status is determined; When the performance status is in an upward trend, the target script data and the initial script data are associated to determine the training sample set.
[0085] Optionally, the method further includes: If the performance status is in a degraded state, a version rollback is performed based on the initial script data.
[0086] Optionally, the step of determining fault information based on the tree structure data, the operational information, and the preset large language model includes: Input information is generated by combining the tree structure data and the runtime information; The input information is input into a preset large language model; Obtain the fault location and the impact score corresponding to the fault location output by the preset large language model; By combining the fault location and the impact score, the fault information is determined.
[0087] Optionally, the step of updating the initial script data based on the fault information and determining the target script data includes: Based on the impact score, the fault locations are sorted in descending order to determine the set of fault locations; The first few fault locations in the fault location set are determined as the fault locations to be corrected. For any of the aforementioned fault locations to be corrected, the fault location to be corrected is mapped to the initial script data to determine the script configuration data; Determine the data type of the script configuration data; Based on the data type, the updated configuration data is determined by matching from a preset rule base; The script configuration data is replaced based on the updated configuration data to determine the target script data.
[0088] Optionally, the step of performing canary release based on the target script data includes: Perform a consistency test on the target script data to determine the verification status; If the verification status is "verification passed", a canary release will be performed based on the target script data.
[0089] This invention, in its embodiments, collects operational information during data processing tasks; in response to the operational information triggering adjustment conditions, obtains the initial script data corresponding to the data processing task; converts the initial script data into tree-structured data; determines fault information based on the tree-structured data, the operational information, and a preset large language model; updates the initial script data based on the fault information, and determines target script data; and performs canary release based on the target script data. By collecting operational information and autonomously triggering the determination of initial script data requiring updates, updates can be detected and triggered in a timely manner, allowing for processing before performance bottlenecks occur. Combining the large language model to correct and update the initial script data, with automatic parsing and problem diagnosis, overcomes the coverage limitations of traditional rule engines, improving the accuracy and efficiency of updates; finally, canary release achieves end-to-end automation, improving the efficiency and reliability of data processing task script update processing, thereby enhancing the performance of the data warehouse.
[0090] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0091] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0092] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0095] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.
[0096] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0097] The foregoing has provided a detailed description of a data processing task script processing method, a data processing task script processing device, an electronic device, and a computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A data processing task script processing method, characterized in that, include: Collect runtime information while performing data processing tasks; In response to the running information triggering the adjustment condition, the initial script data corresponding to the data processing task is obtained; Convert the initial script data into tree structure data; Based on the tree structure data, the operational information, and the preset large language model, fault information is determined; The initial script data is updated based on the fault information, and the target script data is determined. Canary release is performed based on the target script data.
2. The method according to claim 1, characterized in that, The method further includes: After the gray-scale release, collect monitoring information; By combining the monitoring information, the target script data, and the initial script data, a training sample set is determined; The preset large language model is trained based on the training sample set to update the preset large language model.
3. The method according to claim 2, characterized in that, The step of determining the training sample set by combining the monitoring information, the target script data, and the initial script data includes: Based on the monitoring information, the performance status is determined; When the performance status is in an upward trend, the target script data and the initial script data are associated to determine the training sample set.
4. The method according to claim 2, characterized in that, The method further includes: If the performance status is in a degraded state, a version rollback is performed based on the initial script data.
5. The method according to any one of claims 1-4, characterized in that, The step of determining fault information based on the tree structure data, the operational information, and the preset large language model includes: Input information is generated by combining the tree structure data and the runtime information; The input information is input into a preset large language model; Obtain the fault location and the impact score corresponding to the fault location output by the preset large language model; By combining the fault location and the impact score, the fault information is determined.
6. The method according to claim 5, characterized in that, The step of updating the initial script data based on the fault information and determining the target script data includes: Based on the impact score, the fault locations are sorted in descending order to determine the set of fault locations; The first few fault locations in the fault location set are determined as the fault locations to be corrected. For any of the aforementioned fault locations to be corrected, the fault location to be corrected is mapped to the initial script data to determine the script configuration data; Determine the data type of the script configuration data; Based on the data type, the updated configuration data is determined by matching from a preset rule base; The script configuration data is replaced based on the updated configuration data to determine the target script data.
7. The method according to any one of claims 1-4, characterized in that, The step of performing canary release based on the target script data includes: Perform a consistency test on the target script data to determine the verification status; If the verification status is "verification passed", a canary release will be performed based on the target script data.
8. A data processing task script processing device, characterized in that, include: The first acquisition module is used to collect operational information when performing data processing tasks; The acquisition module is used to acquire the initial script data corresponding to the data processing task in response to the running information triggering the adjustment condition; The conversion module is used to convert the initial script data into tree structure data; The fault module is used to determine fault information based on the tree structure data, the running information, and the preset large language model; The update module is used to update the initial script data based on the fault information and determine the target script data; The publishing module is used to perform canary releases based on the target script data.
9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when executed by the processor, the computer program implements the steps of the data processing task script processing method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the data processing task script processing method as described in any one of claims 1 to 7.