Optimization strategy generation method and device, equipment, storage medium and program product

Abstract

The application discloses an optimization strategy generation method and device, equipment, a storage medium and a program product, and belongs to the technical field of data processing. In the application, in response to an optimization request for a target periodic task, running feature data of the target periodic task is generated based on historical running data of the target periodic task; in an optimization case library, historical running feature data that meets specified similarity conditions with the running feature data is determined, the historical running feature data including an optimization strategy for a periodic task corresponding to the historical running feature data; and the running feature data and the historical running feature data are input into a generative language model to obtain a target optimization strategy for the target periodic task. The application improves the optimization efficiency of the target periodic task.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an optimization strategy generation method, apparatus, device, storage medium, and program product. Background Technology

[0002] In enterprise-level big data platforms, periodic scheduling tasks are widely used in various business processes. During long-term operation, the input data volume, data distribution characteristics, system load, and other execution environment factors of these tasks change over time, causing the originally suitable execution configuration to gradually deviate from its optimal state. At this point, operations and maintenance personnel need to use their experience to determine whether parameters need to be adjusted and, if so, which parameters to adjust.

[0003] However, because the tuning experience of different engineers is independent and cannot be shared, each time the operation and maintenance personnel are changed, it is equivalent to starting from scratch. The tuning experience cannot be accumulated and passed on, resulting in the repeated handling of the same or similar abnormal scenarios and low tuning efficiency. Summary of the Invention

[0004] This application provides an optimization strategy generation method, apparatus, device, storage medium, and program product, which can solve the problem of tuning efficiency. The technical solution is as follows: On the one hand, an optimization strategy generation method is provided, the method comprising: In response to an optimization request for a target periodic task, operational characteristic data of the target periodic task is generated based on the historical operational data of the target periodic task. In the optimization case library, historical operational feature data that meets specified similar conditions to the operational feature data are identified, and the historical operational feature data includes optimization strategies for periodic tasks corresponding to the historical operational feature data; The operational feature data and historical operational feature data are input into a generative language model to obtain a target optimization strategy for the target periodic task.

[0005] In one possible implementation, the historical runtime data includes runtime data of task instances generated each time the target periodic task is executed, and the runtime data of the task instances includes basic information data, resource usage data, stage execution data, structured query language feature data, and parameter configuration data.

[0006] In another possible implementation, generating the operational characteristic data of the target periodic task based on its historical operational data includes: The running data of the task instances generated by each execution of the target periodic task are statistically analyzed to obtain the running feature data.

[0007] In another possible implementation, the runtime characteristic data includes the target task identifier of the target periodic task; in the optimization case library, historical runtime characteristic data that meets specified similarity conditions with the runtime characteristic data are determined, including: In the optimized case library, a data set of historical operational feature data corresponding to the task identifier that is the same as the target task identifier is determined; In response to the fact that the number of elements in the data set is greater than a specified number, historical running feature data that meets the first specified similarity condition with the running feature data is determined in the data set.

[0008] In another possible implementation, the method further includes: In response to the number of elements in the dataset being less than or equal to a specified number, the feature vector of the running feature data is obtained, and in the optimization case library, historical running feature data that meets the second specified similarity condition with the running feature data is determined based on the feature vector.

[0009] In another possible implementation, the step of inputting the runtime feature data and historical runtime feature data into a generative language model to obtain a target optimization strategy for the target periodic task includes: The running feature data and historical running feature data are inserted into the prompt word template to obtain model prompt words. The model prompt words are used to instruct the generative language model to generate a target optimization strategy for the target periodic task based on the running feature data and historical running feature data. The model prompts are input into a generative language model to obtain a target optimization strategy for the target periodic task.

[0010] In another possible implementation, after obtaining the target optimization strategy for the target periodic task, the following is also included: The operational characteristic data of the target periodic task and the target optimization strategy are input into the effect prediction model to obtain the prediction result of the optimization effect of the target optimization strategy.

[0011] In another possible implementation, prior to responding to the optimization request for the target periodic task, the following is also included: Obtain the runtime performance data of multiple task instances of the target periodic task within a specified time window; In response to the operational metric data meeting the specified optimization trigger conditions, the optimization request is generated.

[0012] On the other hand, an optimization strategy generation apparatus is provided, the apparatus comprising: The first generation module is configured to generate runtime characteristic data of the target periodic task based on the historical runtime data of the target periodic task in response to an optimization request for the target periodic task. The determination module is configured to determine, in the optimization case library, historical running feature data that meets specified similar conditions to the running feature data, the historical running feature data including optimization strategies for the periodic tasks corresponding to the historical running feature data; The second generation module is configured to input the running feature data and historical running feature data into a generative language model to obtain a target optimization strategy for the target periodic task.

[0013] In one possible implementation, the historical runtime data includes runtime data of task instances generated each time the target periodic task is executed, and the runtime data of the task instances includes basic information data, resource usage data, stage execution data, structured query language feature data, and parameter configuration data.

[0014] In another possible implementation, the first generation module is used for: The running data of the task instances generated by each execution of the target periodic task are statistically analyzed to obtain the running feature data.

[0015] In another possible implementation, the runtime characteristic data includes the target task identifier of the target periodic task; the determining module is configured to: In the optimized case library, a data set of historical operational feature data corresponding to the task identifier that is the same as the target task identifier is determined; In response to the fact that the number of elements in the data set is greater than a specified number, historical running feature data that meets the first specified similarity condition with the running feature data is determined in the data set.

[0016] In another possible implementation, the determining module is used for: In response to the number of elements in the dataset being less than or equal to a specified number, the feature vector of the running feature data is obtained, and in the optimization case library, historical running feature data that meets the second specified similarity condition with the running feature data is determined based on the feature vector.

[0017] In another possible implementation, the second generation module is used for: The running feature data and historical running feature data are inserted into the prompt word template to obtain model prompt words. The model prompt words are used to instruct the generative language model to generate a target optimization strategy for the target periodic task based on the running feature data and historical running feature data. The model prompts are input into a generative language model to obtain a target optimization strategy for the target periodic task.

[0018] In another possible implementation, the second generation module is used for: The operational characteristic data of the target periodic task and the target optimization strategy are input into the effect prediction model to obtain the prediction result of the optimization effect of the target optimization strategy.

[0019] In another possible implementation, the first generation module is used for: Obtain the runtime performance data of multiple task instances of the target periodic task within a specified time window; In response to the operational metric data meeting the specified optimization trigger conditions, the optimization request is generated.

[0020] On the other hand, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the method described in any of the above.

[0021] On the other hand, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described in any of the preceding claims.

[0022] On the other hand, a computer program product is provided, including computer program instructions that, when run on a computer, cause the computer to perform the method described in any of the preceding claims.

[0023] The beneficial effects of the technical solution provided in this application are as follows: It generates operational feature data based on historical operational data, transforming unstructured personal experience into standardized data that can be stored and compared, thus solving the problem of experience not being able to be accumulated and passed on. Furthermore, it identifies historical operational feature data that meets specified similar conditions in the optimization case library, enabling target periodic tasks to be directly associated with verified strategies from similar historical scenarios, avoiding repetitive work starting from scratch after changing maintenance personnel. By inputting the operational feature data and historical data into a generative language model, and utilizing the model's ability to understand context, isolated historical strategies are adapted to the execution environment of the current task, improving optimization efficiency. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 2 This is a flowchart of the optimization strategy generation method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the optimization strategy generation device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0027] In enterprise-level big data platforms, periodic scheduling tasks (such as daily data synchronization, scheduled report generation, and periodic data cleaning) are widely used in various business processes. Over long periods, the input data volume, data distribution characteristics, system load, and other execution environment factors of these periodic tasks change, causing the originally suitable execution configuration to gradually deviate from its optimal state. For example, a daily data synchronization task initially processes 100,000 data entries, and the configured execution resources can run stably; however, as business grows, the data volume increases to 500,000 entries after six months, and the original configuration begins to frequently experience execution timeouts or insufficient resources.

[0028] In related technologies, operations and maintenance personnel typically use task management platforms to view basic metrics such as task execution time and resource consumption. When these metrics exceed preset thresholds, alarms are triggered. Then, manual intervention is required to log into the system, review task logs, and, based on personal experience, determine whether parameters need adjustment and, if so, which parameters to adjust. For example, if a periodic SQL (Structured Query Language) task slows down, engineers need to manually review execution logs, analyze resource usage, and then, based on experience, decide whether to increase memory or adjust parallelism.

[0029] However, because different engineers' tuning experiences are independent and cannot be shared, when the same scheduled job is repeatedly tuned by different personnel, each tuning is essentially starting from scratch, lacking a systematic accumulation and reuse of historical tuning cases. Specifically, Engineer A successfully solved a data skew problem for a task by increasing parallelism, but this tuning experience remains only in Engineer A's personal understanding and is not structured or recorded. When a similar problem occurs again three months later, if Engineer B is in charge, Engineer B will not know Engineer A's previous tuning strategy and its effects, requiring re-analysis, retrials, and potentially even misjudging it as insufficient memory and taking the wrong adjustment direction. This inability to accumulate and transfer tuning experience leads to the repeated handling of the same or similar abnormal scenarios, resulting in low tuning efficiency and difficulty in guaranteeing the accuracy of tuning strategies.

[0030] This disclosure provides an optimization strategy generation method. This method can be applied to a terminal. For example... Figure 1 As shown, the terminal may include a processor 110, a memory 120, and a communication component 130.

[0031] Processor 110 can be a central processing unit (CPU), graphics processing unit (GPU), microcontroller unit (MCU), accelerated processing unit (APU), neural processing unit (NPU), tensor processing unit (TPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processor (DSP), etc. Processor 110 can be used to run generative language models, etc.

[0032] Memory 120 may include volatile memory and / or non-volatile memory. Volatile memory may include random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, non-volatile random access memory (NVRAM), etc. Memory 120 may be used for storage optimization case libraries, etc.

[0033] The communication component 130 can be a wireless communication module (WCM), a subscriber identity module (SIM), a universal subscriber identity module (USIM), an optical network unit (ONU), etc. The communication component 130 can be used for communication between the terminal and a server, or other terminals.

[0034] This application provides a method for generating optimization strategies, such as... Figure 2 As shown, in some embodiments, the method includes: S201. In response to an optimization request for a target periodic task, generate operational characteristic data of the target periodic task based on the historical operational data of the target periodic task.

[0035] In practice, a unique task identifier for the target periodic task is obtained through a task scheduling platform. Based on this task identifier, the execution data of all task instances for that task within a historical time period is retrieved from the historical execution data storage. Historical execution data includes, but is not limited to, information such as the execution duration, resource consumption, execution status, and input data volume of each task instance generated during execution. Subsequently, feature extraction and statistical processing are performed on the acquired historical execution data to generate execution feature data for the target periodic task. This execution feature data is a structured representation of the recent execution status of the target periodic task, reflecting its current execution performance level and operational mode characteristics.

[0036] S202. In the optimization case library, determine historical operating feature data that meets specified similar conditions to the operating feature data, wherein the historical operating feature data includes optimization strategies for periodic tasks corresponding to the historical operating feature data.

[0037] In practice, after obtaining the operational feature data, historical operational feature data that meets specified similarity conditions is retrieved from the optimization case library. The optimization case library stores a large amount of historical operational feature data, each associated with a corresponding optimization strategy. This strategy represents parameter adjustments or process optimizations implemented for the periodic task corresponding to that historical operational feature data. During retrieval, based on preset similarity matching rules, historical operational feature data that meets similarity conditions in the performance feature dimension to the current operational feature data is searched in the optimization case library. The retrieval strategy includes two priority levels: the first priority is precise matching retrieval based on the same task identifier, i.e., prioritizing retrieval from historical cases with the same task identifier as the target periodic task; the second priority is cross-task retrieval based on feature vector similarity, expanding to retrieval of cases with similar features in other task identifiers when the number of historical cases with the same task identifier is insufficient. The recalled historical operational feature data is sorted by similarity, and each case includes the performance characteristics before optimization, the changes in optimization parameters, and the actual execution effect after optimization.

[0038] S203. Input the operational feature data and historical operational feature data into a generative language model to obtain a target optimization strategy for the target periodic task.

[0039] Generative language models refer to deep learning models trained on large-scale corpora that possess natural language understanding and generation capabilities, and can receive structured input data and generate text-based output results according to instructions.

[0040] In practice, runtime feature data and retrieved historical runtime feature data are input into the generative language model. After receiving this structured data, the generative language model combines its pre-trained knowledge of performance optimization for distributed computing frameworks with a comprehensive analysis of the current task's runtime status and optimization experience from similar historical cases to generate a target optimization strategy for the target periodic task. This target optimization strategy is a structured output, including but not limited to bottleneck location analysis, specific parameter adjustment suggestions and their magnitudes, data processing flow rewriting suggestions, estimated optimization benefits, and suggested priorities.

[0041] Take the daily data synchronization task in an enterprise-level big data platform as an example. Assume the target periodic task is the "daily synchronization of user behavior data" task, and its task identifier is schedule_001.

[0042] In response to the optimization request for this task, the runtime data of all task instances within the last 90 days was retrieved from the historical runtime data storage. Specifically, the retrieved historical runtime data includes: the task has been executed 270 times since its launch, generating a total of 270 task instances. The execution time of the most recent instance has increased from 15 minutes initially to 52 minutes currently; the Executor resource usage is 10 Executors, each with 4 cores and 16GB of memory; the data exchange read / write volume has increased from 5GB initially to 25GB currently; and the garbage collection time percentage has increased from 5% to 18%.

[0043] Feature extraction and statistical processing were performed on the above historical operation data to generate operation feature data, which includes: task identifier schedule_001, execution time P50 (50th percentile) of 45 minutes, execution time P95 (95th percentile) of 55 minutes, median number of Executors of 10, average Shuffle (data exchange) write data volume of 25GB with continuously increasing variance, GC (Garbage Collection) time accounting for 18%, data skewness (max (maximum) time / median (median) time) of 6.2, and the number of input records showing a linear growth trend.

[0044] Subsequently, a serial dual-priority search was performed in the optimized case library. First, the first-priority search was executed: 5 valid cases were found in the historical cases identified as task_001. Since 5 cases are greater than the specified number 3, the number of cases retrieved by the first-priority search is sufficient, and the second-priority cross-task search is not executed. Among these 5 cases from the same task, the 3 most similar cases that meet the first specified similarity condition with the currently running feature data were selected. Case 1: 60 days ago, when the data volume increased to 20GB, shuffle.partitions was adjusted from 200 to 600, reducing execution time by 30%; Case 2: 45 days ago, to address the data skew issue, a broadcast hint was added to the dim_user table, reducing Stage 3 time by 50%; Case 3: 30 days ago, increasing the Executor memory to 20GB reduced the GC time percentage from 15% to 8%.

[0045] The above runtime characteristic data and three historical runtime characteristic data points were input into the generative language model. The specific model prompt was: "You are a Spark job performance tuning expert. Based on the following information, please analyze the performance bottleneck of the current scheduled job and provide optimization suggestions." The current scheduled job performance profile is as follows: Task identifier is schedule_001, task name is daily synchronization of user behavior data, execution time is 45 minutes for P50 and 55 minutes for P95, Executor configuration is 10, each with 4 cores and 16GB, average Shuffle write data volume is 25GB with continuously increasing variance, GC time accounts for 18%, Stage 3 data skew (max / median time ratio) is 6.2, the number of input records has shown a linear growth trend over the past 90 days, currently around 500,000 records / day, and the current parameter configuration is spark.sql.shuffle.partitions=200, spark.executor.memory=16g, spark.sql.adaptive.enabled=false. Historical optimization examples are as follows: Example 1 is a case from 60 days ago with the same task schedule_001. Before optimization, the characteristics were 20GB of Shuffle write data and shuffle.partitions=200. The optimization operation was to adjust spark.sql.shuffle.partitions from 200 to 600, resulting in a reduction in execution time from 50 minutes to 35 minutes, a 30% reduction. Example 2 is a case from 45 days ago with the same task schedule_001. Before optimization, the characteristics were Stage 3 data skew of 8.5 and the dim_user table participating in JOIN. The optimization operation was to add a broadcast hint to the dim_user table, resulting in a Stage 3 data skew of 8.5. Case 3 reduced the time from 25 minutes to 12 minutes, a reduction of 52%; Case 3 is a case of the same task schedule_001 30 days ago. Before optimization, the characteristics were that GC time accounted for 15% and Executor memory was 16GB. The optimization operation was to adjust spark.executor.memory from 16g to 20g. The optimization effect was that the GC time accounted for 8% from 15%.Spark best practices are as follows: The `shuffle.partitions` parameter should be related to the amount of data processed by a single Task; it is recommended that the amount of shuffle data processed by a single Task should not exceed 128MB. Broadcast hints are suitable for JOIN scenarios involving large and small tables, with a default threshold of 10MB for small tables. If GC time accounts for more than 10% of total execution time, it is recommended to increase Executor memory or adjust memory management parameters. AQE (Adaptive Query Execution) is suitable for scenarios with large fluctuations in data volume and can dynamically adjust the execution plan based on intermediate results at runtime. Output should be structured as follows: bottleneck location and confidence level, recommended parameter adjustments and adjustment ranges (including values ​​before and after adjustment), estimated benefits, and recommended priority (high / medium / low).

[0046] Based on the above inputs, the optimization strategy for the output target of the generative language model is as follows: The bottlenecks are identified as Stage 3 data skew (90% confidence), insufficient Shuffle parallelism (85% confidence), and excessive GC time (80% confidence). Parameter adjustments are recommended: increase spark.sql.shuffle.partitions from 200 to 800, increase spark.executor.memory from 16g to 20g, and add BROADCAST hints to the dim_user table. The estimated benefits are: execution time is expected to be reduced from 52 minutes to 32 minutes, a reduction of approximately 38%; Shuffle data distribution uniformity is improved, and data skew is expected to decrease from 6.2 to below 2.0; GC time is expected to decrease from 18% to below 8%. The recommended priority is high.

[0047] In this embodiment, optimization knowledge scattered in the personal experience of individual engineers is transformed into structured and searchable cases. This makes the generation of optimization strategies for target periodic tasks no longer entirely dependent on the personal experience of a single engineer, realizing the reusability and cross-personnel transfer of optimization experience, and improving the accuracy and efficiency of optimization strategies.

[0048] In some embodiments, the historical running data includes running data of task instances generated each time the target periodic task is executed, and the running data of the task instances includes basic information data, resource usage data, stage execution data, structured query language feature data, and parameter configuration data.

[0049] Specifically, the historical execution data of the target periodic task includes the execution data of the task instances generated each time the target periodic task is executed. The execution data of each task instance covers five dimensions: basic information data, resource usage data, stage execution data, structured query language feature data, and parameter configuration data.

[0050] The basic information data includes the system's collection of basic operational information for each task instance, such as task identifier, task instance identifier, execution start time, execution end time, execution status (success / failure / timeout), and input data volume. It also summarizes metrics such as execution success rate, average execution time, maximum execution time, and number of reruns based on multiple task instances under the same task identifier. Resource usage data includes information on the computing resources allocated and used by each task instance during execution, including but not limited to the number of Executors, the number of cores and memory configuration for each execution process, the amount of data read / write during Shuffle, the average time and percentage of GC, and the amount of data spilled. This resource usage data reflects the actual consumption of cluster resources by the task instance. Stage execution data involves dividing the execution process of each task instance into multiple Stages according to data exchange boundaries. For each stage, the system collects its execution time, the number of Tasks included, the number of input / output records for each task, the amount of data exchanged (read / write), and the degree of data skew between tasks (represented by the ratio of the maximum to the median time for each task). This stage execution data is used to identify which specific stage in the task instance's execution process has a performance bottleneck. The Structured Query Language (SCL) feature data includes static analysis of the SCL scripts corresponding to the task instances, extracting feature information such as, but not limited to, the number of tables involved, the number and size of rows in each table, the type and order of JOIN operations (inner join, outer join, Cartesian product, etc.), the number of dimensions in GroupBy and the type of aggregation operations, the window size and partitioning method of window functions, and whether UDFs (User Defined Functions) are included. This SCL feature data characterizes the data processing logic complexity of the task instances. The parameter configuration data includes snapshots of all runtime parameter configurations used by each task instance during execution, including the current values ​​and system default values ​​of each parameter, facilitating subsequent comparative analysis of the effects of parameter adjustments.

[0051] Taking a specific task instance (execution identifier: task_20260701_001) of the target periodic task schedule_001 as an example, the runtime data collected by the system includes: Basic information data: Task ID: schedule_001, Task instance ID: task_20260701_001, Execution start time: 2026-07-01 02:00:00, Execution end time: 2026-07-01 02:52:18, Execution status: Successful, Input data volume: 500,000 records. Summary statistics for the 270 task instances under this task ID: Execution success rate: 98.5%, Execution time P50: 45 minutes, Execution time P95: 55 minutes, Number of retries: 3.

[0052] Resource usage data: 10 Executors, each Executor is configured with 4 cores and 16GB of memory, Shuffle read data volume is 22GB, Shuffle write data volume is 25GB, GC average time consumption is 18%, and disk spill volume is 1.2GB.

[0053] Stage execution data: This task instance is divided into 5 stages. Stage 0 took 2 minutes, contained 200 tasks, and had no data skew; Stage 1 took 8 minutes, contained 200 tasks, and had a data skew of 1.5; Stage 2 took 12 minutes, contained 600 tasks, and had a data skew of 2.1; Stage 3 took 25 minutes, contained 200 tasks, and had a data skew of 6.2 (some tasks took up to 45 minutes, while the median task took only 7 minutes); Stage 4 took 5 minutes, contained 200 tasks, and had a data skew of 1.2.

[0054] Structured Query Language (SCL) feature data: This involves 4 tables (user_behavior table 5 million rows / 2GB, dim_user table 50,000 rows / 8MB, dim_product table 100,000 rows / 15MB, order_detail table 3 million rows / 1.5GB); JOIN types are user_behavior LEFT JOIN dim_user, INNER JOIN dim_product, and INNER JOIN order_detail; GroupBy has 3 dimensions (date, user type, product category); it includes 1 window function (ROW_NUMBER (row number function), partitioned by date and sorted by amount); and it includes 1 custom function for data format conversion.

[0055] Parameter configuration data: spark.sql.shuffle.partitions=200, spark.executor.memory=16g, spark.executor.cores=4, spark.sql.adaptive.enabled=false, spark.sql.broadcastTimeout=300.

[0056] This embodiment refines historical operational data into five dimensions: basic information data at the task instance level, resource usage data, stage execution data, structured query language feature data, and parameter configuration data. This enables a comprehensive depiction of the operational status of the target periodic task, providing a sufficient data foundation for subsequent generation of operational feature data and retrieval of historical optimization cases. This allows the generation of optimization strategies to comprehensively consider multiple factors such as resource usage, execution stage bottlenecks, structured query language structural features, and parameter configuration, thereby improving the targeting and accuracy of the optimization strategies.

[0057] In some embodiments, generating operational characteristic data of the target periodic task based on its historical operational data includes: The running data of the task instances generated by each execution of the target periodic task are statistically analyzed to obtain the running feature data.

[0058] Specifically, for multiple task instances under the same task identifier, pre-defined statistical indicators are calculated for the operational data of each dimension. These statistical indicators include, but are not limited to: mean, median, P50, P95, variance, standard deviation, and trend slope. Through feature statistics, the raw operational data of multiple task instances are transformed into structured feature data that reflects the overall operational status and changing trends of the task.

[0059] The feature statistics process includes, for the basic information data dimension, counting the total number of task instances, execution success rate, median execution time and P95, number of reruns, etc.; for the resource usage data dimension, taking the median of the number of Executors, cores, and memory configuration for each task instance, and calculating the mean and variance of Shuffle read / write data volume, GC time ratio, and Spill volume; for the stage execution data dimension, calculating the P50 and P95 for the execution time of each Stage, and calculating the maximum and mean of the data skew for each Task; for the structured query language feature data dimension, extracting the fixed features (table information, JOIN type, number of GroupBy dimensions, etc.) of the SQL script corresponding to the task identifier; and for the parameter configuration data dimension, recording a snapshot of all currently used parameter configurations.

[0060] Taking the target periodic task schedule_001 as an example, this task was executed 90 times in the last 90 days, generating 90 task instances. The system performs feature statistics on the runtime data of these 90 task instances: Basic information dimensions: Total number of task instances: 90; Execution success rate: 98.9% (89 successes / 1 failure); Execution time P50: 45 minutes; Execution time P95: 55 minutes; Number of reruns: 3.

[0061] Resource usage dimensions: median number of Executors: 10; median number of cores: 4; median memory configuration: 16GB; average Shuffle read data volume: 22GB; average Shuffle write data volume: 25GB with a variance of 3.2 (showing an increasing trend); average GC time consumption: 18%; average Spill volume: 1.2GB.

[0062] Stage execution dimensions: Stage 3 time is 20 minutes for P50 and 30 minutes for P95; Stage 3 contains 200 Tasks; the average data skew of Stage 3 (max time / median time) is 6.2, showing an upward trend (4.0 30 days ago, currently 6.2).

[0063] Structured Query Language features: Involves 4 tables, 3 JOIN operations, 3 GroupBy dimensions, 1 window function, and 1 UDF. These features have not changed in the last 90 days.

[0064] Parameter configuration dimensions: The current parameter configuration is spark.sql.shuffle.partitions=200, spark.executor.memory=16g, etc., which has not changed compared to 30 days ago.

[0065] This embodiment performs feature statistics on the running data of task instances generated by each execution of the target periodic task, and aggregates the original running data of multiple task instances into structured running feature data. This eliminates the interference of fluctuations in the execution of a single task instance, enabling the running feature data to stably and accurately reflect the overall running status and changing trend of the target periodic task, and providing a reliable feature foundation for subsequent similar case retrieval in the optimization case library.

[0066] In some embodiments, the operational feature data includes the target task identifier of the target periodic task; in the optimization case library, historical operational feature data that meets specified similarity conditions with the operational feature data is determined, including: In the optimized case library, a data set of historical operational feature data corresponding to the same task identifier as the target task identifier is determined; in response to the number of elements in the data set being greater than a specified number, historical operational feature data that satisfies the first specified similarity condition with the operational feature data is determined in the data set.

[0067] The runtime feature data includes the target task identifier for the target periodic task. When retrieving historical runtime feature data from the optimization case library that meets specified similarity conditions to the runtime feature data, a dual-priority retrieval strategy can be adopted. The first priority refers to identifying the set of historical runtime feature data corresponding to task identifiers identical to the target task identifier in the optimization case library. Since periodic tasks corresponding to the same task identifier have the same SQL scripts and parameter configurations, their historical runtime feature data is the most direct and relevant reference. First, all historical runtime feature data identical to the target task identifier are filtered from the optimization case library to form a data set. Then, it is determined whether the number of elements in the data set is greater than a specified number. The specified number is a pre-set threshold, such as 3 entries. When the number of elements in the data set is greater than the specified number, it indicates that sufficient historical optimization cases have been accumulated under that task identifier. At this point, historical runtime feature data that meets the first specified similarity condition is identified in the data set. The first specified similarity condition is based on the similarity of feature values ​​in various dimensions of the runtime feature data, including but not limited to: the similarity of Shuffle data volume, the similarity of data skew, the similarity of GC time consumption ratio, and the similarity of Stage time distribution. Sort the dataset from highest to lowest similarity and select the historical operational feature data that meets the similarity threshold or the top N data as the search results.

[0068] Taking the target periodic task schedule_001 as an example, its target task identifier is schedule_001. The historical execution characteristic data of task identifier schedule_001 is retrieved from the optimization case library, and 8 valid cases (more than the specified number 3) are found, forming a dataset. The 8 historical cases are as follows: Case A (60 days ago): Shuffle write data volume 20GB, data skew 5.0, GC ratio 15%.

[0069] Case B (45 days ago): Shuffle write data volume 22GB, data skew 8.5, GC percentage 16%.

[0070] Case C (30 days ago): Shuffle write data volume 23GB, data skew 4.0, GC ratio 15%.

[0071] Case D (25 days ago): Shuffle write data volume 24GB, data skew 5.5, GC percentage 17%.

[0072] Case E (20 days ago): Shuffle write data volume 25GB, data skew 6.0, GC percentage 18%.

[0073] Case F (15 days ago): Shuffle write data volume 26GB, data skew 6.5, GC percentage 17%.

[0074] Case G (10 days ago): Shuffle write data volume 25GB, data skew 6.2, GC percentage 18%.

[0075] Case H (5 days ago): Shuffle write data volume 27GB, data skew 6.8, GC percentage 19%.

[0076] The current running feature data is as follows: Shuffle write data volume 25GB, data skewness 6.2, and GC ratio 18%. Based on the weighted distance of feature values ​​of each dimension, similarity is calculated. Case E (20 days ago), Case G (10 days ago), and Case F (15 days ago) are identified as the most similar to the current running feature data (data volume, skewness, and GC ratio are all close), and are thus selected as historical running feature data that meet the first specified similarity condition.

[0077] This embodiment implements a retrieval strategy that prioritizes utilizing the historical optimization experience of the target periodic task itself. This is achieved by identifying historical operational feature data in the dataset that meets a first specified similarity condition to the operational feature data when the number of elements in the dataset exceeds a specified number. Since the SQL scripts and parameter configurations corresponding to the same task identifier are completely identical, the optimization strategies and effects in its own historical cases have the highest reference value, thereby improving the relevance of the retrieved historical operational feature data and the accuracy of the target optimization strategy.

[0078] In some embodiments, the method further includes: In response to the number of elements in the dataset being less than or equal to a specified number, the feature vector of the running feature data is obtained, and in the optimization case library, historical running feature data that meets the second specified similarity condition with the running feature data is determined based on the feature vector.

[0079] In practice, when the number of elements in the dataset is less than or equal to a specified number, it indicates that the historical optimization cases under the target task identifier are insufficient to support effective similarity matching (e.g., the task is a newly launched task, or has had few historical executions). In this case, the feature vectors of the running feature data are obtained, and historical running feature data that meets the second specified similarity condition is determined based on these feature vectors in the optimization case library. Specifically, the running feature data is encoded into fixed-length feature vectors using a pre-trained embedding model. These feature vectors characterize the comprehensive performance features of the running feature data in the vector space, and the distance between vectors reflects the degree of similarity between different running feature data. In the full set of historical running feature data in the optimization case library (not limited to the same task identifier), the similarity (e.g., cosine similarity) between the feature vectors of each historical running feature data and the feature vector of the current running feature data is calculated. Historical running feature data with similarity greater than a preset threshold or ranked in the top K are identified as historical running feature data that meet the second specified similarity condition.

[0080] Taking the target periodic task schedule_002 (a newly launched weekly data cleaning task) as an example, its target task identifier is schedule_002. Searching the optimization case library for historical runtime feature data of task identifier schedule_002 only yielded 2 valid cases (less than or equal to the specified number of 3), indicating insufficient elements in the dataset. The feature vector of the current runtime feature data is obtained. This feature vector is a 768-dimensional floating-point vector, encoded by an embedding model using SQL structure features (including the number of tables, JOIN type, GroupBy dimension, etc.), data volume features (number of input records, Shuffle read / write volume, etc.), stage time distribution features (stage time P50 / P95, data skew, etc.), and resource configuration features (number of Executors, memory configuration, etc.). Vector similarity is then performed on the full historical runtime feature data in the optimization case library, calculating the cosine similarity between the current feature vector and each historical feature vector. The top-ranked cases in the search results include: The daily data synchronization task with task identifier schedule_015 (cosine similarity 0.92) has a similar SQL structure and data volume growth pattern.

[0081] The weekly report generation task with task identifier schedule_023 (cosine similarity 0.87) has a similar stage time distribution and data skew pattern.

[0082] The daily ETL (Extract, Transform, Load) task with task identifier schedule_008 (cosine similarity 0.85) has similar resource configuration and Shuffle data volume characteristics.

[0083] The above three historical operational feature data were identified as retrieval results that met the second specified similarity condition and were used as input references for the generative language model.

[0084] This embodiment obtains the feature vector of the running feature data when the number of elements in the dataset is less than or equal to a specified number, and determines the historical running feature data that meets the second specified similarity condition in the optimization case library based on the feature vector. This realizes the automatic expansion to similar case retrieval across task identifiers when the target task itself lacks historical cases, ensuring the feasibility of generating optimization strategies, avoiding the problem of not being able to generate effective optimization strategies due to insufficient historical data, and improving the robustness and applicability of the system.

[0085] In some embodiments, inputting the runtime feature data and historical runtime feature data into a generative language model to obtain a target optimization strategy for the target periodic task includes: The operational feature data and historical operational feature data are inserted into the prompt word template to obtain model prompt words. The model prompt words are used to instruct the generative language model to generate a target optimization strategy for the target periodic task based on the operational feature data and historical operational feature data. The model prompt words are then input into the generative language model to obtain the target optimization strategy for the target periodic task.

[0086] In practice, running feature data and historical running feature data are inserted into a prompt word template to obtain model prompt words. The prompt word template is a predefined text framework containing the following areas: a task description area, used to describe the task background that needs optimization; a current feature filling area, used to fill in the specific content of the running feature data; a historical case filling area, used to fill in the retrieved historical running feature data and its associated optimization strategies and execution effects; a domain knowledge area, used to fill in best practice rules or tuning guidelines related to the task type; and an output instruction area, used to instruct the generative language model to output optimization strategies according to a preset structured format. Running feature data is filled into the current feature filling area, historical running feature data (including corresponding optimization strategies and effect data) is filled into the historical case filling area, and domain knowledge is filled into the domain knowledge area, thus assembling a complete model prompt word. The model prompt word is used to instruct the generative language model to generate a target optimization strategy for the target periodic task based on the running feature data and historical running feature data. Subsequently, the model prompts are input into the generative language model. Based on its pre-trained natural language understanding capabilities and domain knowledge, the generative language model comprehensively analyzes the current operating characteristics and historical optimization experience, and generates a target optimization strategy for the target periodic task according to the structured format required by the output instruction area.

[0087] Taking the target periodic task schedule_001 as an example, the constructed prompt template contains the following: You are a {domain role}. Based on the following information, please analyze the performance bottleneck of the current {task type} and provide optimization suggestions. The current {task type} performance profile section is used to fill in {current running characteristic data}, the historical optimization case section is used to fill in {historical running characteristic data and corresponding optimization strategies and effects}, and the {domain} best practice reference section is used to fill in {domain knowledge}. The output requirements section indicates that the model should output in the following structure: bottleneck location and confidence level, suggested adjustment parameters and adjustment range (including values ​​before and after adjustment), SQL rewriting suggestions (if applicable), estimated benefits, and suggestion priority (high / medium / low).

[0088] After filling in the runtime feature data and historical runtime feature data, the model prompt message is: You are a Spark job performance tuning expert. Based on the following information, please analyze the performance bottleneck of the current scheduled job and provide optimization suggestions. The current scheduled job performance profile is as follows: The task identifier is schedule_001, the task name is daily synchronization of user behavior data, the execution time is 45 minutes for P50 and 55 minutes for P95, the Executor configuration is 10, each with 4 cores and 16GB, the average Shuffle write data volume is 25GB and the variance is continuously increasing, the GC time accounts for 18%, the Stage 3 data skew (max / median time ratio) is 6.2, the number of input records has shown a linear growth trend over the past 90 days and is currently about 500,000 records / day, and the current parameter configuration is spark.sql.shuffle.partitions=200, spark.executor.memory=16g, spark.sql.adaptive.enabled=false. Historical optimization examples are as follows: Example 1 is a case from 20 days ago for the same task schedule_001. Before optimization, the characteristics were 25GB of Shuffle write data, a data skew of 6.0, and a GC ratio of 18%. The optimization operation was to adjust spark.sql.shuffle.partitions from 200 to 400, resulting in a 20% reduction in execution time from 50 minutes to 40 minutes. Example 2 is a case for a similar task schedule_015. Before optimization, the characteristics were large data volume fluctuations and AQE not enabled. The optimization operation was to set spark.sql.adaptive.enabled to true, resulting in a 40% reduction in execution time fluctuations. Spark best practices are as follows: shuffle.partitions should be related to the amount of data processed by a single Task, and it is recommended that the amount of Shuffle data processed by a single Task should not exceed 128MB. Broadcast hints are suitable for large and small table JOIN scenarios. If the GC time ratio exceeds 10%, it is recommended to increase Executor memory. AQE is suitable for scenarios with large data volume fluctuations. The output should be structured as follows: bottleneck location and confidence level, suggested adjustment parameters and adjustment range (including values ​​before and after adjustment), SQL rewriting suggestions (if applicable), estimated benefits, and suggested priority (high / medium / low).

[0089] Generative language models output structured target optimization strategies based on the aforementioned model prompts.

[0090] This embodiment transforms structured operational data and historical cases into text input that a generative language model can effectively understand and process. This allows the generative language model to fully combine the current operational status and historical optimization experience for comprehensive analysis, thereby improving the quality and relevance of the generated target optimization strategy.

[0091] In some embodiments, after obtaining the target optimization strategy for the target periodic task, the method further includes: The operational characteristic data of the target periodic task and the target optimization strategy are input into the effect prediction model to obtain the prediction result of the optimization effect of the target optimization strategy.

[0092] The performance prediction model is trained based on actual performance data from historical optimization cases. During the training phase, multiple historical optimization cases are extracted from the optimization case library. Each case includes pre-optimization operational characteristics, optimization operations, and post-optimization performance data (such as the percentage change in execution time and resource consumption). The performance prediction model learns the mapping relationship between operational characteristics and optimization operations and the performance data. This model can be implemented using weighted regression based on similar cases or a machine learning regression model. During the prediction phase, the operational characteristics of the target periodic task and the target optimization strategy (including specific parameter adjustments and their magnitudes) are input into the performance prediction model. Based on the mapping relationship learned during training, the model outputs prediction results, including but not limited to: the estimated percentage change in execution time, the estimated change in resource consumption, the estimated absolute value of execution time, and the corresponding confidence interval.

[0093] Specifically, firstly, historical tuning operations similar to those adjusted in the current target optimization strategy are retrieved from the optimization case library, and the mean and variance of the actual effect data of these historical tuning operations (such as the percentage reduction in execution time) are calculated. Then, the impact of parameter adjustment is estimated stage by stage by combining the Stage time distribution of the current target periodic task (for example, increasing shuffle.partitions will reduce the amount of data exchanged by a single Task, thereby shortening the time of the data exchange stage, but it will increase scheduling overhead and metadata operation time). Finally, the estimation results of each stage are combined to output the estimated execution time and confidence interval.

[0094] Taking the target periodic task schedule_001 as an example, the target optimization strategy includes: adjusting spark.sql.shuffle.partitions from 200 to 800, adjusting spark.executor.memory from 16g to 20g, and enabling the AQE mechanism.

[0095] The feature data and target optimization strategy are input into the performance prediction model. The performance prediction model first retrieves the actual effects of historically similar optimization operations from the optimization case library: In the historical case studies, 12 cases involved adjusting `shuffle.partitions` from 200 to 400 or 600, with an average execution time reduction of 28% and a standard deviation of 8%. In the historical case studies, 8 cases involved adjusting Executor memory from 16GB to 20GB, with an average reduction in GC time percentage of execution time of 7 percentage points and a standard deviation of 2 percentage points. In the historical case studies, 5 cases involved enabling AQE, with an average reduction in execution time fluctuation of 35%.

[0096] The performance prediction model, based on the historical performance data and the current task's Stage time distribution, performs a stage-by-stage breakdown estimation: Shuffle Stage (Stage 3): After increasing shuffle.partitions from 200 to 800, the amount of data processed by a single Task decreases from 125MB to 31MB, and the expected Stage time is reduced from 25 minutes to 15 minutes. Memory-related optimizations: With increased Executor memory, the GC time percentage is expected to decrease from 18% to 9%, resulting in an estimated global time reduction of approximately 3 minutes. Enabling AQE and dynamic partitioning adjustments is expected to further reduce the Shuffle stage time by approximately 2 minutes.

[0097] Final output prediction results: The estimated execution time was reduced from 52 minutes to 30 minutes (a reduction of approximately 42%), with a confidence interval of [26 minutes, 36 minutes] (95% confidence level).

[0098] This embodiment inputs the operational characteristic data of the target periodic task and the target optimization strategy into the effect prediction model to obtain the prediction result of the optimization effect of the target optimization strategy. This allows users to know the expected optimization effect before executing the target optimization strategy, reducing the trial and error cost of parameter adjustment.

[0099] In some embodiments, prior to responding to an optimization request for a target periodic task, the method further includes: Obtain operational metric data for multiple task instances of the target periodic task within a specified time window; generate the optimization request in response to the operational metric data meeting the specified optimization triggering conditions.

[0100] Specifically, the process acquires runtime metrics data for multiple task instances of the target periodic task within a specified time window. The specified time window is a pre-defined time range, such as the last 7 days, the last 14 days, or the last 30 days. The runtime metrics data are the execution performance indicators of multiple task instances, including but not limited to execution time, resource utilization, execution success status, and input data volume for each task instance. The acquired runtime metrics data is analyzed to determine if it meets specified optimization trigger conditions. These specified optimization trigger conditions are a pre-defined set of rules used to determine if a task has performance issues requiring optimization, including but not limited to combinations of one or more of the following conditions: the average execution time exceeds a preset absolute threshold; the execution time shows a continuous upward trend (e.g., determined by the slope of linear regression exceeding a preset threshold); the variance of the execution time exceeds a preset threshold (indicating decreased execution stability); the execution failure rate exceeds a preset threshold; resource utilization is abnormal (e.g., the GC time percentage continuously exceeds a preset threshold); the increase in input data volume exceeds a preset threshold without corresponding parameter adjustments. When the runtime metrics data meets any of the specified optimization trigger conditions, an optimization request is automatically generated, triggering the optimization strategy generation process for the target periodic task. The optimization request includes the task identifier of the target periodic task, the type of triggering condition, and related metric data.

[0101] Taking the target periodic task schedule_001 as an example, the specified time window is set to the most recent 14 days. The specified optimization trigger conditions include: execution time P50 exceeding 40 minutes, execution time showing an upward trend with a linear regression slope greater than 0.5 minutes / day, and GC time accounting for more than 15%. The runtime metrics data for 14 task instances of schedule_001 in the most recent 14 days are obtained: execution time P50 is 45 minutes (exceeding the threshold of 40 minutes), meeting the trigger conditions. The execution times P50 for the most recent 14 days are: 38, 39, 40, 41, 42, 43, 43, 44, 44, 45, 45, 46, 46, and 47 minutes, with a linear regression slope of 0.62 minutes / day (greater than the threshold of 0.5 minutes / day), meeting the trigger conditions. The GC time accounting for 18% is 18% (exceeding the threshold of 15%), meeting the trigger conditions. Since the runtime metrics data simultaneously meet three specified optimization trigger conditions, an optimization request is automatically generated. This request includes: task identifier `schedule_001`, trigger condition type (execution time exceeded limit + execution time increasing trend + GC time exceeded limit), and relevant runtime metrics data. Based on this optimization request, the optimization strategy generation process then begins. Alternatively, users can manually initiate optimization requests through the task management platform, without waiting for automatic system triggering.

[0102] This embodiment can proactively detect and trigger optimization in the early stages of performance degradation, effectively reducing the risk of business impact caused by performance issues and improving the timeliness and proactivity of task operation and maintenance.

[0103] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0104] Based on the same inventive concept, corresponding to the optimization strategy generation method provided in the embodiments of this application, this application also provides an optimization strategy generation apparatus.

[0105] refer to Figure 3 The optimization strategy generation device includes: The first generation module 301 is configured to generate operational characteristic data of the target periodic task based on the historical operational data of the target periodic task in response to an optimization request for the target periodic task. The determination module 302 is configured to determine, in the optimization case library, historical running feature data that meets specified similar conditions to the running feature data, the historical running feature data including optimization strategies for the periodic tasks corresponding to the historical running feature data; The second generation module 303 is configured to input the running feature data and historical running feature data into a generative language model to obtain a target optimization strategy for the target periodic task.

[0106] In one possible implementation, the historical runtime data includes runtime data of task instances generated each time the target periodic task is executed, and the runtime data of the task instances includes basic information data, resource usage data, stage execution data, structured query language feature data, and parameter configuration data.

[0107] In another possible implementation, the first generation module is used for: The running data of the task instances generated by each execution of the target periodic task are statistically analyzed to obtain the running feature data.

[0108] In another possible implementation, the runtime characteristic data includes the target task identifier of the target periodic task; the determining module 302 is configured to: In the optimized case library, a data set of historical operational feature data corresponding to the task identifier that is the same as the target task identifier is determined; In response to the fact that the number of elements in the data set is greater than a specified number, historical running feature data that meets the first specified similarity condition with the running feature data is determined in the data set.

[0109] In another possible implementation, the determining module 302 is used to: In response to the number of elements in the dataset being less than or equal to a specified number, the feature vector of the running feature data is obtained, and in the optimization case library, historical running feature data that meets the second specified similarity condition with the running feature data is determined based on the feature vector.

[0110] In another possible implementation, the second generation module 303 is used for: The running feature data and historical running feature data are inserted into the prompt word template to obtain model prompt words. The model prompt words are used to instruct the generative language model to generate a target optimization strategy for the target periodic task based on the running feature data and historical running feature data. The model prompts are input into a generative language model to obtain a target optimization strategy for the target periodic task.

[0111] In another possible implementation, the second generation module 303 is used for: The operational characteristic data of the target periodic task and the target optimization strategy are input into the effect prediction model to obtain the prediction result of the optimization effect of the target optimization strategy.

[0112] In another possible implementation, the first generation module 301 is configured to: Obtain the runtime performance data of multiple task instances of the target periodic task within a specified time window; In response to the operational metric data meeting the specified optimization trigger conditions, the optimization request is generated.

[0113] It should be noted that the optimization strategy generation device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the optimization strategy generation device and the optimization strategy generation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0114] Based on the same inventive concept, corresponding to the optimization strategy generation method provided in the embodiments of this application, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the optimization strategy generation method described in the above embodiments.

[0115] Figure 4This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0116] The processor 1010 can be implemented using a general-purpose CPU (central processing unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0117] The memory 1020 can be implemented in the form of ROM (read-only memory), RAM (random access memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0118] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0119] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0120] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0121] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0122] The electronic device described above is used to implement the corresponding optimization strategy generation method in the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0123] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the optimization strategy generation method in the above embodiments. This computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be ROM (read-only memory), RAM (random access memory), CD-ROM (compact discread-only memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0124] In an exemplary embodiment, a computer program product is also provided, including computer program instructions that, when executed on a computer, cause the computer to perform the optimization strategy generation method described above.

[0125] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0126] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0127] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0128] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating optimization strategies, characterized in that, include: In response to an optimization request for a target periodic task, operational characteristic data of the target periodic task is generated based on the historical operational data of the target periodic task. In the optimization case library, historical operational feature data that meets specified similar conditions to the operational feature data are identified, and the historical operational feature data includes optimization strategies for periodic tasks corresponding to the historical operational feature data; The operational feature data and historical operational feature data are input into a generative language model to obtain a target optimization strategy for the target periodic task.

2. The optimization strategy generation method according to claim 1, characterized in that, The historical operational data includes the operational data of task instances generated each time the target periodic task is executed. The operational data of the task instances includes basic information data, resource usage data, stage execution data, structured query language feature data, and parameter configuration data.

3. The optimization strategy generation method according to claim 2, characterized in that, The step of generating operational characteristic data for the target periodic task based on its historical operational data includes: The running data of the task instances generated by each execution of the target periodic task are statistically analyzed to obtain the running feature data.

4. The optimization strategy generation method according to claim 3, characterized in that, The operational characteristic data includes the target task identifier of the target periodic task; In the optimized case library, historical operational feature data that meets specified similarity conditions to the operational feature data are identified, including: In the optimized case library, a data set of historical operational feature data corresponding to the task identifier that is the same as the target task identifier is determined; In response to the fact that the number of elements in the data set is greater than a specified number, historical running feature data that meets the first specified similarity condition with the running feature data is determined in the data set.

5. The optimization strategy generation method according to claim 4, characterized in that, The method further includes: In response to the number of elements in the dataset being less than or equal to a specified number, the feature vector of the running feature data is obtained, and in the optimization case library, historical running feature data that meets the second specified similarity condition with the running feature data is determined based on the feature vector.

6. The optimization strategy generation method according to any one of claims 1 to 5, characterized in that, The step of inputting the operational feature data and historical operational feature data into a generative language model to obtain a target optimization strategy for the target periodic task includes: The running feature data and historical running feature data are inserted into the prompt word template to obtain model prompt words. The model prompt words are used to instruct the generative language model to generate a target optimization strategy for the target periodic task based on the running feature data and historical running feature data. The model prompts are input into a generative language model to obtain a target optimization strategy for the target periodic task.

7. The optimization strategy generation method according to any one of claims 1 to 5, characterized in that, After obtaining the target optimization strategy for the target periodic task, the method further includes: The operational characteristic data of the target periodic task and the target optimization strategy are input into the effect prediction model to obtain the prediction result of the optimization effect of the target optimization strategy.

8. The optimization strategy generation method according to any one of claims 1 to 5, characterized in that, Prior to responding to the optimization request for the target periodic task, the method further includes: Obtain the runtime performance data of multiple task instances of the target periodic task within a specified time window; In response to the operational metric data meeting the specified optimization trigger conditions, the optimization request is generated.

9. An optimization strategy generation apparatus, characterized in that, include: The first generation module is configured to generate runtime characteristic data of the target periodic task based on the historical runtime data of the target periodic task in response to an optimization request for the target periodic task. The determination module is configured to determine, in the optimization case library, historical running feature data that meets specified similar conditions to the running feature data, the historical running feature data including optimization strategies for the periodic tasks corresponding to the historical running feature data; The second generation module is configured to input the running feature data and historical running feature data into a generative language model to obtain a target optimization strategy for the target periodic task.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method described in any one of claims 1 to 8.

12. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1 to 8.

Images