Information generation methods, apparatus, electronic devices and computer-readable storage media

Abstract

This disclosure provides an information generation method, apparatus, electronic device, and computer-readable storage medium, belonging to the field of computer technology. This disclosure acquires task data of a task to be analyzed in a target distributed computing cluster; multiple intelligent agents collaboratively analyze the task data to obtain analysis results, the analysis results including the current performance problems of the target distributed computing cluster and optimization reference information, the optimization reference information including target optimization knowledge for the current performance problems from various optimization knowledge and / or target tasks matching the task to be analyzed from the completed historical tasks of the target distributed computing cluster; based on the analysis results, performance optimization information of the target distributed computing cluster is generated, improving the accuracy and efficiency of performance optimization.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and specifically to an information generation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Platform performance optimization refers to the techniques used to eliminate platform performance bottlenecks and improve resource utilization, ultimately optimizing service level objectives such as throughput and latency. Currently, platform performance is optimized manually based on individual experience. However, manual optimization has limitations. On the one hand, platforms are becoming increasingly complex, making manual optimization inefficient; on the other hand, individual experience varies, and reliance on subjective experience makes it difficult to guarantee accuracy. Summary of the Invention

[0003] This disclosure provides an information generation method, apparatus, electronic device, and computer-readable storage medium, which can improve the performance optimization efficiency and accuracy of the platform.

[0004] In a first aspect, embodiments of this disclosure provide an information generation method, including: Obtain task data for the task to be analyzed in the target distributed computing cluster; The above task data is analyzed collaboratively by multiple intelligent agents to obtain analysis results. The analysis results include the current performance problems and optimization reference information of the target distributed computing cluster. The optimization reference information includes target optimization knowledge for the current performance problems from various optimization knowledge and / or target tasks that match the task to be analyzed from the completed historical tasks of the target distributed computing cluster. Based on the above analysis results, performance optimization information for the target distributed computing cluster is generated.

[0005] Secondly, embodiments of this disclosure provide an information generation method, wherein a target distributed computing cluster executes a task to be analyzed through an execution unit in a task execution engine, the method comprising: Based on the operating data of at least one of the execution units that performed the task to be analyzed, processor core performance indication information for the task to be analyzed is determined. Based on the processor core performance indication information for each of the above-mentioned tasks to be analyzed and the parallelism of the above-mentioned tasks to be analyzed, the performance evaluation information of the above-mentioned target distributed computing cluster is determined.

[0006] Thirdly, embodiments of this disclosure provide an information generation apparatus, including: The acquisition module is used to acquire task data of the task to be analyzed in the target distributed computing cluster; The analysis module is used to perform collaborative analysis of the above task data through multiple intelligent agents to obtain analysis results. The analysis results include the current performance problems and optimization reference information of the target distributed computing cluster. The optimization reference information includes target optimization knowledge for the current performance problems from various optimization knowledge and / or target tasks that match the task to be analyzed from the completed historical tasks of the target distributed computing cluster. The generation module is used to generate performance optimization information for the target distributed computing cluster based on the above analysis results.

[0007] Fourthly, embodiments of this disclosure provide an information generation apparatus, wherein a target distributed computing cluster executes a task to be analyzed through an execution unit in a task execution engine. The apparatus includes: The first determining module is used to determine processor core performance indication information for the task to be analyzed based on the running data of at least one of the execution units executing the task to be analyzed. The second determining module is used to determine the performance evaluation information of the target distributed computing cluster based on the processor core performance indication information for each of the above-mentioned tasks to be analyzed and the parallelism of the above-mentioned tasks to be analyzed.

[0008] Fifthly, embodiments of this disclosure also provide an electronic device, including a memory storing a plurality of instructions; a processor loading instructions from the memory to execute the steps of any information generation method provided in embodiments of this disclosure.

[0009] Sixthly, embodiments of this disclosure also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform the steps of any of the information generation methods provided in embodiments of this disclosure.

[0010] In a seventh aspect, embodiments of this disclosure also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in any of the information generation methods provided in embodiments of this disclosure.

[0011] In this embodiment, task data of the task to be analyzed in the target distributed computing cluster is acquired; multiple intelligent agents collaboratively analyze the task data to obtain analysis results, which include the current performance problems of the target distributed computing cluster and optimization reference information. The optimization reference information includes target optimization knowledge for the current performance problem from various optimization knowledge and / or target tasks from the completed historical tasks of the target distributed computing cluster that match the task to be analyzed; based on the analysis results, performance optimization information of the target distributed computing cluster is generated. This realizes the automatic generation of performance optimization information by multiple intelligent agents, combined with optimization knowledge and / or target tasks, thereby improving the efficiency of platform performance optimization and eliminating reliance on human subjective experience, thus improving the accuracy of platform performance optimization.

[0012] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram illustrating an application scenario of the information generation method provided by an exemplary embodiment of this disclosure; Figure 2 This is a flowchart illustrating an exemplary embodiment of the information generation method provided in this disclosure; Figure 3 This is a schematic diagram of the structure of the information generation system provided in an exemplary embodiment of this disclosure; Figure 4 This is a schematic diagram illustrating the process by which the multi-agent collaborative analysis module, provided in an exemplary embodiment of this disclosure, obtains target optimization information, current performance issues, and performance configuration information of the target task. Figure 5 This is a flowchart illustrating another information generation method provided by an exemplary embodiment of this disclosure; Figure 6 This is a schematic diagram of the structure of the information generation apparatus provided in an exemplary embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of another information generation apparatus provided by an exemplary embodiment of this disclosure; Figure 8 This is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of this disclosure. Detailed Implementation

[0015] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure. Furthermore, in the description of the embodiments of this disclosure, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. Therefore, features defined with "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this disclosure, "multiple" means two or more, unless otherwise explicitly specified.

[0016] This disclosure provides an information generation method, apparatus, electronic device, and computer-readable storage medium. Specifically, this embodiment will be described from the perspective of an information generation apparatus, which can be integrated into an electronic device, meaning the information generation method of this disclosure can be executed by an electronic device. Optionally, the electronic device may include a terminal device or a server. Optionally, the terminal device may be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, game console, or personal computer (PC), etc.

[0017] Optionally, the server can be a standalone server, or a server network or server cluster, including but not limited to computers, network hosts, single network servers, multiple network server sets, or cloud servers composed of multiple servers. Cloud servers consist of a large number of computers or network servers based on cloud computing.

[0018] When the electronic device is a server, it can be like... Figure 1 As shown, the server acquires task data of the task to be analyzed in the target distributed computing cluster; multiple agents collaboratively analyze the task data to obtain analysis results, which include the current performance problems of the target distributed computing cluster and optimization reference information. The optimization reference information includes target optimization knowledge for the current performance problem from various optimization knowledge and / or target tasks that match the task to be analyzed from the completed historical tasks of the target distributed computing cluster; based on the analysis results, performance optimization information of the target distributed computing cluster is generated and output to optimize the performance of the target distributed computing cluster based on the performance optimization information.

[0019] The following detailed description is provided in conjunction with the accompanying drawings. In this embodiment, the execution subject is a terminal device as an example. It should be noted that the order of description in the following embodiments is not intended to limit the preferred order of the embodiments. Although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown in the accompanying drawings.

[0020] Please refer to Figure 2 The specific process of this information generation method can be summarized in steps 201 to 203, where: Step 201: Obtain the task data of the task to be analyzed in the target distributed computing cluster.

[0021] In this context, the target distributed computing cluster refers to a server cluster with deployed tasks and scheduled task execution code. A task refers to a collection of code that processes data. A task to be analyzed refers to a task that is not offline; it can be a task awaiting triggering or currently running. Task data refers to all data generated during the task's lifecycle; for example, task data can include at least one of task metadata and task execution data. Task metadata refers to a collection of static information describing the task; for example, it can include at least one of task logic indication data, engine configuration parameters, task execution engine version information, and basic attribute data. Task logic indication data is used to indicate the task's processing logic; for example, task logic indication data includes at least one of the task's query statement (SQL) and task topology (e.g., a Directed Acyclic Graph, DAG). Engine configuration parameters refer to the parameters used by the task execution engine during task execution; for example, engine configuration parameters can include at least one of cache configuration, parallelism, and upstream / downstream storage connector parameters. Upstream / downstream storage connector parameters refer to at least one of the storage space types for storing task input data, storage space types for storing task output data, and batch size. Task execution data refers to the data generated during task execution. For example, task execution data includes at least one of the execution unit's execution data and the task's operator metrics data. Execution unit execution data may include at least one of processor utilization, allocated processor cores, and task data processing rate. Operator metrics data may include at least one of operator backpressure status, operator throughput information, operator latency information, and GC logs. Operator backpressure status refers to the mechanism where the downstream operator's processing efficiency is lower than the upstream operator's data transmission efficiency, causing data to accumulate in the buffer between the two operators, leading to the upstream operator reducing its data transmission efficiency. GC logs refer to files that automatically record memory cleanup activities during the execution of the process carrying the task.

[0022] In related technologies, one way to optimize the performance of a target distributed computing cluster is through manual adjustments based on personal experience. However, manual adjustments based on personal experience only involve general suggestions such as memory configuration, data partitioning, and network parameters. When faced with issues such as task SQL logic, data skew, and storage connector configuration, it is difficult to quickly obtain optimization solutions. This approach suffers from drawbacks such as high knowledge threshold, slow response, and poor reproducibility, especially when the target distributed computing cluster is large in scale and has a large number of tasks. Another approach is to directly optimize engine configuration parameters (batch size, parallelism, and cache configuration, etc.) using Bayesian optimization algorithms or reinforcement learning algorithms. However, this method treats the target distributed computing cluster as a "black box," and the optimization process is completely independent of the upper-layer task logic and real-time task running status. This results in a lack of interpretability in the optimization results, making it impossible to locate and explain the business root cause of performance problems. Furthermore, when faced with tasks composed of complex SQL and topology, the search space formed by the parameters and logic is huge, leading to high optimization costs and low efficiency.

[0023] In this embodiment, task data may include at least one of task metadata and task execution data. Task metadata includes at least one of task logic indication data, engine configuration parameters, version characteristic information of the task execution engine, and basic attribute data. This enables the automatic identification of various performance issues and the generation of performance optimization information for each issue. It has a low knowledge threshold, fast response, and strong replicability. Furthermore, the optimization process considers the upper-level task logic and real-time task execution status, making the optimization results interpretable. It can locate and explain the business root cause of performance problems, achieving low optimization costs and high optimization efficiency even when facing tasks composed of complex SQL and topologies.

[0024] Step 202: Collaboratively analyze the task data through multiple intelligent agents to obtain analysis results. The analysis results include the current performance problems of the target distributed computing cluster and optimization reference information. The optimization reference information includes target optimization knowledge for the current performance problems from various optimization knowledge and / or target tasks that match the task to be analyzed from the completed historical tasks of the target distributed computing cluster.

[0025] Here, an intelligent agent refers to a computing entity that can analyze and make decisions. The number of intelligent agents and their specific analytical functions can be selected according to the actual situation. For example, the number of multiple intelligent agents can include 4 or 5. Or, multiple intelligent agents can include a problem-solving intelligent agent and / or a task-matching intelligent agent. The problem-solving intelligent agent is used to analyze task data to obtain the current performance problem of the target distributed computing cluster and the target optimization knowledge for the current performance problem. The task-matching intelligent agent is used to determine the target task that matches the task to be analyzed from the completed historical tasks of the target distributed computing cluster. This embodiment does not limit this.

[0026] Collaborative analysis can be understood as multiple intelligent agents interacting and cooperating to jointly obtain current performance problems and optimization reference information. Current performance problems refer to obstacles that prevent the target distributed computing cluster from meeting predetermined standards in key metrics. Key metrics include throughput, latency, and / or resource utilization. Current performance problems may include at least one of the following: batch size too small or too large, data skew, suboptimal operator chain, hotspot data skew, computational inefficiency, slow data retrieval, slow data writing, and engine feature adaptation issues. When current performance problems include engine feature adaptation issues, the existence of such issues can be determined based on version feature information and task characteristics. Version feature information refers to the characteristics of the task execution engine's version information, which indicates the features possessed by that version of the task execution engine.

[0027] Optimization knowledge can reside in an optimization knowledge base, which can be industry-wide optimization experience knowledge. For example, optimization knowledge can be official documentation (such as Apache Flink, Doris tuning guides), community best practices and cases (such as tuning suggestions from Company A, Flink-Paimon optimization practices from Company B), and accumulated enterprise experience (such as memory management solutions from Company C, adaptive join optimization knowledge from Company D), etc. Optionally, the optimization knowledge base can be based on retrieval-enhanced generation techniques to return targeted optimization knowledge for the current performance problem.

[0028] Target optimization knowledge refers to optimization knowledge related to the current performance problem, which indicates how to solve the current performance problem and thus optimize the performance of the target distributed computing cluster. Optionally, optimization knowledge may include at least one of optimization theory and optimization strategy. Target optimization knowledge may include target optimization theory and target optimization strategy, where optimization strategy refers to specific optimization methods verified in practice.

[0029] Completed tasks refer to tasks that have been taken offline in the target distributed computing cluster. Target tasks matching the task to be analyzed refer to completed tasks whose structural similarity to the task to be analyzed meets preset similarity conditions. These preset similarity conditions can be the highest similarity or a similarity exceeding a threshold; this embodiment does not limit this. Optionally, the structural similarity between the task to be analyzed and completed tasks can be determined based on the feature fingerprints of the task to be analyzed and the feature fingerprints of completed tasks. Feature fingerprints can be represented by topological hashing or operators. Specifically, feature fingerprints can be converted into vectors, and a search can be performed in a task library based on these vectors to obtain the target task. The task library includes completed tasks. The search algorithm can be set according to actual conditions; for example, the search algorithm can be a K-nearest neighbor search algorithm or an inverted file index (IVF) algorithm. This embodiment does not limit this. Optionally, the task library may include complete performance snapshots of completed tasks, such as feature fingerprints, peak performance metrics (performance evaluation information, average performance per processor core indicator, and / or throughput, etc.), performance issues of completed tasks, and optimal performance configuration information for resolving those issues. This task library can provide efficient retrieval based on feature fingerprints, offering empirical evidence for performance optimization (such as optimal configuration information highly correlated with "data skew," "frequent GC," and "Kafka to Doris write scenarios"). Optionally, the analysis results may include performance configuration information for the target task. The performance configuration information for the target task can refer to the configuration information of the target task under the condition that the target distributed computing cluster performs well. For example, the performance configuration information of the target task may include batch size and the processing capacity that the target task can achieve, etc., which is not limited in this embodiment.

[0030] Optionally, the multiple agents may include at least one problem-solving agent, which, based on task data, determines the current performance problem of the target distributed computing cluster and searches for target optimization knowledge for the current performance problem from a variety of optimization knowledge.

[0031] Optionally, the multiple agents may include a task matching agent, which searches for target tasks that match the task to be analyzed from the completed tasks of the target distributed computing cluster, thereby improving the accuracy and efficiency of the target task.

[0032] Optionally, when the optimization reference information includes target optimization knowledge and target tasks, it is possible to find a target task that matches the task to be analyzed from the completed tasks of the target distributed computing cluster after determining that there is a current performance problem, thereby reducing the computer resources consumed by task matching.

[0033] Alternatively, each agent can be a reinforcement learning agent. In this case, each agent learns while identifying the current performance problem and optimization reference information. However, this method involves online intervention, which carries high risks, huge training costs, and may cause performance fluctuations during the convergence process. Therefore, it is possible to set each agent to stop learning while identifying the current performance problem and optimization reference information, thereby achieving offline analysis and making it safe and controllable.

[0034] In some embodiments, the agent includes at least one problem-handling agent, with different problem-handling agents used to analyze and determine different current performance problems; when the optimization reference information includes target optimization knowledge, multiple agents collaboratively analyze the task data to obtain analysis results, including: The problem-solving intelligent agent analyzes the task data to obtain the first initial analysis result, and based on the first initial analysis result, determines the current performance problem of the target distributed computing cluster. The problem-solving agent searches for target optimization knowledge related to the current performance problem from various optimization knowledge sources; The problem-solving agent integrates the current performance problem with the target optimization knowledge to obtain the analysis results.

[0035] The first initial analysis result is a phased output of the task data analysis process. It reveals objective findings of patterns, trends, or anomalies in the data and serves as the basis for identifying and locating specific current performance problems. For example, if the first initial analysis result shows that the frequency of a certain value in field A is 'a', and the frequency of other values ​​in field A is 'b', and 'a' is much greater than 'b', then the current performance problem can be identified as data skew. As another example, if the number of values ​​in field B of the GROUP BY clause is 'c', and the number of values ​​in field C of the GROUP BY clause is 'd', and 'c' is much less than 'd', then the current performance problem can be identified as the low-cardinality segment of the GROUP BY clause.

[0036] Optionally, the current performance problem identified by different problem-solving agents can be set according to the actual situation. For example, the agents include at least one of logic analysis agents, storage connector analysis agents, engine feature analysis agents, and resource analysis agents. The current performance problem identified by the logic analysis agent is a task logic problem, the current performance problem identified by the storage connector analysis agent is a storage connector configuration problem, the current performance problem identified by the engine feature analysis agent is an engine feature adaptation problem, and the current performance problem identified by the resource analysis agent is a computing resource problem. This embodiment does not limit this.

[0037] Optionally, the task data may include task metadata and task execution data. At least one problem-solving agent can determine the current performance problem of the target distributed computing cluster based on the task metadata and task execution data. This embodiment does not limit this.

[0038] Optionally, when the task metadata includes task logic indication data, the task logic indication data is analyzed by a logic analysis agent to obtain logic analysis results. Based on the logic analysis results, the task logic problem is obtained. The task logic problem may be, for example, a data skew key, a low-radix numeric field, and / or a non-optimal operator chain. A data skew key refers to a field value that occurs frequently. In this case, the current performance problem includes the task logic problem. Optionally, when the task logic indication data includes the query statement (SQL) of the task to be analyzed and the task topology (directed acyclic graph topology), the SQL can be converted into an Abstract Syntax Tree (AST). The AST is scanned using a predefined rule set. The task logic problem is determined based on the scan results. Alternatively, the task topology can be analyzed based on the task's operator chain and data partitioning strategy to obtain the task logic problem. The task logic problem determined based on the scan results may be, for example, a GROUP BY low-radix numeric field. A GROUP BY low-radix numeric field refers to a field in the GROUP BY block with fewer possible values. The task logic problem obtained by analyzing the task's topology may be, for example, a non-optimal operator chain.

[0039] Optionally, when the task metadata includes engine configuration parameters, these parameters can be analyzed to identify storage connector configuration issues. In this case, the current performance problem includes storage connector configuration issues. For example, when the engine configuration parameters include batch size, the storage connector analysis agent can determine whether the batch size is too large or too small based on the batch size. The storage connector configuration problem can be whether the batch size is too large or too small. Specifically, the batch size can be compared with the batch size in the optimization knowledge to determine whether the batch size is too large or too small. For example, if the upstream storage type is Kafka, the downstream storage type is Doris, the batch size written to Doris is 1, the hotspot operator is found to be a Doris write operator, and the batch size written in the optimization knowledge is greater than the batch size in the engine configuration parameters, then the batch size in the engine configuration parameters is determined to be too small.

[0040] Optionally, when the task metadata includes the version information of the task execution engine, the engine feature analysis agent can determine engine feature compatibility issues based on the task characteristics of the task to be analyzed and the version feature information of the task execution engine. For example, if the task to be analyzed contains an aggregation operator and the task execution engine version is Flink 1.20.0+, the version feature information of Flink 1.20.0+ indicates that it can use TVF for efficient aggregation, thus identifying an engine feature compatibility issue. Or, if the task to be analyzed contains a window aggregation operator and the task feature is window aggregation, the version feature information of the task execution engine indicates that it supports incremental window aggregation algorithms, but the current task still uses full aggregation mode, thus identifying an engine feature compatibility issue.

[0041] Optionally, when the task data includes task execution data, the task execution data can be analyzed by a resource analysis agent to obtain the computational resource problem. In this case, the current performance problem may include the computational resource problem. For example, task execution data includes processor utilization, the number of allocated processor cores, data processing rate, GC logs, and operator backpressure status. Computational resource issues include bottlenecks, such as hot data skew, computational inefficiency, slow data retrieval, and slow data writing. Specifically, a resource analysis agent can determine the average performance per processor core for each execution unit based on its processor utilization, the number of allocated processor cores, and data processing rate. Time-series data analysis of the average performance per processor core indicator is then performed to obtain variance and outliers (outliers are execution units whose average performance per processor core indicator deviates from the normal range). Hot data skew is determined based on variance and outliers. Computationally inefficient operators or hot operators are identified based on backpressure status using specific rules. Slow data retrieval and slow data writing issues are identified based on the average performance per processor core indicator and backpressure status. Computational inefficiency issues are identified through log pattern matching (matching Full GC patterns in GC logs).

[0042] Optionally, a problem-solving agent can also determine target optimization knowledge based on the output of other problem-solving agents. For example, a problem-solving agent may include a logic analysis agent and an engine feature analysis agent. The engine feature analysis agent can search for target optimization knowledge from optimization knowledge based on the logic analysis results and version information output by the logic analysis agent.

[0043] Optionally, after obtaining the knowledge of the current performance problem and the target optimization through the problem-processing agent, the problem-processing agent can fuse the knowledge of the current performance problem and the target optimization according to a preset structured format to obtain an analysis result that includes the knowledge of the current performance problem and the target optimization.

[0044] Optionally, to improve the credibility of the current performance problem in the analysis results, the initial analysis results, the current performance problem, and the target optimization knowledge can be integrated to obtain the analysis results.

[0045] Optionally, each problem-solving agent can fuse its current performance problem and target optimization knowledge to obtain sub-analysis results, and then fuse the sub-analysis results obtained by each problem-solving agent to obtain the analysis result.

[0046] In this embodiment, the intelligent agent includes at least one problem-processing intelligent agent. Different problem-processing intelligent agents are used to analyze and determine different current performance problems. When the optimization reference information includes target optimization knowledge, the problem-processing intelligent agents analyze the task data separately to obtain a first initial analysis result. Based on the first initial analysis result, the current performance problem of the target distributed computing cluster is determined. The problem-processing intelligent agents search for target optimization knowledge related to the current performance problem from various optimization knowledge sources. The problem-processing intelligent agents fuse the current performance problem with the target optimization knowledge to obtain an analysis result. This enables the problem-processing intelligent agents to determine different current performance problems, allowing for a comprehensive exploration of the current performance problems existing in the target distributed computing cluster, avoiding omissions of current performance problems, and enabling further performance optimization of the platform based on the current performance problems, thereby further improving the platform's performance.

[0047] In some embodiments, the intelligent agents include a problem-solving intelligent agent and a task-matching intelligent agent. When the optimization reference information includes the target task, multiple intelligent agents collaboratively analyze the task data to obtain analysis results, including: By analyzing task data through a problem-solving intelligent agent, the current performance issues of the target distributed computing cluster can be identified. If there is a target performance problem that meets the preset case matching conditions in the current performance problem, the task matching agent searches for a target task that matches the task to be analyzed from the completed historical tasks of the target distributed computing cluster. Analysis results are generated based on the current performance issues and the performance configuration information of the target task.

[0048] Some performance issues are complex, while others are simple. When a performance issue is complex, it is necessary to refer to the performance configuration information of the target task for performance optimization. Therefore, the preset case matching conditions are used to describe the type of the more complex performance issue. For example, the preset case matching conditions can include task logic issues, computing resource issues, or storage connector configuration issues. When the current performance issues include data skew, slow data writing, or severe mismatch between upstream and downstream batch sizes, data skew belongs to task logic issues, slow data writing belongs to computing resource issues, and severe mismatch between upstream and downstream batch sizes belongs to storage connector configuration issues. The target performance issues that meet the preset case matching conditions include data skew, slow data writing, or severe mismatch between upstream and downstream batch sizes.

[0049] Optionally, the problem-solving agent can search for target optimization knowledge related to the current performance problem from various optimization knowledge sources, and then generate analysis results based on the current performance problem, target optimization knowledge, and performance configuration information of the target task.

[0050] In this embodiment, the intelligent agent includes a problem-solving intelligent agent and a task-matching intelligent agent. When the optimization reference information includes the target task, the problem-solving intelligent agent analyzes the task data to obtain the current performance problem of the target distributed computing cluster. If there is a target performance problem that meets the preset case matching conditions among the current performance problems, the task-matching intelligent agent searches for a target task that matches the task to be analyzed from the completed historical tasks of the target distributed computing cluster. Based on the performance configuration information of the current performance problem and the target task, the analysis result is generated, so that task search is only performed when a target performance problem exists, and no task search is required when no target performance problem exists, thus reducing the computer resources consumed in searching for the target task.

[0051] In some embodiments, the multiple agents include a problem-solving agent and a first agent. The multiple agents collaboratively analyze task data to obtain analysis results, including: Through the task context manager, the problem-handling agent is triggered in parallel to analyze the task data and obtain a second initial analysis result, which includes the current performance problem. The task context manager receives the second initial analysis results sent by each problem-handling agent. Upon receiving the second initial analysis results, the corresponding first agent is triggered to perform analysis based on the type of the second initial analysis results to obtain the third initial analysis results. The third initial analysis results include target optimization knowledge and / or target tasks. The second and third initial analysis results are assembled using the task context manager to obtain the final analysis result.

[0052] The type of the second initial analysis result can be understood as the type of the current performance problem included in the second initial analysis result. For example, the type of the current performance problem may be a task logic problem, a computing resource problem, or a storage connector configuration problem, etc., which is not limited in this embodiment. For example, when the type of the current performance problem is a task logic problem, the storage connector analysis agent is triggered to determine the storage connector configuration problem and the task matching agent is triggered to determine the target task; as another example, when the type of the current performance problem is a computing resource problem, the task matching agent is triggered to determine the target task and the engine feature analysis agent is triggered to determine the engine feature adaptation problem.

[0053] Optionally, when the third initial analysis result includes target optimization knowledge, the first agent can be a problem-solving agent or a knowledge-finding agent. When the first agent is a problem-solving agent, for the sake of distinction, the problem-solving agent that obtains the second initial analysis result is referred to as the first problem-solving agent, and the problem-solving agent that obtains the third initial analysis result is referred to as the second problem-solving agent. Optionally, the first problem-solving agent and the second problem-solving agent can be the same or different. When the third initial analysis result includes the target task, the first agent can be a task-matching agent.

[0054] Optionally, the second initial analysis result may also include the first initial analysis result. The first agent can determine the current performance problem, target optimization knowledge, and / or target task based on the first initial analysis result. The first agent can be a second problem-solving agent and / or a task-matching agent, and the second problem-solving agent is different from the first problem-solving agent. For example, the third initial analysis result includes the current performance problem, the first problem-solving agent includes a logic analysis agent, and the second problem-solving agent includes an engine feature analysis agent. The first initial analysis result is obtained through the logic analysis agent. The first initial analysis result includes the task characteristics of the task to be analyzed. The task logic problem is determined based on the first initial analysis result. The second initial analysis result is obtained based on the task logic problem and the first initial analysis result. When the task context manager receives the second initial analysis result, it triggers the engine feature analysis agent to determine the engine feature adaptation problem based on the version feature information of the task execution engine and the task features in the second initial analysis result. For example, the first problem-solving agent includes a logic analysis agent, which obtains a first initial analysis result, determines the task logic problem based on the first initial analysis result, and obtains a second initial analysis result based on the task logic problem and the first initial analysis result. When the task context manager receives the second initial analysis result, it triggers the task matching agent to search for target optimization knowledge for the first initial analysis result from various optimization knowledge sources and to determine the target task from completed tasks.

[0055] Optionally, when the first intelligent agent includes an engine feature analysis intelligent agent and a task matching intelligent agent, the task context manager can send the first initial analysis result in the second initial analysis result to the engine feature analysis intelligent agent to trigger the engine feature analysis intelligent agent to determine the engine feature adaptation problem based on the task features and version feature information of the task execution engine in the second initial analysis result. If the type of the current performance problem in the second initial analysis result meets the preset case matching conditions, the task matching intelligent agent can be triggered to determine the target task.

[0056] Optionally, when the first problem-processing agent searches for target optimization knowledge related to the current performance problem from various optimization knowledge sources, it can fuse the second initial analysis result with the target optimization knowledge to obtain a first knowledge analysis result. The first knowledge analysis result and the third initial analysis result are then assembled using a task context manager to obtain the final analysis result. Optionally, when the first agent includes a second problem-processing agent and a task-matching agent, the second problem-processing agent can generate a second knowledge analysis result based on the target optimization knowledge, the task-matching agent can generate a configuration analysis result based on the performance configuration information of the target task, and finally, the task context manager can assemble the first knowledge analysis result, the second knowledge analysis result, and the configuration analysis result to obtain the final analysis result.

[0057] In this embodiment, multiple intelligent agents include a problem-handling intelligent agent and a first intelligent agent. Through a task context manager, the problem-handling intelligent agents are triggered in parallel to analyze task data, obtaining a second initial analysis result, which includes the current performance problem. The task context manager receives the second initial analysis results sent by each problem-handling intelligent agent and, upon receiving the second initial analysis result, triggers the corresponding first intelligent agent to perform analysis based on the type of the second initial analysis result, obtaining a third initial analysis result, which includes target optimization knowledge and / or target tasks. The task context manager assembles the second and third initial analysis results to obtain the analysis result. This achieves coordinated scheduling of multiple intelligent agents through the task context manager, forming an event-driven, information-sharing collaborative network under the scheduling of the task context manager. This allows for the rapid acquisition of analysis results even with a highly complex target distributed computing cluster through multiple intelligent agents.

[0058] In some embodiments, task data includes task metadata, which includes at least one of the following: the query statement of the task to be analyzed, the directed acyclic graph topology, the connector parameters of upstream and downstream storage, and the version characteristic information of the task execution engine. The type of task execution engine can be set according to actual conditions; for example, the task execution engine can be Flink or Spark, which is not limited in this embodiment. Version characteristic information refers to the characteristics of the version information of the task execution engine. The connector parameters of upstream and downstream storage are used to describe the characteristics of the storage space for the input data and the storage space for the output data of the task. For example, they may include at least one of the following: the type of storage space for the input data, the type of storage space for the output data, and the batch size.

[0059] In related technologies, one way to optimize the performance of a target distributed computing cluster is through manual adjustments based on personal experience. However, manual adjustments based on personal experience only involve general suggestions such as memory configuration, data partitioning, and network parameters. When faced with issues such as task SQL logic, data skew, and storage connector configuration, it is difficult to quickly obtain optimization solutions. This approach suffers from drawbacks such as high knowledge threshold, slow response, and poor reproducibility, especially when the target distributed computing cluster is large in scale and has a large number of tasks. Another approach is to directly optimize engine configuration parameters (batch size, parallelism, and cache configuration, etc.) using Bayesian optimization algorithms or reinforcement learning algorithms. However, this method treats the target distributed computing cluster as a "black box," with the optimization process completely independent of the upper-level task logic. This results in a lack of interpretability in the optimization results, making it impossible to locate and explain the business root cause of performance problems. Furthermore, when faced with tasks composed of complex SQL and topology, the search space formed by the parameters and logic is huge, leading to high optimization costs and low efficiency.

[0060] In this embodiment, the task data may include task metadata, which includes at least one of the query statement of the task to be analyzed, the directed acyclic graph topology, and the connector parameters of upstream and downstream storage. This can automatically identify various performance problems and obtain performance optimization information for each problem. It has a low knowledge threshold, fast response, and strong replicability. Furthermore, the optimization process takes into account the upper-level task logic, making the optimization results interpretable. It can locate and explain the business root cause of performance problems. Even when faced with tasks composed of complex SQL and topology, it can achieve low optimization cost and high optimization efficiency.

[0061] Furthermore, while related technologies allow task execution engines to adaptively optimize, such as dynamically selecting the Join algorithm based on runtime data volume (e.g., switching to Broadcast Hash Join), this optimization is based on predefined rules within the engine and has limited flexibility, failing to address optimizations to external engine configurations (storage layer parameters) and version characteristics. In this embodiment, task metadata includes at least one of the connector parameters of upstream and downstream storage and version characteristic information of the task execution engine, enabling optimizations to address external engine configurations and to be performed based on version characteristic information, such as using higher-order optimization modes (e.g., using TVF).

[0062] In some embodiments, the agent includes a logic analysis agent, which is used to determine the task logic problem based on at least one of the query statement of the task to be analyzed and the topology of the directed acyclic graph.

[0063] In this context, "task logic problem" refers to the aspects of the task logic indicator data that need improvement. Examples include data skew keys, low-cardinality numeric fields, and / or suboptimal operator chains. Data skew keys refer to frequently occurring field values. In this case, the current performance problem includes task logic problems. Optionally, when the task logic indicator data includes the query statement (SQL) of the task to be analyzed and the task's topology (directed acyclic graph topology), the SQL can be converted into an Abstract Syntax Tree (AST). The AST is then scanned using a predefined rule set. Based on the scan results, the task logic problem is determined. Alternatively, the task's topology can be analyzed based on the task's operator chains and data partitioning strategy to obtain the task logic problem. For example, a task logic problem determined based on the scan results might be a low-cardinality numeric field in the GROUP BY clause, where the field has fewer possible values. A task logic problem obtained through topology analysis might be a suboptimal operator chain.

[0064] Optionally, the logic analysis agent is used to determine the logic analysis result based on at least one of the query statement of the task to be analyzed and the topology of the directed acyclic graph, and to obtain the task logic problem based on the logic analysis result. When a first initial analysis result exists, the first initial analysis result is the logic analysis result.

[0065] In some embodiments, the agent includes a storage connector analysis agent, which determines storage connector configuration problems based on the connector parameters of the upstream and downstream storage of the task to be analyzed. The storage connector configuration problem can refer to the configuration of the storage space for the input data and the storage space for the output data of the storage task. For example, the storage connector configuration problem could be that the batch size of the storage space for the input data and the storage space for the output data of the storage task is too large or too small. In this case, the current performance problem includes the storage connector configuration problem. Specifically, the batch size can be compared with the batch size in the optimization knowledge to determine whether the batch size is too large or too small. For example, if the upstream storage type is Kafka, the downstream storage type is Doris, the batch size for writing to Doris is 1, the hotspot operator is a Doris write operator, and the batch size for writing in the optimization knowledge is greater than the batch size in the engine configuration parameters, then the batch size is determined to be too small.

[0066] In some embodiments, the agent includes an engine feature analysis agent, which is used to determine engine feature compatibility issues based on the task features of the task to be analyzed and the version feature information of the task execution engine. The engine feature compatibility issue indicates that the task features are incompatible with the version of the task execution engine. Task features refer to attribute information extracted from the task's query statement and / or directed acyclic graph topology, used to characterize the essence of the task's operation mode. These features may include, but are not limited to, operator type features (such as the presence of aggregation, join, or window operations), data skew features (such as the concentration of key-value distribution), and state access pattern features (such as whether large state reads and writes are involved). These features can be automatically extracted and transmitted by the SQL / topology analysis agent during the parsing phase, or they can be directly identified by the engine feature analysis agent. For example, if the task to be analyzed contains aggregation operators, the task features are aggregation features, and the task execution engine version is Flink 1.20.0+, the version feature information of Flink 1.20.0+ indicates that it can use TVF for efficient aggregation, thus determining that there is an engine feature compatibility issue.

[0067] Optionally, task characteristics can be obtained through other intelligent agents, or through an engine characteristic analysis intelligent agent; this embodiment does not impose any limitations on this. Optionally, task metadata may include version information of the task execution engine, and version characteristic information can be obtained based on the version information by an engine characteristic analysis intelligent agent.

[0068] In some embodiments, task data includes task execution data; task execution data includes execution data of each execution unit executing the task to be analyzed, and the execution data includes at least one of processor utilization, allocated number of processor cores, data processing rate, operator backpressure status, and operator latency information.

[0069] Among them, task execution data refers to the data generated during the execution of a task. For example, task execution data includes at least one of the execution data of the execution unit that executes the task and the indicator data of the task's operators. The execution data of the execution unit is, for example, at least one of the processor utilization rate, the number of allocated processor cores, and the data processing rate of the task. The indicator data of the operators is, for example, at least one of the operator backpressure status, operator throughput information, operator latency information, and GC logs. The operator backpressure status refers to the mechanism in which the processing efficiency of the downstream operator is lower than the data sending efficiency of the upstream operator, causing data to accumulate in the buffer between the two operators, resulting in the upstream operator reducing the data sending efficiency. The GC log refers to the file that automatically records the memory cleanup activities during the execution of the process carrying the task.

[0070] In related technologies, Bayesian optimization algorithms or reinforcement learning algorithms are used to directly optimize engine configuration parameters (batch size, parallelism, and cache configuration, etc.). However, this method treats the target distributed computing cluster as a "black box," and the optimization process is completely independent of the real-time task running status, resulting in a lack of interpretability of the optimization results and an inability to locate and explain the business root cause of performance problems.

[0071] In this embodiment, the task data includes task execution data; the task execution data includes the execution data of each execution unit executing the task to be analyzed. The execution data includes at least one of processor utilization, allocated number of processor cores, data processing rate, operator backpressure status, and operator latency information. It can automatically identify various performance problems and obtain performance optimization information for various performance problems. It has a low knowledge threshold, fast response, and strong replicability. Moreover, the optimization process takes into account the real-time task execution status, making the optimization results interpretable and able to locate and explain the business root cause of performance problems.

[0072] In some embodiments, the intelligent agent includes a resource analysis intelligent agent, which is used to determine computational resource problems based on the runtime data of each execution unit performing the task to be analyzed. Here, computational resource problems refer to problems caused by computer resources.

[0073] Optionally, the runtime data of each execution unit can be analyzed by a resource analysis agent to obtain the computational resource problem. In this case, the current performance problem may include the computational resource problem. For example, when runtime data includes processor utilization, the number of allocated processor cores, data processing rate, GC logs, and operator backpressure status, computational resource issues include bottlenecks such as hot data skew, computational inefficiency, slow data retrieval, and slow data writing. Specifically, a resource analysis agent can determine the average performance per processor core for each execution unit based on its processor utilization, the number of allocated processor cores, and data processing rate. Time-series data analysis of this average performance per processor core indicator yields variance and outliers (outliers are execution units whose average performance per processor core indicator deviates from the normal range). Based on variance and outliers, hot data skew is identified. Using specific rules, computationally inefficient operators or hot operators are identified based on backpressure status. Slow data retrieval and slow data writing issues are identified based on the average performance per processor core indicator and backpressure status. Computational inefficiency issues are identified through log pattern matching (matching Full GC patterns in GC logs).

[0074] In some embodiments, the resource analysis agent is used to determine computational resource problems based on average per-processor core performance indicator information, wherein the average per-processor core performance indicator information is determined based on the operating data of each execution unit performing the task to be analyzed, and the process of determining the average per-processor core performance indicator information includes: Multiply the processor utilization rate by the number of allocated processor cores to obtain processor metric information; Dividing the data processing rate by the processor performance metrics yields the average performance per processor core for the execution unit.

[0075] The average performance per processor core of an execution unit can refer to the average number of records per second per processor core of that execution unit. Specifically, the processor utilization, the number of processor cores, and the data processing rate can be substituted into formula (1) to calculate the average performance per processor core of the execution unit. (1) in, E j Indicates the first j Average performance per processor core across execution units. inputRecordsQps j Indicates data processing rate. cpuUsage j Indicates usage rate, cpuCoreCount jIndicates the number of processor cores.

[0076] In related technologies, computing resource issues are identified through basic resource metrics (CPU, memory, network I / O) or internal engine metrics (backpressure, throughput, latency). However, identifying computing resource issues through basic or internal engine metrics suffers from the problem of saturation, which reduces the accuracy and efficiency of obtaining computing resource information. For example, in practice, it has been found that simply monitoring container-level memory usage cannot accurately pinpoint the problem of low heap memory utilization, requiring more refined analysis methods.

[0077] In this embodiment, the processor utilization rate and the allocated number of processor cores are multiplied to obtain processor performance information; the data processing rate and the processor performance information are divided to obtain the average performance per processor core of the execution unit. Based on the average performance per processor core, the resource analysis agent determines the computing resource problem, simplifying the process of obtaining the average performance per processor core indicator, thereby improving the efficiency and accuracy of the obtained computing resource problem and enabling cross-task comparison.

[0078] Step 203: Based on the analysis results, generate performance optimization information for the target distributed computing cluster.

[0079] The performance optimization information can be used to indicate how to optimize the target distributed computing cluster. For example, the performance optimization information may include optimization suggestions, such as actionable code, configuration, or architecture modifications. Optionally, to improve the credibility of the performance optimization information, it may also include at least one of the current performance issues and theoretical knowledge of the target optimization. Optionally, the performance optimization information may also include the potential performance improvements from optimization and the risks that need to be considered. For example, the performance optimization information may be: 1. Bottleneck location: The task has a serious data skew in the user_id field, which causes an excessive load on a certain downstream execution unit.

[0080] 2. Specific recommendations: a. Use REBALANCE() to redistribute data in Flink SQL. b. Consider adjusting the batch.size of DorisSink from 1000 to 5000 and setting batch.interval to 2s. c. Check and optimize the custom state data structure used in the operator corresponding to this execution unit to avoid frequent garbage collection. d. Replace the original logic with a Flink TVF function; the rewritten code is xxx.

[0081] 3. Expected benefits and risks: Estimate the performance improvement that the adjustment may bring and the risks that need to be noted.

[0082] 4. Relevant basis: xxxxxxx.

[0083] Optionally, after obtaining performance optimization information, the performance of the target distributed computing cluster can be automatically optimized based on the performance optimization information. Alternatively, the performance of the target distributed computing cluster can be optimized based on the performance optimization information after confirmation by staff. Optionally, the process of optimizing the performance of the target distributed computing cluster based on performance optimization information can be as follows: filling the parameters in the optimization suggestions into the work order template or script template, and modifying them through the work order template or script template.

[0084] In some embodiments, performance optimization information for the target distributed computing cluster can be generated based on the analysis results using preset mapping rules, or, based on the analysis results, performance optimization information for the target distributed computing cluster can be generated, including: Based on the analysis results, performance optimization information for the target distributed computing cluster is generated using a large language model.

[0085] In this context, a Large Language Model (LLM) refers to a deep learning model with a large number of parameters, possessing powerful natural language understanding, generation, and reasoning capabilities, such as GPT or GLM. This embodiment does not limit the LLM to these specific models. Instead of directly inputting task data into the LLM, this embodiment first obtains current performance issues and optimization reference information based on the task data. Then, it inputs these issues and reference information into the LLM, combining multiple agents, Retrieval-Augmented Generation (RAG) technology, and the LLM to obtain performance optimization information, further improving the accuracy of the obtained performance optimization information.

[0086] In some embodiments, before generating performance optimization information, the performance of the target distributed computing cluster can be evaluated to obtain performance evaluation information. If the performance evaluation information indicates that optimization is needed, multiple agents can collaboratively analyze the task data to obtain analysis results. Alternatively, after obtaining the performance evaluation information, multiple agents can directly collaboratively analyze the task data to obtain analysis results. Or, both performance evaluation information and performance optimization information can be determined simultaneously, and finally, the performance evaluation information and performance optimization information can be output together.

[0087] In some embodiments, this embodiment further includes: Based on performance optimization information, the performance of the target distributed computing cluster is optimized; The performance evaluation information of the target distributed computing cluster is obtained. This information includes a first performance evaluation before optimization and a second performance evaluation after optimization. By comparing the first and second performance evaluations, the performance optimization result of the target distributed computing cluster is determined. For example, the percentage improvement of the second performance evaluation relative to the first performance evaluation is calculated. When this percentage improvement is greater than a preset percentage threshold (e.g., 10%), the performance optimization is considered successful.

[0088] The performance evaluation information is used to assess the performance of the target distributed computing cluster. Optionally, the method for determining the performance evaluation information can be selected according to the actual situation, and this embodiment does not limit it. For example, the average processor utilization of the target distributed computing cluster can be used as the performance evaluation information. Or, the target distributed computing cluster executes the task to be analyzed through the execution unit in the task execution engine. The task data includes task running data, which includes the running data of the execution unit. Based on the running data, processor core performance indicator information for the task to be analyzed is determined. Based on the processor core performance indicator information for each task to be analyzed and the parallelism of the task to be analyzed, the performance evaluation information of the target distributed computing cluster is determined. This embodiment does not limit it.

[0089] In related technologies, the performance evaluation process focuses on comparing different task execution engines under a preset standardized load. However, the test scenarios are very different from the complex and ever-changing business logic, data patterns, and real-time traffic in the production environment (which refers to the actual usage scenario). Therefore, it is impossible to conduct a quantitative evaluation of the performance of the target distributed computing cluster under the real load that can be compared historically.

[0090] In this embodiment, the performance of the target distributed computing cluster is optimized based on performance optimization information; performance evaluation information of the target distributed computing cluster is obtained, including first performance evaluation information of the target distributed computing cluster before performance optimization and second performance evaluation information after performance optimization. By comparing the first performance evaluation information and the second performance evaluation information, the performance optimization result of the target distributed computing cluster is determined, thereby providing a historically comparable quantitative evaluation of the performance of the target distributed computing cluster through the first performance evaluation information and the second performance evaluation information.

[0091] In some embodiments, the target distributed computing cluster executes the task to be analyzed through execution units in the task execution engine. The task data includes task execution data, which in turn includes the execution data of the execution units. The goal is to obtain performance evaluation information of the target distributed computing cluster, including: Based on the runtime data of at least one execution unit performing the task to be analyzed, determine processor core performance indication information for the task to be analyzed; Based on the processor core performance indicators and the parallelism of each task to be analyzed, the performance evaluation information of the target distributed computing cluster is determined.

[0092] In this context, the execution unit refers to the basic unit for executing tasks within the task execution engine, which can also be called a pod. A task to be analyzed can be executed by at least one execution unit. The execution unit's runtime data can refer to monitoring metrics and / or status data generated during the execution of the execution unit. For example, the execution unit's runtime data may include the processor (CPU) utilization (cpuUsage) allocated to the execution unit, the number of processor cores allocated to the execution unit (cpuCoreCount), the execution unit's data processing rate (also known as inputRecordsQps), memory utilization, and / or runtime, etc., which are not limited in this embodiment.

[0093] The processor core performance indicator information of a task to be analyzed is used to indicate the single-core bottleneck performance of the task to be analyzed. Through the processor core performance indicator information, the changes of a task to be analyzed before and after performance optimization can be known and / or it can be determined whether the task to be analyzed is a bottleneck task affecting the overall performance of the target distributed computing cluster.

[0094] After obtaining the processor core performance indicator information, performance evaluation information can be determined in various ways. In one implementation, the processor core performance indicator information can be multiplied by the parallelism of the task to be analyzed and summed. The summation result is the sum of the parallelism of all tasks to be analyzed, thus obtaining the performance evaluation information. Specifically, the processor core performance indicator information can be substituted into formula (2) for calculation to obtain the performance evaluation information: (2) in, Cphi This indicates performance evaluation information. E iavg Indicates the first i Processor core performance indicators for each task to be analyzed. parallelism i Indicates the first i The parallelism of each task to be analyzed.

[0095] In another implementation, the processor core performance indicators of each task to be analyzed can be weighted and sorted according to their parallelism to construct a cluster performance distribution histogram. The statistical characteristic values ​​of this distribution histogram, such as the median or the single-core performance corresponding to a specified quantile, can be used as the performance evaluation information of the target distributed computing cluster, thereby reflecting the distribution and dispersion of the overall processing capacity of the cluster.

[0096] In this embodiment, the target distributed computing cluster executes the task to be analyzed through execution units in the task execution engine. The task data includes task execution data, which in turn includes the execution data of the execution units. Based on the execution data of at least one execution unit executing the task to be analyzed, processor core performance indicator information for the task to be analyzed is determined. Based on the processor core performance indicator information for each task to be analyzed and the parallelism of the task to be analyzed, performance evaluation information of the target distributed computing cluster is determined. The performance evaluation information is output, thereby providing a quantifiable, normalized, and unified objective standard for the performance of the target distributed computing cluster. This makes the performance evaluation of the target distributed computing cluster more rational than subjective. Furthermore, determining the performance evaluation information based on the parallelism of the task to be analyzed can better reflect the overall business processing capability of the target distributed computing cluster and provide an objective benchmark for performance comparison.

[0097] In some embodiments, determining processor core performance indication information for the task under analysis based on runtime data from at least one execution unit performing the task under analysis includes: Based on the operational data of each execution unit, determine the average per-processor core performance indicator for each execution unit; Based on the average per-processor-core performance indicator of the execution unit performing the task to be analyzed, determine the processor-core performance indicator for the task to be analyzed.

[0098] The average performance per processor core indicator for an execution unit can refer to the average number of records per second per processor core of that execution unit. After obtaining the average performance per processor core indicator, the processor core performance indicator for the task to be analyzed can be determined based on the average performance per processor core indicator and the parallelism of the task to be analyzed. Specifically, the average performance per processor core indicator and the parallelism of the task to be analyzed can be substituted into formula (3) for calculation to obtain the processor core performance indicator: (3) in, E iavg Indicates the first i Processor core performance indicators for each task to be analyzed. parallelismi Indicates the first i The degree of parallelism of the tasks to be analyzed. E ij Indicates execution of the first i Average per-processor-core performance indicator for the j-th execution unit of a task to be analyzed.

[0099] Optionally, the average performance per processor core information may include a first average performance per processor core information before performance optimization and a second average performance per processor core information after performance optimization.

[0100] In this embodiment, since the average per-processor-core performance indicator information of each execution unit can eliminate the impact of differences in computer resource specifications, the average per-processor-core performance indicator information of each execution unit is determined based on the running data of each execution unit. Based on the average per-processor-core performance indicator information of the execution unit executing the task to be analyzed, the processor core performance indicator information for the task to be analyzed is determined. This can provide a comparable evaluation standard for heterogeneous and varied tasks, thereby providing a global performance benchmark that can be compared across tasks.

[0101] In some embodiments, the operational data of an execution unit includes the processor utilization rate allocated to the execution unit, the number of allocated processor cores, and the data processing rate of the execution unit. Based on the operational data of each execution unit, average performance per processor core indicator information for each execution unit is determined, including: Multiply the processor utilization rate by the number of processor cores to obtain processor metrics information; Dividing the data processing rate by the processor performance metrics yields the average performance per processor core for the execution unit.

[0102] Specifically, the utilization rate, number of processor cores and data processing rate can be substituted into the above formula (1) for calculation to obtain the average performance indication information per processor core of the execution unit.

[0103] In related technologies, performance evaluation results are determined through basic resource metrics (CPU, memory, network I / O) or internal engine metrics (backpressure, throughput, latency). However, performance analysis and optimization using basic or internal engine metrics suffer from the problem of cumbersome metrics, which reduces the accuracy and efficiency of obtaining performance evaluation results. For example, in practice, it has been found that simply monitoring container-level memory usage cannot accurately pinpoint the problem of low heap memory utilization, requiring more refined analysis methods.

[0104] In this embodiment, the running data of the execution unit includes the processor utilization rate allocated to the execution unit, the number of processor cores allocated, and the data processing rate of the execution unit. Multiplying the processor utilization rate and the number of processor cores yields processor metric information; dividing the data processing rate and the processor metric information yields the average performance per processor core indicator information of the execution unit. This simplifies the metric for obtaining the average performance per processor core indicator information, thereby simplifying the metric for obtaining the performance evaluation result and improving the efficiency and accuracy of the obtained average performance evaluation result.

[0105] The following is based on Figure 3 and Figure 4 The information generation method provided in this disclosure will be further explained. For example... Figure 3 As shown, in this embodiment, the information generation method can be implemented through an information generation system. The information generation system includes an input layer, a quantitative evaluation module, an analysis layer, and an output feedback layer. The input layer collects task execution data from the execution unit pod and obtains task metadata from the task management platform. The quantitative evaluation module determines the first processor core performance indicator information and the first performance evaluation information based on the task execution data. The analysis layer, through a multi-agent collaborative analysis module, determines the current performance problem based on the task metadata, task execution data, and the first average performance indicator information per processor core. It also searches for target tasks matching the task to be analyzed from completed tasks in the task library and searches for optimization knowledge bases targeting the current performance problem. The target optimization knowledge output feedback layer uses a large language model to generate performance optimization information based on the current performance problem, the performance configuration information of the target task, and the target optimization knowledge. It then outputs the performance optimization information, the first processor core performance indicator information, and the first performance evaluation information to the staff. The system automatically performs performance optimization based on the performance optimization information, or performs performance optimization based on the performance optimization information after confirmation by the staff. Through the quantitative evaluation module, the system determines the second processor core performance indicator information and the second performance evaluation information based on the task running data. If the second performance evaluation information indicates an improvement in the performance of the target distributed computing cluster, the optimized performance configuration information of the current task to be analyzed is stored in the task library when the current task to be analyzed is completed.

[0106] The execution process of the multi-agent collaborative analysis module can be as follows: Figure 4As shown, the intelligent agents include a logic analysis agent, a resource analysis agent, a storage connector analysis agent, an engine characteristic analysis agent, and a task matching agent. After receiving an optimization request, the task context management triggers the logic analysis agent to determine the task logic problem based on the task logic indication data and search for target optimization knowledge for the task logic problem from the optimization knowledge base. It also triggers the resource analysis agent to determine the computing resource problem based on the task running data and search for target optimization knowledge for the computing resource problem from the optimization knowledge base.

[0107] When the task context manager receives a task logic problem and / or computational resource problem, it triggers a task matching agent to search for a target task matching the task to be analyzed in the task library. Based on the logic analysis results of the logic analysis agent, it searches for target optimization theory knowledge for the logic analysis results in the optimization knowledge base. It also triggers a storage connector analysis agent to determine the storage connector configuration problem based on the connector parameters of upstream and downstream storage and searches for target optimization knowledge for the storage connector configuration problem in the optimization knowledge base. Finally, it triggers an engine feature analysis agent to determine the engine feature adaptation problem and searches for target optimization knowledge for version feature information in the optimization knowledge base.

[0108] The task matching agent sends the performance configuration information of the target task to the logic analysis agent, resource analysis agent, storage connector analysis agent, and engine feature analysis agent. Each agent outputs a structured summary, including relevant optimization knowledge, to the task context manager. For example, the logic analysis agent sends the task logic problem, the target optimization knowledge for the task logic problem, and configuration information related to the task logic problem from the target task's performance configuration information to the task context manager; the resource analysis agent sends the computational resource problem, the target optimization knowledge for the computational resource problem, and configuration information related to the computational resource problem from the target task's performance configuration information to the task context manager; the storage connector analysis agent sends the storage connector configuration problem, the target optimization knowledge for the storage connector configuration problem, and configuration information related to the storage connector configuration problem from the target task's performance configuration information to the task context manager; the engine feature analysis agent sends the engine feature adaptation problem, the target optimization knowledge for the engine feature adaptation problem, and configuration information related to version information from the target task's performance configuration information to the task context manager; and the task matching agent sends the performance configuration information of the target task to the task context manager. The task context manager then aggregates and assembles the structured results from each agent into a total structured result, which is then input into the large language model.

[0109] Specifically, the logic analysis agent identifies task logic problems based on task logic indication data and retrieves target optimization knowledge from the optimization knowledge base. For example, if the task logic problem is a multi-layer Join algorithm, the target optimization knowledge would be the optimization theory and strategies for multi-layer Join algorithms. The resource analysis agent identifies computational resource problems based on task execution data and retrieves target optimization knowledge from the optimization knowledge base. For example, if the computational resource problem is frequent garbage collection (GC), the target optimization knowledge would be the optimization theory and strategies for frequent GC. The storage connector analysis agent identifies storage connector configuration problems based on the connector parameters of upstream and downstream storage and retrieves target optimization knowledge from the optimization knowledge base. For example, if the storage connector configuration problem is that the batch size for Doris writes is too small, the target optimization knowledge would be the theoretical knowledge and historically used batch sizes for Doris writes. The engine feature analysis agent identifies engine feature adaptation issues based on task characteristics and the version characteristics of the task execution engine, and determines the target optimization knowledge for these issues. For example, if the task feature is aggregation and the version information is Flink 1.20.0+, indicating that TVF can be used for efficient aggregation, a problem exists in engine feature adaptation. The target optimization knowledge could be the rewriting results of SQL using TVF from the optimization knowledge base. The task matching agent, based on the logical analysis results of the logic analysis agent, searches the optimization knowledge base for target optimization theory knowledge related to the logical analysis results. For example, if the logical analysis results include aggregation operators, the target optimization theory knowledge would be optimization theory knowledge containing aggregation operators.

[0110] In this embodiment, performance evaluation information provides a quantifiable and historically comparable objective standard for the overall performance of the target distributed computing cluster, enabling performance evaluation to move from subjective to rational.

[0111] In this embodiment, by employing retrieval enhancement generation technology, multi-agent white-box analysis, and a collaborative mechanism of the task library, complex problems are decomposed into intermediate results that are supported by both data and theoretical basis. Then, based on these intermediate results, performance optimization information with both theoretical basis and practical verification is generated. This enables a comprehensive "white-box" analysis of task logic, task execution data, and optimization knowledge (white-box means that the performance optimization information contains optimization suggestions and relevant evidence). As a result, accurate, interpretable, and actionable performance optimization information is generated, realizing a shift from "black-box parameter search" (black-box means only results without evidence or process) to "intelligent root cause diagnosis," greatly reducing reliance on manual intervention and improving the accuracy and efficiency of tuning.

[0112] In this embodiment, after optimization, the performance of the target distributed computing cluster is verified again, forming a complete closed loop of "evaluation-diagnosis-tuning-verification". This not only solves the current performance problem, but also enables the successful tuning cases generated during continuous operation to be automatically transformed into reusable experience and stored in the task library. This gives the system the ability to self-evolve and continuously learn, thereby ensuring that the target distributed computing cluster runs in a high-efficiency state for a long time.

[0113] As can be seen from the above, in this embodiment of the disclosure, task data of the task to be analyzed in the target distributed computing cluster is obtained; multiple intelligent agents collaboratively analyze the task data to obtain analysis results, which include the current performance problems of the target distributed computing cluster and optimization reference information. The optimization reference information includes target optimization knowledge for the current performance problem from various optimization knowledge and / or target tasks that match the task to be analyzed from the completed historical tasks of the target distributed computing cluster; based on the analysis results, performance optimization information of the target distributed computing cluster is generated. This realizes the automatic generation of performance optimization information by multiple intelligent agents, combined with optimization knowledge and / or target tasks, thereby improving the efficiency of platform performance optimization and eliminating reliance on human subjective experience, thus improving the accuracy of platform performance optimization.

[0114] The following provides a further description of another information generation method provided in this disclosure. Detailed descriptions are given below in conjunction with the accompanying drawings. In this embodiment, a terminal device is used as the execution subject. It should be noted that the order of description in the following embodiments is not intended to limit the preferred order of the embodiments. Although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown in the accompanying drawings.

[0115] Please refer to Figure 5 The specific process of this information generation method can be summarized in steps 501-502, where: Step 501: Based on the running data of at least one execution unit executing the task to be analyzed, determine the processor core performance indication information for the task to be analyzed.

[0116] In this embodiment, the target distributed computing cluster executes the tasks to be analyzed through execution units in the task execution engine. An execution unit refers to the basic unit for executing tasks in the task execution engine, which can also be called a pod. A task to be analyzed can be executed by at least one execution unit. The runtime data of the execution unit can refer to monitoring metrics and / or status data generated during the execution of the execution unit. For example, the runtime data of the execution unit may include the CPU utilization (cpuUsage) allocated to the execution unit, the number of processor cores allocated to the execution unit (cpuCoreCount), the data processing rate of the execution unit (also known as the input record rate (inputRecordsQps)), memory utilization, and / or runtime, etc., which are not limited in this embodiment.

[0117] The processor core performance indicator information of a task to be analyzed is used to indicate the single-core bottleneck performance of the task to be analyzed. Through the processor core performance indicator information, the changes of a task to be analyzed before and after performance optimization can be known and / or it can be determined whether the task to be analyzed is a bottleneck task affecting the overall performance of the target distributed computing cluster.

[0118] In some embodiments, determining processor core performance indication information for the task under analysis based on runtime data from at least one execution unit performing the task under analysis includes: Based on the operational data of each execution unit, determine the average per-processor core performance indicator for each execution unit; Based on the average per-processor-core performance indicator of the execution unit performing the task to be analyzed, determine the processor-core performance indicator for the task to be analyzed.

[0119] The average performance per processor core of an execution unit can refer to the average number of records per second per processor core of that execution unit. After obtaining the average performance per processor core, the processor core performance indicator for the task to be analyzed can be determined based on the average performance per processor core and the parallelism of the task to be analyzed. Specifically, the average performance per processor core and the parallelism of the task to be analyzed can be substituted into the above formula (3) for calculation to obtain the processor core performance indicator.

[0120] In this embodiment, since the average per-processor-core performance indicator information of each execution unit can eliminate the impact of differences in computer resource specifications, the average per-processor-core performance indicator information of each execution unit is determined based on the running data of each execution unit. Based on the average per-processor-core performance indicator information of the execution unit executing the task to be analyzed, the processor core performance indicator information for the task to be analyzed is determined. This can provide a comparable evaluation standard for heterogeneous and varied tasks, thereby providing a global performance benchmark that can be compared across tasks.

[0121] In some embodiments, the operational data of an execution unit includes the processor utilization rate allocated to the execution unit, the number of allocated processor cores, and the data processing rate of the execution unit. Based on the operational data of each execution unit, average performance per processor core indicator information for each execution unit is determined, including: Multiply the processor utilization rate by the number of processor cores to obtain processor metrics information; Dividing the data processing rate by the processor performance metrics yields the average performance per processor core for the execution unit.

[0122] Specifically, the utilization rate, number of processor cores and data processing rate can be substituted into the above formula (1) for calculation to obtain the average performance indication information per processor core of the execution unit.

[0123] In related technologies, performance evaluation results are determined through basic resource metrics (CPU, memory, network I / O) or internal engine metrics (backpressure, throughput, latency). However, performance analysis and optimization using basic or internal engine metrics suffer from the problem of cumbersome metrics, which reduces the accuracy and efficiency of obtaining performance evaluation results. For example, in practice, it has been found that simply monitoring container-level memory usage cannot accurately pinpoint the problem of low heap memory utilization, requiring more refined analysis methods.

[0124] In this embodiment, the running data of the execution unit includes the processor utilization rate allocated to the execution unit, the number of processor cores allocated, and the data processing rate of the execution unit. Multiplying the processor utilization rate and the number of processor cores yields processor metric information; dividing the data processing rate and the processor metric information yields the average performance per processor core indicator information of the execution unit. This simplifies the metric for obtaining the average performance per processor core indicator information, thereby simplifying the metric for obtaining the performance evaluation result and improving the efficiency and accuracy of the obtained average performance evaluation result.

[0125] Step 502: Based on the processor core performance indication information and the parallelism of each task to be analyzed, determine the performance evaluation information of the target distributed computing cluster.

[0126] Among them, the performance evaluation information is used to evaluate the performance of the target distributed computing cluster. After obtaining the processor core performance indicator information, the processor core performance indicator information can be multiplied by the parallelism of the task to be analyzed and accumulated. The accumulated result is the sum of the parallelism of all the tasks to be analyzed to obtain the performance evaluation information. Specifically, the processor core performance indicator information can be substituted into the above formula (2) for calculation to obtain the performance evaluation information.

[0127] In related technologies, the performance evaluation process focuses on comparing different task execution engines under a preset standardized load. However, the test scenarios are very different from the complex and ever-changing business logic, data patterns, and real-time traffic in the production environment (which refers to the actual usage scenario). Therefore, it is impossible to conduct a quantitative evaluation of the performance of the target distributed computing cluster under the real load that can be compared historically.

[0128] In this embodiment, the target distributed computing cluster executes the task to be analyzed through execution units in the task execution engine. The task data includes task execution data, which in turn includes the execution data of the execution units. Based on the execution data of at least one execution unit executing the task to be analyzed, processor core performance indicator information for the task to be analyzed is determined. Based on the processor core performance indicator information for each task to be analyzed and the parallelism of the task to be analyzed, performance evaluation information of the target distributed computing cluster is determined. This provides a quantifiable, normalized, and unified objective standard for the performance of the target distributed computing cluster through performance evaluation information, enabling the performance evaluation of the target distributed computing cluster to move from subjective to rational. Furthermore, determining the performance evaluation information based on the parallelism of the task to be analyzed can better reflect the overall business processing capability of the target distributed computing cluster, providing an objective benchmark for performance comparison.

[0129] In some embodiments, after obtaining performance evaluation information, if the performance evaluation information indicates that optimization is needed, multiple intelligent agents collaboratively analyze the task data to obtain analysis results. Based on the analysis results, performance optimization information for the target distributed computing cluster is generated. Alternatively, after obtaining performance evaluation information, multiple intelligent agents can directly collaboratively analyze the task data to obtain analysis results. Based on the analysis results, performance optimization information for the target distributed computing cluster is generated. Or, the terminal device can determine performance evaluation information while simultaneously collaboratively analyzing the task data with multiple intelligent agents to obtain analysis results. Based on the analysis results, performance optimization information for the target distributed computing cluster is generated. Alternatively, multiple intelligent agents can first collaboratively analyze the task data to obtain analysis results. Based on the analysis results, performance optimization information for the target distributed computing cluster is generated. The target distributed computing cluster is optimized based on the performance optimization information, and then performance evaluation information is determined so that the performance optimization results of the target distributed computing cluster can be measured through the performance evaluation information. This embodiment does not limit the scope of the embodiments.

[0130] The process of generating performance optimization information for the target distributed computing cluster by having multiple intelligent agents collaboratively analyze task data to obtain analysis results, and then generating such information based on the analysis results, can be referred to in the above-described information generation method embodiment, which will not be repeated here.

[0131] This embodiment also provides an information generation device, such as... Figure 6 As shown, the information generation device may include: The acquisition module 601 is used to acquire task data of the task to be analyzed in the target distributed computing cluster.

[0132] Analysis module 602 is used to perform collaborative analysis of task data through multiple intelligent agents to obtain analysis results. The analysis results include the current performance problems of the target distributed computing cluster and optimization reference information. The optimization reference information includes target optimization knowledge for the current performance problem from various optimization knowledge and / or target tasks that match the task to be analyzed from the completed historical tasks of the target distributed computing cluster.

[0133] The generation module 603 is used to generate performance optimization information for the target distributed computing cluster based on the analysis results.

[0134] In some embodiments, the agent includes at least one problem-solving agent, with different problem-solving agents used to analyze and determine different current performance problems; where the optimization reference information includes target optimization knowledge, the analysis module 602 is specifically used to perform: The problem-solving intelligent agent analyzes the task data to obtain the first initial analysis result, and based on the first initial analysis result, determines the current performance problem of the target distributed computing cluster. The problem-solving agent searches for target optimization knowledge related to the current performance problem from various optimization knowledge sources; The problem-solving agent integrates the current performance problem with the target optimization knowledge to obtain the analysis results.

[0135] In some embodiments, the intelligent agent includes a problem-solving intelligent agent and a task-matching intelligent agent. When the optimization reference information includes the target task, the analysis module 602 is specifically used to perform: By analyzing task data through a problem-solving intelligent agent, the current performance issues of the target distributed computing cluster can be identified. If there is a target performance problem that meets the preset case matching conditions in the current performance problem, the task matching agent searches for a target task that matches the task to be analyzed from the completed historical tasks of the target distributed computing cluster. Analysis results are generated based on the current performance issues and the performance configuration information of the target task.

[0136] In some embodiments, the plurality of agents include a problem-solving agent and a first agent, and the analysis module 602 is specifically used to perform: Through the task context manager, the problem-handling agent is triggered in parallel to analyze the task data and obtain a second initial analysis result, which includes the current performance problem. The task context manager receives the second initial analysis results sent by each problem-handling agent. Upon receiving the second initial analysis results, the corresponding first agent is triggered to perform analysis based on the type of the second initial analysis results to obtain the third initial analysis results. The third initial analysis results include target optimization knowledge and / or target tasks. The second and third initial analysis results are assembled using the task context manager to obtain the final analysis result.

[0137] In some embodiments, the generation module 603 is specifically used to perform: Based on the analysis results, performance optimization information for the target distributed computing cluster is generated using a large language model.

[0138] In some embodiments, task data includes task metadata, which includes at least one of the following: query statement of the task to be analyzed, directed acyclic graph topology, connector parameters of upstream and downstream storage, and version feature information of the task execution engine.

[0139] In some embodiments, the agent includes a logic analysis agent, which is used to determine the task logic problem based on at least one of the query statement of the task to be analyzed and the topology of the directed acyclic graph.

[0140] In some embodiments, the agent includes a storage connector analysis agent, which determines storage connector configuration issues based on the connector parameters of upstream and downstream storage of the task to be analyzed.

[0141] In some embodiments, the agent includes an engine feature analysis agent, which is used to determine engine feature compatibility issues based on the task characteristics of the task to be analyzed and the version feature information of the task execution engine.

[0142] In some embodiments, task data includes task execution data; task execution data includes execution data of each execution unit executing the task to be analyzed, and the execution data includes at least one of processor utilization, allocated number of processor cores, data processing rate, operator backpressure status, and operator latency information.

[0143] In some embodiments, the agent includes a resource analysis agent, which is used to determine computational resource problems based on the operating data of each execution unit performing the task to be analyzed.

[0144] In some embodiments, the resource analysis agent is used to determine computational resource problems based on average per-processor core performance indicator information, wherein the average per-processor core performance indicator information is determined based on the running data of each execution unit performing the task to be analyzed, and the analysis module 602 is further used to perform: Multiply the processor utilization rate by the number of allocated processor cores to obtain processor metric information; Dividing the data processing rate by the processor performance metrics yields the average performance per processor core for the execution unit.

[0145] In some embodiments, the apparatus further includes an optimization module and a determination module. The optimization module is configured to: optimize the performance of the target distributed computing cluster based on performance optimization information. The determination module is configured to: acquire performance evaluation information of the target distributed computing cluster, the performance evaluation information including first performance evaluation information of the target distributed computing cluster before performance optimization and second performance evaluation information after performance optimization, and determine the performance optimization result of the target distributed computing cluster by comparing the first performance evaluation information and the second performance evaluation information.

[0146] In some embodiments, the target distributed computing cluster executes the task to be analyzed through the execution unit in the task execution engine. The task data includes task execution data, which in turn includes the execution data of the execution unit. The determination module is specifically used to execute: Based on the runtime data of at least one execution unit performing the task to be analyzed, determine processor core performance indication information for the task to be analyzed; Based on the processor core performance indicators and the parallelism of each task to be analyzed, the performance evaluation information of the target distributed computing cluster is determined.

[0147] In some embodiments, the determining module is specifically used to perform: Based on the operational data of each execution unit, determine the average per-processor core performance indicator for each execution unit; Based on the average per-processor-core performance indicator of the execution unit performing the task to be analyzed, determine the processor-core performance indicator for the task to be analyzed.

[0148] In practice, each of the above modules can be implemented as an independent entity or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation methods and corresponding beneficial effects of each of the above modules, please refer to the previous method embodiments, which will not be repeated here.

[0149] This embodiment also provides another information generation device, in which the target distributed computing cluster executes the task to be analyzed through the execution unit in the task execution engine. This information generation device is, for example, as shown in... Figure 7 As shown, the information generation device may include: The first determining module 701 is used to determine processor core performance indication information for the task to be analyzed based on the running data of at least one execution unit executing the task to be analyzed.

[0150] The second determining module 702 is used to determine the performance evaluation information of the target distributed computing cluster based on the processor core performance indication information for each task to be analyzed and the parallelism of the task to be analyzed.

[0151] In some embodiments, the first determining module 701 is specifically used to perform: Based on the operational data of each execution unit, determine the average per-processor core performance indicator for each execution unit; Based on the average per-processor-core performance indicator of the execution unit performing the task to be analyzed, determine the processor-core performance indicator for the task to be analyzed.

[0152] In some embodiments, the operating data of the execution unit includes the processor utilization rate allocated to the execution unit, the number of allocated processor cores, and the data processing rate of the execution unit. Based on the operating data of each execution unit, the first determining module 701 is specifically used to execute: Multiply the processor utilization rate by the number of processor cores to obtain processor metrics information; Dividing the data processing rate by the processor performance metrics yields the average performance per processor core for the execution unit.

[0153] In practice, each of the above modules can be implemented as an independent entity or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation methods and corresponding beneficial effects of each of the above modules, please refer to the previous method embodiments, which will not be repeated here.

[0154] Accordingly, embodiments of this disclosure also provide an electronic device, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device 800 includes a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, and a computer program stored on the memory 802 and executable on the processor. The processor 801 and the memory 802 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0155] The processor 801 is the control center of the electronic device 800. It connects various parts of the electronic device 800 via various interfaces and lines. By running or loading software programs and / or units stored in the memory 802, and by calling data stored in the memory 802, it executes various functions and processes data of the electronic device 800, thereby providing overall monitoring of the electronic device 800. The processor 801 can be a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc., and can implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of this disclosure.

[0156] In this embodiment of the disclosure, the processor 801 in the electronic device 800 loads the instructions corresponding to the processes of one or more application programs into the memory 802 according to the following steps, and the processor 801 runs the application programs stored in the memory 802 to realize various functions, such as: Obtain task data for the task to be analyzed in the target distributed computing cluster; The task data is analyzed collaboratively by multiple intelligent agents to obtain analysis results. The analysis results include the current performance problems of the target distributed computing cluster and optimization reference information. The optimization reference information includes target optimization knowledge for the current performance problem from various optimization knowledge and / or target tasks that match the task to be analyzed from the completed historical tasks of the target distributed computing cluster. Based on the analysis results, performance optimization information for the target distributed computing cluster is generated.

[0157] The specific implementation of each of the above operations and their corresponding beneficial effects can be found in the previous embodiments, and will not be repeated here.

[0158] Optionally, such as Figure 8 As shown, the electronic device 800 also includes: a touch display screen 803, a radio frequency circuit 804, an audio circuit 805, an input unit 806, and a power supply 807. The processor 801 is electrically connected to the touch display screen 803, the radio frequency circuit 804, the audio circuit 805, the input unit 806, and the power supply 807. Those skilled in the art will understand that... Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0159] The touch display screen 803 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 803 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 801. It can also receive and execute commands from the processor 801. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 801 to determine the type of touch event. Subsequently, the processor 801 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 803 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 803 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 803 can also be used as part of the input unit 806 to achieve input functions.

[0160] The radio frequency circuit 804 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0161] Audio circuit 805 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuit 805 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 805, converted back into audio data, and then processed by processor 801 before being transmitted via radio frequency circuit 804 to, for example, another electronic device, or output to memory 802 for further processing. Audio circuit 805 may also include an earphone jack to provide communication between peripheral headphones and electronic devices.

[0162] The input unit 806 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0163] Power supply 807 is used to supply power to various components of electronic device 800. Optionally, power supply 807 can be logically connected to processor 801 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 807 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0164] although Figure 8 As not shown in the diagram, the electronic device 800 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0165] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0166] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0167] Therefore, embodiments of this disclosure provide a computer-readable storage medium storing a plurality of computer programs, which can be loaded by a processor to execute any of the information generation methods provided in embodiments of this disclosure. The computer program can execute the steps of the following information generation method: Obtain task data for the task to be analyzed in the target distributed computing cluster; The task data is analyzed collaboratively by multiple intelligent agents to obtain analysis results. The analysis results include the current performance problems of the target distributed computing cluster and optimization reference information. The optimization reference information includes target optimization knowledge for the current performance problem from various optimization knowledge and / or target tasks that match the task to be analyzed from the completed historical tasks of the target distributed computing cluster. Based on the analysis results, performance optimization information for the target distributed computing cluster is generated.

[0168] The specific implementation of each of the above operations and their corresponding beneficial effects can be found in the previous embodiments, and will not be repeated here.

[0169] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0170] Since the computer program stored in the computer-readable storage medium can execute any of the information generation methods provided in the embodiments of this disclosure, the beneficial effects that any of the information generation methods provided in the embodiments of this disclosure can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0171] According to one aspect of this disclosure, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations of the above embodiments.

[0172] In the above embodiments of the information generation apparatus, computer-readable storage medium, electronic device, and computer program product, the descriptions of each embodiment have different focuses. Parts not described in detail in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes and beneficial effects of the information generation apparatus, computer-readable storage medium, computer program product, electronic device, and their corresponding units described above can be referred to the description of the information generation method in the above embodiments, and will not be repeated here.

[0173] The foregoing has provided a detailed description of an information generation method, apparatus, electronic device, computer-readable storage medium, and computer program product provided by the embodiments of this disclosure. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. An information generation method, characterized in that, include: Obtain task data for the task to be analyzed in the target distributed computing cluster; The task data is analyzed collaboratively by multiple intelligent agents to obtain analysis results. The analysis results include the current performance problems of the target distributed computing cluster and optimization reference information. The optimization reference information includes target optimization knowledge for the current performance problem from various optimization knowledge and / or target tasks that match the task to be analyzed from the completed historical tasks of the target distributed computing cluster. Based on the analysis results, performance optimization information for the target distributed computing cluster is generated.

2. The method according to claim 1, characterized in that, The agent includes at least one problem-handling agent, and different problem-handling agents are used to analyze and determine different current performance problems; When the optimization reference information includes the target optimization knowledge, the step of collaboratively analyzing the task data through multiple agents to obtain analysis results includes: The problem-solving agent analyzes the task data to obtain a first initial analysis result, and based on the first initial analysis result, determines the current performance problem of the target distributed computing cluster. The problem-solving agent searches for target optimization knowledge related to the current performance problem from various optimization knowledge sources. The problem-solving agent integrates the current performance problem with the target optimization knowledge to obtain the analysis results.

3. The method according to claim 1, characterized in that, The intelligent agent includes a problem-solving intelligent agent and a task-matching intelligent agent, provided that the optimization reference information includes the target task. The collaborative analysis of the task data by multiple intelligent agents to obtain analysis results includes: The problem-solving agent analyzes the task data to obtain the current performance problem of the target distributed computing cluster. If there is a target performance problem that meets the preset case matching conditions among the current performance problems, the task matching agent searches for a target task that matches the task to be analyzed from the completed historical tasks of the target distributed computing cluster. Analysis results are generated based on the current performance issues and the performance configuration information of the target task.

4. The method according to claim 1, characterized in that, The multiple intelligent agents include a problem-solving intelligent agent and a first intelligent agent. The collaborative analysis of the task data by the multiple intelligent agents to obtain analysis results includes: The task context manager triggers the problem-handling agent to analyze the task data in parallel to obtain a second initial analysis result, which includes the current performance problem. The task context manager receives the second initial analysis results sent by each of the problem-handling agents, and upon receiving the second initial analysis results, triggers the corresponding first agent to perform analysis according to the type of the second initial analysis results to obtain a third initial analysis result, wherein the third initial analysis result includes the target optimization knowledge and / or the target task; The task context manager assembles the second initial analysis result and the third initial analysis result to obtain the analysis result.

5. The method according to any one of claims 1-4, characterized in that, The process of generating performance optimization information for the target distributed computing cluster based on the analysis results includes: Based on the analysis results, the performance optimization information of the target distributed computing cluster is generated using a large language model.

6. The method according to claim 1, characterized in that, The task data includes task metadata, which includes at least one of the following: the query statement of the task to be analyzed, the directed acyclic graph topology, the connector parameters of upstream and downstream storage, and the version feature information of the task execution engine.

7. An information generation method, characterized in that, The target distributed computing cluster executes the task to be analyzed through the execution unit in the task execution engine, the method including: Based on the running data of at least one of the execution units executing the task to be analyzed, processor core performance indication information for the task to be analyzed is determined; Based on the processor core performance indication information for each task to be analyzed and the parallelism of the task to be analyzed, the performance evaluation information of the target distributed computing cluster is determined.

8. An information generation device, characterized in that, The device includes: The acquisition module is used to acquire task data of the task to be analyzed in the target distributed computing cluster; An analysis module is used to perform collaborative analysis of the task data by multiple intelligent agents to obtain analysis results. The analysis results include the current performance problems of the target distributed computing cluster and optimization reference information. The optimization reference information includes target optimization knowledge for the current performance problem from a variety of optimization knowledge and / or target tasks that match the task to be analyzed from the completed historical tasks of the target distributed computing cluster. The generation module is used to generate performance optimization information for the target distributed computing cluster based on the analysis results.

9. An electronic device, characterized in that, The system includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps of the information generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the information generation method as described in any one of claims 1 to 7.

Images