Method for dynamically adjusting streaming computation engine data traffic based on AI intelligent agent

Abstract

The invention discloses a method for dynamically adjusting streaming computation engine data traffic based on an AI (artificial intelligence) agent, which specifically comprises the following steps of: constructing the AI agent, and performing multi-dimensional acquisition and structured processing on an Flink task running log by the AI agent by utilizing a large model with time sequence cognition and causal reasoning capabilities to obtain a Flink task running log; performing learning modeling on historical operation behaviors to form an interpretable operation characteristic knowledge graph, and performing predictive adjustment on task parallelism degree parameters based on the knowledge graph; the method comprises an automatic verification and feedback mechanism which is used for dynamically correcting the reasoning deviation of a large model, and continuous optimization of Flink task operation parameters and maximization of the resource utilization rate are achieved.

Description

Technical Field

[0001] This invention relates to the field of data traffic adjustment technology, and mainly to a method for dynamically adjusting the data traffic of a streaming computing engine based on an AI agent. Background Technology

[0002] With the rapid development of information technology in recent years, the volume of business data in various industries has also grown exponentially. In order to achieve data timeliness, the streaming computing engine Flink plays an important role in data flow. In the actual data extraction process, it is necessary to work with the message queue Kafka to achieve high-speed data flow and thus ensure that a large amount of business data is provided to downstream users in a timely manner.

[0003] In practical business applications, an increasing number of data processing scenarios are shifting towards this model, especially for trajectory data. In earlier years, the timeliness of trajectory data was not high, often resulting in "outdated" analytical information. In recent years, with the gradual improvement of my country's infrastructure and the increasing convenience of travel, traveling across multiple provinces and cities within a single day has become commonplace. Therefore, the timeliness requirements for processing trajectory data must be guaranteed to keep pace with the demands of analytical tasks. For example, vehicle checkpoint trajectory data is currently processed using a Kafka + Flink technical architecture. However, due to the nature of this type of data, the data volume may increase by 1-2 times during holidays, leading to data backlog in Flink tasks and resulting in data flow delays. Furthermore, as the number of data processing tasks increases, Flink cluster resources are limited, and many tasks still occupy a large amount of computing resources during off-peak periods, resulting in significant resource waste.

[0004] In scenarios involving large-scale data flow, the Kafka+Flink architecture can maturely solve timeliness issues. Combined with Flink's checkpointing mechanism, it ensures no data loss within Flink tasks—an advantage previously unattainable by offline architectures. However, in scenarios with fluctuating business data volumes, the Kafka+Flink streaming engine architecture lacks a good solution. This type of business scenario is frequently encountered across industries, and currently, it's mostly handled manually. Before holidays or major e-commerce promotions, operations engineers increase the parallelism of tasks where the expected data volume fluctuates significantly to avoid data backlog. For each data type, data governance engineers need to calculate the average data volume and size over a period of time, define a fixed Flink task parallelism, and manually adjust unreasonable Flink task parallelism during periods of low data volume to ensure reasonable resource utilization while maintaining normal Flink task operation. This undoubtedly increases the workload of data operations engineers, especially when dealing with diverse data types and large data processing volumes. Summary of the Invention

[0005] To address the above problems and considering the rapid development of artificial intelligence technology in recent years, this invention proposes a method to replace manual judgment and maintenance by constructing an AI agent. The agent dynamically adjusts the parallelism of Flink tasks by analyzing historical data traffic and determining holiday periods. According to a first aspect of this invention, a method for dynamically adjusting the data traffic of a streaming computing engine based on an AI agent is proposed, with the following specific steps: An AI agent is constructed. The AI ​​agent collects and structures Flink task operation logs from multiple dimensions, uses a large model with time series cognition and causal reasoning capabilities to learn and model historical operation behavior, forms an interpretable operation feature knowledge graph, and makes predictive adjustments to the task parallelism parameters based on the knowledge graph. The method includes an automatic verification and feedback mechanism to dynamically correct inference biases in large models, thereby achieving continuous optimization of Flink task execution parameters and maximizing resource utilization.

[0006] Furthermore, the multi-dimensional collection of the logs includes: real-time data traffic metrics, node backpressure rate and task latency metrics, and cluster CPU, memory and network parameters; after semantic parsing and timestamp alignment, the collected logs are converted into a multimodal input format that can be recognized by large models.

[0007] Furthermore, the large model integrates historical operating data and external constraints during the learning process. These external constraints include: Flink cluster resource limit parameters, Kafka topic partition configuration, and holiday calendar information, in order to model the data traffic fluctuation characteristics under special time windows.

[0008] Furthermore, the large model learning process includes the following: Establish a data traffic pattern model for holidays and identify the correlation between holidays and data traffic; Establish a holiday task backpressure characteristic model to identify the distribution pattern of computing power bottlenecks in different time periods; Identify the resource utilization boundary of a task under a given degree of parallelism and determine whether there is resource waste. By combining linear regression with attention mechanisms, a nonlinear proportional relationship model between data flow and parallelism is established.

[0009] Furthermore, the verification includes: performance difference analysis before and after parallelism adjustment, task topology consistency check, CheckPoint mechanism effectiveness detection, and adaptive correction of large model parameters through feedback weight update mechanism.

[0010] Furthermore, the verified large model output results are executed through an automated scheduling module, which distinguishes resource priorities based on node roles, dynamically adjusts the parallelism parameters of each node, and restarts the task to make the adjustments take effect immediately.

[0011] Furthermore, the large model constructs a task operation optimization knowledge graph based on the holiday traffic patterns, backpressure characteristics, resource utilization patterns, and parallelism ratios obtained from the analysis. This graph supports the rapid self-learning and automatic optimization of subsequent Flink tasks.

[0012] According to a second aspect of the present invention, a computer program product is provided, on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0013] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects: In the Kafka+Flink streaming data processing architecture, a dynamic data flow adjustment mechanism has been added, which can save computing resources to a greater extent and avoid waste of hardware resources.

[0014] During implementation, data governance engineers no longer need to precisely calculate the parallelism of Flink tasks, reducing the complexity and rework rate of governance work.

[0015] This addresses data backlog issues caused by sudden increases in data flow during streaming computing and helps avoid data latency.

[0016] In the operation and maintenance of a large number of Flink tasks, it is no longer necessary for operation and maintenance personnel to perform operation and maintenance on each task individually. They only need to focus on the overall resource usage of the Flink cluster, thereby reducing the complexity of operation and maintenance work and thus reducing the enterprise's operation and maintenance costs.

[0017] This invention is used in a streaming data governance architecture based on Kafka+Flink and is applicable to various industries to ensure data timeliness, especially in scenarios with high timeliness requirements and occasional sudden increases in data traffic. Attached Figure Description

[0018] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0019] Figure 1 A flowchart illustrating a method for dynamically adjusting the data flow of a streaming computing engine based on an AI agent, according to an embodiment of the present invention, is shown.

[0020] Figure 2 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0021] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] Figure 1 A flowchart illustrating a method for dynamically adjusting the data flow of a streaming computing engine based on an AI agent, according to an embodiment of the present invention, is shown below. Figure 1 As shown: S1. Log collection; The first step in the AI ​​agent is to collect logs from Flink tasks. This includes collecting data traffic, backpressure data for each stage of the Flink task, and system parameters of the Flink cluster. The collected data is stored in a database and also provided to large models for analysis. This step mainly involves collecting basic data and organizing it into a format that large models can recognize for analysis.

[0024] The log collection includes: real-time data traffic metrics, node backpressure rate, task latency metrics, and cluster CPU, memory, and network parameters. After semantic parsing and timestamp alignment, the collected logs are converted into a multimodal input format recognizable by large models.

[0025] The main task of step S1 is to collect basic data and organize the data into a format that can be recognized by the large model for recognition and analysis.

[0026] S2, Large Model Analysis; Before feeding real-time data into the large model, it is necessary to provide the large model with historical execution logs of Flink tasks for analysis, and to provide the large model with some analysis constraints, such as: Flink cluster memory parameters, official holiday schedules, number of Kafka topic partitions on the source side, etc.

[0027] The large model integrates historical operational data with external constraints during the learning process. These external constraints include Flink cluster resource limits, Kafka topic partitioning configurations, and holiday calendar information, used to model data traffic fluctuation characteristics under specific time windows. Based on these data traffic fluctuation characteristics, the large model forms an interpretable operational feature knowledge graph, and predictively adjusts task parallelism parameters based on this knowledge graph. The large model analysis process specifically includes the following steps: S2.1 Analyze the historical data traffic logs of Flink tasks; Based on data traffic and holiday data, we analyze the differences between data traffic during holidays and data traffic on weekdays, thereby gaining knowledge about the relationship between holidays and data traffic.

[0028] S2.2 Analyze the backpressure logs of historical nodes in Flink tasks; Based on 100% back pressure data and holiday data, we analyze the differences between back pressure during holidays and normal times to obtain knowledge about the relationship between holidays and back pressure.

[0029] In Flink, backpressure occurs when downstream tasks lack sufficient processing capacity, preventing data from being processed in a timely manner. Backpressure logs are part of Flink system monitoring and typically include metrics such as processing rate per node, cache queue length, and latency. Flink automatically detects backpressure and records relevant data in its logs to help developers identify bottlenecks.

[0030] Typical manifestations of back pressure: Increased latency: The reduced processing speed of the data stream leads to increased processing latency.

[0031] Queue backlog: Data accumulates in the queues of various nodes, further increasing latency.

[0032] Excessive resource consumption: Processing tasks consumes more computing resources (such as memory and CPU), leading to excessive consumption of cluster resources.

[0033] The differences in back pressure between holidays and normal working days mainly come from the following aspects: Data traffic fluctuations: During holidays, user behavior and data traffic patterns in many systems change significantly. For example, promotional activities on e-commerce platforms and surges in social media traffic can lead to sudden increases in data traffic. Under normal circumstances, these traffic fluctuations are relatively stable.

[0034] Different system loads: During holidays, user activity may increase significantly, leading to increased task processing load. If the system resources are insufficient, more back pressure may occur.

[0035] Task Prioritization and Resource Allocation: During holidays, certain tasks may require priority processing, such as order processing and payment verification during promotional activities. This can lead to resource contention and backpressure on other tasks.

[0036] System maintenance and updates: During holidays, due to holidays and other reasons, maintenance or updates to some systems may affect the operation of the business and increase the probability of back pressure.

[0037] To analyze the differences in back pressure in depth, a detailed log analysis can be performed using the following steps: S2.2.1 Collect back pressure log data; Select backpressure log data from holidays (such as Spring Festival and National Day) and regular working days. The log data should cover a certain time span to ensure that it can capture traffic fluctuations and changes in system load.

[0038] Key metrics can be extracted from Flink's backpressure logs, such as: Node processing rate (Throughput): the processing speed of each task; Queue length: the amount of data to be processed in a task; Latency: the time from data input to task completion; Resource utilization (CPU, Memory): the resource usage of task nodes.

[0039] S2.2.2 Calculate the back pressure occurrence rate; The occurrence rate of back pressure can be calculated using the following formula: in, Indicates the incidence of back pressure. This represents the number of back pressure events that occur within a given time period. This indicates the total number of events processed within that time period.

[0040] S2.2.3 Comparative analysis between holidays and regular days; By comparing peak traffic volumes during holidays with those on weekdays, statistical methods are used to analyze traffic fluctuations. This includes analyzing whether traffic fluctuations are more severe during holidays and whether back pressure is related to traffic fluctuations. The frequency and duration of back pressure during holidays and weekdays are also statistically analyzed to assess whether back pressure is more frequent or severe during holidays.

[0041] S2.2.4 Establish a back pressure model for holidays; By comparing backpressure levels during holidays with those during normal times, a backpressure optimization model based on traffic prediction and task scheduling can be established. This model should include the following components: Data traffic forecasting: Using historical data to predict traffic during holidays and understand the trend of traffic fluctuations in advance.

[0042] Adaptive Parallelism Adjustment: Dynamically adjusts the parallelism of tasks based on traffic forecasts and system load to avoid severe backpressure during traffic surges.

[0043] Resource scheduling optimization: Based on task load and backpressure logs, dynamically adjust the allocation of cluster resources, prioritize resources for critical tasks, and avoid global resource bottlenecks.

[0044] S2.3 Analyze the historical back pressure of the task under the current parallelism; The analysis examines whether back pressure or doubling of data traffic occurred during the historical execution of the task under a given degree of parallelism, identifies the resource utilization boundary of the task under a given degree of parallelism, and thus determines whether there is any waste of resources.

[0045] S2.4 Analyze the parallelism requirements of Flink tasks within the data traffic range; Based on the range of data traffic, we analyze how much parallelism is needed to process the data without wasting resources. By combining linear regression with attention mechanisms, we establish a non-linear proportional relationship model between data traffic and parallelism, thereby obtaining knowledge about the proportional relationship between data traffic and parallelism.

[0046] S2.4.1 The fundamental relationship between parallelism and data flow; Flink's parallelism settings affect the number of execution instances per operation (i.e., the number of parallel threads per operation). For a given Flink task, its processing capacity (throughput) and the required parallelism are closely related. Ideally, the parallelism of a task should be dynamically adjusted based on real-time data traffic to ensure that the system avoids overload while processing data without wasting resources.

[0047] In its most basic case, the relationship between data flow and parallelism can be expressed by the following simple linear relationship: Where C represents the amount of data (processing capacity) that each Flink execution instance can process per unit time, in units of data items per second, and is a constant obtained through statistical analysis of historical data traffic and task processing capacity; Q represents the input data traffic, in units of data items per second; and P represents the required parallelism, i.e. the number of execution instances allocated to the Flink task.

[0048] S2.4.2, Fluctuations in data traffic; In real-world systems, data traffic typically fluctuates, necessitating dynamic adjustments to the parallelism based on these changes. During actual deployment, the following issues may arise: Large fluctuations in data traffic can lead to improper parallelism settings and potentially waste of resources.

[0049] System resource limitations (such as CPU, memory, etc.) also limit the maximum value of parallelism.

[0050] Therefore, this invention divides data traffic into multiple intervals and sets different parallelism requirements for different intervals. Assume the data traffic fluctuation interval is... Based on the current data traffic The parallelism is dynamically adjusted. The specific adjustment formula is as follows: in, This indicates the current real-time data flow. This represents the minimum parallelism, ensuring the system can still operate normally under minimal load (e.g., a minimum of 1). This indicates rounding up to the nearest integer, ensuring that the parallelism is an integer value.

[0051] Based on this, the parallelism can be dynamically adjusted according to changes in data traffic, avoiding resource waste and improving system flexibility.

[0052] S2.4.3 Dynamic adjustment mechanism and resource utilization; In addition to the aforementioned proportional relationships, practical systems also need to consider the limitations of cluster resources and the specific characteristics of the tasks. To better control parallelism, the following mechanisms can be introduced: Adaptive traffic monitoring: The system should monitor changes in data traffic in real time and estimate future traffic trends through a sliding window, thereby making adjustments to the parallelism in advance.

[0053] Resource constraints: Consider the cluster's resource limitations, such as CPU, memory, and network bandwidth. For example, when system resources reach their limit, even if data traffic continues to increase, the parallelism should not be increased further. In this case, Flink will implement rate limiting strategies or forward some traffic to other tasks.

[0054] Task load awareness: The complexity of a task also affects the parallelism setting. For example, some tasks may require more resources to process a single data stream due to their high computational load or complex state storage. Therefore, different computational capability models can be used to estimate parallelism requirements for different types of tasks.

[0055] By analyzing parallelism requirements based on data traffic ranges, Flink can dynamically adjust task parallelism according to real-time traffic, ensuring high efficiency and avoiding wasted computing resources. This guarantees the stability, performance, and scalability of large-scale data processing systems.

[0056] S3. Results Analysis; The results of the large model analysis are validated to confirm their accuracy. This mainly involves judging whether adjustments to the parallelism level, the location of parallelism adjustments, and whether the Flink task's checkpoint mechanism is enabled, taking into account the actual production environment. Unreasonable results are manually analyzed to determine whether the issue stems from the production environment or the large model analysis itself, thereby adjusting the accuracy of the large model analysis results.

[0057] In one embodiment of the present invention, the CheckPoint mechanism specifically includes: S3.1 Configuring CheckPoint; Set the CheckPoint interval to 5 minutes and configure asynchronous storage (e.g., HDFS). Enable task state saving to ensure that Flink saves the task state (traffic data statistics) to the specified storage every time a CheckPoint is triggered.

[0058] S3.2, First CheckPoint Trigger; The Flink stream processing task is launched to begin processing simulated network traffic data. Each time an event is processed, the system updates traffic statistics and checks for anomalies. After 5 minutes of operation, Flink triggers the first Checkpoint and saves the current task state to HDFS.

[0059] S3.3 Simulation of task failures and recovery; Randomly simulate task node failure, stopping one node of the Flink task to simulate network latency or node crash. When a failure occurs, record the system's traffic processing status (e.g., the last processed event, current status data, etc.).

[0060] After a node crashes, Flink restores the task from the most recent checkpoint (i.e., the state 5 minutes ago). After the system restarts, it checks whether the Flink task has successfully restored to the state it was in when the checkpoint was saved.

[0061] Record the time from node failure to recovery, and calculate the recovery process time span. During recovery, check if the status data is completely consistent and verify that no traffic statistics have been lost.

[0062] S3.4, State consistency verification; During the recovery process, the recovered state is compared with the previously saved CheckPoint state to ensure that the recovered state is consistent with the saved CheckPoint state.

[0063] The data saved by CheckPoint and the data after recovery are hashed and compared to ensure that no data is lost or abnormal.

[0064] In the restored state, replay the traffic data for a period of time to confirm whether the restored state can correctly reflect all traffic processing results.

[0065] Check whether the traffic statistics after recovery (such as total traffic, number of abnormal traffic, etc.) are consistent with those at the CheckPoint.

[0066] S4. Modify the task; Based on the large model analysis and results analysis, the parallelism of the corresponding nodes for the Flink task is adjusted, and the task is restarted to ensure the adjustment takes effect immediately. The scheduling module distinguishes resource priorities according to node roles and dynamically adjusts the parallelism parameters of each node to ensure that the task runs at the adjusted parallelism.

[0067] The validated large model outputs are executed through the automated scheduling module, further optimizing Flink tasks. The scheduling module prioritizes resources based on node roles, dynamically adjusts the parallelism of each node according to new parallelism parameters, and restarts the task to ensure the adjustments take effect immediately.

[0068] The large model constructs a task execution optimization knowledge graph based on the analyzed holiday traffic patterns, backpressure characteristics, resource utilization patterns, and parallelism ratios. This graph supports rapid self-learning and automatic tuning of subsequent Flink tasks. Through this continuous optimization process, the running parameters of Flink tasks can be dynamically adjusted to maximize resource utilization.

[0069] The following is for reference. Figure 2 It shows a schematic diagram of the structure of a computer system 200 suitable for implementing electronic devices according to embodiments of the present application. Figure 2 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0070] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 202 or programs loaded from storage section 208 into random access memory (RAM) 203. The RAM 203 also stores various programs and data required for the operation of the system 200. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0071] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a liquid crystal display (LCD) and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card and a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to I / O interface 205 as needed. A removable medium 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 210 as needed so that computer programs read from it can be installed into storage section 208 as needed.

[0072] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the computer program is executed by central processing unit (CPU) 201, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0073] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0074] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0075] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0076] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: construct an AI agent, which, through multi-dimensional collection and structured processing of Flink task runtime logs, utilizes a large model with time-series cognition and causal reasoning capabilities to learn and model historical runtime behavior, forming an interpretable runtime feature knowledge graph, and predictively adjusts task parallelism parameters based on the knowledge graph; wherein, the method includes an automatic verification and feedback mechanism for dynamically correcting inference biases of the large model, achieving continuous optimization of Flink task runtime parameters and maximizing resource utilization.

[0077] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for dynamically adjusting the data flow of a streaming computing engine based on an AI agent, characterized in that, include: An AI agent is constructed. The AI ​​agent collects and structures Flink task operation logs from multiple dimensions, uses a large model with time series cognition and causal reasoning capabilities to learn and model historical operation behavior, forms an interpretable operation feature knowledge graph, and makes predictive adjustments to the task parallelism parameters based on the knowledge graph. It also includes an automatic verification and feedback mechanism to dynamically correct inference biases in large models, thereby achieving continuous optimization of Flink task running parameters and maximizing resource utilization.

2. The method according to claim 1, characterized in that, The multi-dimensional collection of logs includes: real-time data traffic metrics, node backpressure rate and task latency metrics, and cluster CPU, memory and network parameters; after semantic parsing and timestamp alignment, the collected logs are converted into a multimodal input format that can be recognized by large models.

3. The method according to claim 1, characterized in that, The large model integrates historical running data and external constraints during the learning process. These external constraints include: Flink cluster resource limit parameters, Kafka topic partition configuration, and holiday calendar information, which are used to model the data traffic fluctuation characteristics under special time windows.

4. The method according to claim 1, characterized in that, The large model learning process includes the following: Establish a data traffic pattern model for holidays and identify the correlation between holidays and data traffic; Establish a holiday task backpressure characteristic model to identify the distribution pattern of computing power bottlenecks in different time periods; Identify the resource utilization boundary of a task under a given degree of parallelism and determine whether there is resource waste. By combining linear regression with attention mechanisms, a nonlinear proportional relationship model between data flow and parallelism is established.

5. The method according to claim 1, characterized in that, The verification includes: performance difference analysis before and after parallelism adjustment, task topology consistency check, CheckPoint mechanism effectiveness detection, and adaptive correction of large model parameters through feedback weight update mechanism.

6. The method according to claim 1, characterized in that, The verified output of the large model is executed through an automated scheduling module. The scheduling module distinguishes resource priorities based on node roles, dynamically adjusts the parallelism parameters of each node, and restarts the task so that the adjustments take effect immediately.

7. The method according to claim 1, characterized in that, The large model constructs a task operation optimization knowledge graph based on the holiday traffic patterns, backpressure characteristics, resource utilization patterns, and parallelism ratios obtained from the analysis. This graph supports the rapid self-learning and automatic tuning of subsequent Flink tasks.

8. A computer program product, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

9. A computing system, characterized in that, It includes a processor and a memory, the processor being configured to perform the method as described in any one of claims 1-7.

Images