Link generation method and device, reinforcement learning method and device, data processing method and device and storage medium

Abstract

The embodiment of the invention provides a link generation method, a reinforcement learning method, a data processing method, equipment and a storage medium. The method comprises the steps of obtaining a first task demand of a first data processing task; according to a first task demand, matching a plurality of required first operators from the plurality of alternative operators, and generating a first to-be-optimized data processing link based on the plurality of first operators; according to the first to-be-optimized data processing link, the performance estimation of the first to-be-optimized data processing link and the current operation states of the plurality of alternative operators, optimizing the first to-be-optimized data processing link by using a trained reinforcement learning model to obtain a first target data processing link; wherein the first target data processing link is used for processing the first data processing task.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a link generation, reinforcement learning, data processing method, device, and storage medium. Background Technology

[0002] In the era of continuous development of Artificial Intelligence (AI), big data and AI complement each other, becoming the core driving force for technological progress. Data is the foundation of AI, and its quality and quantity directly determine the performance and reliability of AI models. Especially in the training of large models, high-quality, large-scale data can improve the model's reasoning ability and reduce uncertainty in the training process.

[0003] Current big data platforms typically rely on experts to manually configure data processing pipelines or automatically generate them based on pre-configured rule bases. However, this approach suffers from high labor costs and sometimes results in poor performance of the constructed pipelines due to the limitations of experts' knowledge and thinking. Summary of the Invention

[0004] This specification provides, in various aspects, a link generation, reinforcement learning method, device, storage medium, and program product for improving the execution performance of automatically generated data processing links.

[0005] The first aspect of this specification provides a link generation method, including: Obtain the primary task requirements for the first data processing task; According to the first task requirements, multiple first operators are matched from multiple candidate operators and a first data processing link to be optimized is generated based on the multiple first operators; Based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, the first data processing link to be optimized is optimized using a trained reinforcement learning model to obtain the first target data processing link. The first target data processing link is used to process the first data processing task.

[0006] The second aspect of this specification provides a reinforcement learning method, including: At the start of this round, the second data processing link to be optimized corresponding to the second data processing task is obtained. The second data processing link to be optimized is determined based on the second task requirements of the second data processing task and multiple alternative operators. Based on the current running status of the second data processing link to be optimized, the multiple candidate operators, and the performance estimate of the second data processing link to be optimized, the optimization action is determined using the reinforcement learning model to be trained. Based on the optimization action, the second data processing link to be optimized is optimized to obtain a new second data processing link to be optimized; The reinforcement learning model to be trained is optimized based on the environmental reward after the optimized action. If the current round does not meet the termination condition, the second data processing link to be optimized after this round is taken as the new second data processing link to be optimized, and the process returns to the step of determining the optimization action based on the current running status of the second data processing link to be optimized, the multiple candidate operators, and the performance estimate of the second data processing link to be optimized, using the reinforcement learning model to be trained. The current turn ends when the end condition is met.

[0007] A third aspect of this specification provides a data processing method, comprising: Obtain a first data processing task, wherein the first data processing task includes a first task requirement; According to the first task requirements, multiple first operators are matched from multiple candidate operators and a first data processing link to be optimized is generated based on the multiple first operators; Based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, the first data processing link to be optimized is optimized using a trained reinforcement learning model to obtain the first target data processing link. Run the first target data processing link to process the data source to be processed corresponding to the first data processing task.

[0008] A fourth aspect of this specification provides a link generation apparatus, comprising: The acquisition module is used to acquire the first task requirements of the first data processing task. The generation module is used to match multiple required first operators from multiple candidate operators according to the first task requirements and generate a first data processing link to be optimized based on the multiple first operators; The optimization module is used to optimize the first data processing link to be optimized based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, using a trained reinforcement learning model to obtain the first target data processing link. The first target data processing link is used to process the first data processing task.

[0009] A fifth aspect of this specification provides a reinforcement learning device, comprising: The acquisition module is used to acquire the second data processing link to be optimized corresponding to the second data processing task at the beginning of this round. The second data processing link to be optimized is determined based on the second task requirements of the second data processing task and multiple alternative operators. The determination module is used to determine the optimization action based on the second data processing link to be optimized, the current running status of the multiple candidate operators, and the performance estimate of the second data processing link to be optimized, using the reinforcement learning model to be trained. The link optimization module is used to optimize the second data processing link to be optimized based on the optimization action to obtain a new second data processing link to be optimized. The model optimization module is used to optimize the reinforcement learning model to be trained based on the environmental reward after the optimization action. The decision module is used to, when the current round does not meet the termination condition, take the optimized second data processing link as the new second data processing link to be optimized, and return to execute the step of determining the optimization action based on the current running status of the second data processing link to be optimized, the multiple candidate operators, and the performance estimate of the second data processing link to be optimized, using the reinforcement learning model to be trained; the acquisition module is used to end the current round when the current round meets the termination condition.

[0010] A sixth aspect of this specification provides a data processing apparatus, comprising: The acquisition module is used to acquire a first data processing task, wherein the first data processing task includes a first task requirement; The generation module is used to match multiple required first operators from multiple candidate operators according to the first task requirements and generate a first data processing link to be optimized based on the multiple first operators; The optimization module is used to optimize the first data processing link to be optimized based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, using a trained reinforcement learning model to obtain the first target data processing link. The running module is used to run the first target data processing link to process the data source to be processed corresponding to the first data processing task.

[0011] A seventh aspect of this specification provides an electronic device, comprising: a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method described in any of the preceding embodiments.

[0012] An eighth aspect of this specification provides a computer-readable storage medium storing a computer program that, when executed by a computer, enables the implementation of any of the methods described above.

[0013] A ninth aspect of this specification provides a computer program product including a computer program that, when executed by a processor, implements the method described in any of the preceding descriptions.

[0014] In the technical solution provided in the embodiments of this specification, multiple first operators are matched from multiple candidate operators according to task requirements, and a data processing link to be optimized is generated based on the multiple first operators. Then, based on the data processing link to be optimized, its performance estimate, and the current running status of the multiple candidate operators, a trained reinforcement learning model is used to optimize the data processing link to be optimized, thereby obtaining the optimized data processing link. It can be seen that this solution, based on reinforcement learning technology, achieves intelligent optimization of the data processing link, thereby improving link performance. Attached Figure Description

[0015] The accompanying drawings, which are provided to further illustrate this specification, form part of this specification.

[0016] Figure 1 A flowchart illustrating a link generation method provided as an exemplary embodiment of this specification; Figure 2 A flowchart illustrating an exemplary embodiment of the reinforcement learning method provided in this specification; Figure 3 A flowchart illustrating a data processing method provided in an exemplary embodiment of this specification; Figure 4 A schematic diagram of the reinforcement learning training architecture provided in an exemplary embodiment of this specification; Figure 5 A flowchart illustrating a reinforcement learning method provided in yet another embodiment of this specification; Figure 6 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments, and not all of the embodiments. Based on these embodiments, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0018] It should be noted that, in the cases involving user information in the embodiments of this specification, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this specification (including but not limited to reinforcement learning models) comply with relevant laws and standards.

[0019] Currently, big data platforms face the following challenges: 1. The data is massive and quality control is complex.

[0020] Faced with complex and diverse data sources (such as user behavior data, remote sensing data, and IoT data), how to quickly clean, integrate, and process these data has become a major challenge for big data platforms. Inappropriate processing can lead to a significant decline in data usability and accuracy.

[0021] 2. The problem of combinatorial diversity of operators.

[0022] Data processing pipelines typically involve operators such as data cleaning, transformation, grouping, and aggregation. The combination of different operators directly affects the final data processing quality, execution cost, and runtime. Current data processing pipeline construction schemes rarely achieve globally optimal pipelines.

[0023] 3. Task performance and resource utilization cannot be balanced.

[0024] During data processing, operator tasks require underlying resources (including CPU, memory, graphics card, etc.). How to dynamically optimize the execution order of operators, balance task performance and computing resource costs, and further improve system throughput remains a challenge that urgently needs to be addressed.

[0025] To address or partially address the aforementioned technical problems, embodiments of this specification provide a data processing link construction scheme based on reinforcement learning, aiming to improve link design efficiency and execution performance through automation and intelligent technologies.

[0026] Before introducing the specific details of the embodiments in this specification, a brief explanation of the reinforcement learning technique mentioned herein is provided. Reinforcement learning (RL) is a machine learning method. Its basic framework is the Markov decision process, which allows an agent to learn a better policy through trial and error in its interaction with the environment. The agent performs actions in the environment and receives feedback, i.e., rewards, based on the results of these actions. These reward signals guide the agent to adjust its policy to maximize long-term cumulative rewards. In the embodiments of this specification, the agent includes a reinforcement learning model, which includes a policy obtained based on reinforcement learning. The reinforcement learning process can involve multiple episodes. An episode, also called a "scene" or "trial" in reinforcement learning, refers to a complete, cyclical interaction between the agent and the environment. An episode typically begins with an initial state (usually generated by a reset of the environment) and ends with the attainment of a certain termination state, containing all chronologically ordered interaction steps from start to finish. In this manual, a round is the process of optimizing the data processing link corresponding to a certain data processing task from the beginning to the end.

[0027] The reinforcement learning method and data processing link generation method provided in the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0028] Figure 1 This is a flowchart illustrating the data processing link generation method provided in the embodiments of this specification. Figure 1 As shown, the method includes: 100. Obtain the first task requirements for the first data processing task.

[0029] 102. Based on the first task requirements, match multiple first operators from multiple candidate operators and generate a first data processing link to be optimized based on the multiple first operators.

[0030] 104. Based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, the first data processing link to be optimized is optimized using a trained reinforcement learning model to obtain the first target data processing link.

[0031] The first target data processing link is used to process the first data processing task.

[0032] Of the 100 requirements mentioned above, the first task requirement indicates the data source to be processed, its parameters, data processing objectives, performance expectations, etc. Data source parameters may include the data source type and format (or schema). Data source types may include database tables, JSON (JavaScript Object Notation) files, CSV (Comma-Separated Values) data, etc. Data source formats include fields, field types, field descriptions, lineage, etc. Data processing objectives may include the required data processing type and data output requirements. Data processing types may include, but are not limited to, data cleaning, feature transformation, grouping, etc. Performance expectations may include expectations regarding runtime and / or computational resource consumption, for example: "We hope for the shortest possible runtime and / or the lowest possible computational resource consumption."

[0033] The first task requirement for the aforementioned first data processing task can be submitted by the user.

[0034] In some embodiments of the above 102, multiple first operators can be matched from multiple candidate operators based on preset rules and according to the first task requirements, and a first data processing link to be optimized can be generated based on the multiple first operators. The preset rules can be rules written by experts based on experience. For example, according to the first task requirements, the first data processing task can be divided into multiple subtasks. For each subtask among the multiple first operators, operators capable of executing the subtask are matched from multiple candidate operators to obtain multiple first operators. Based on the data dependencies between the multiple subtasks, the multiple first operators are connected to obtain the first data processing link to be optimized.

[0035] In other embodiments, heuristic algorithms can be used to match multiple first operators from multiple candidate operators based on the first task requirements, and generate a first data processing link to be optimized based on the multiple first operators. Specifically, semantic parsing is performed on the first task requirements to extract the data processing target, data source type, data source format, and performance expectations; according to the operator library, multiple candidate operators are scored and filtered using heuristic rules to match multiple first operators that meet the task requirements; based on data flow dependencies and topological constraints between operators, a greedy strategy or backtracking search is used to construct a legal operator execution sequence; the generated candidate links are subjected to feasibility verification and optimization (such as removing redundant operators and merging similar operations), and the first data processing link to be optimized is output.

[0036] In step 104 above, a first state vector is determined based on the first data processing link to be optimized, its performance estimate, and the current running state of the multiple candidate operators. This first state vector is then input into a reinforcement learning model, which outputs optimization actions. The first data processing link to be optimized is then optimized according to these optimization actions. The optimization actions may include: deleting operators, adding operators, adjusting the order of adjacent operators, or replacing operators.

[0037] The reinforcement learning model described above is trained to optimize link performance. During training, the model is optimized based on the environmental rewards resulting from the optimization actions output by the reinforcement learning model. These environmental rewards are determined based on a performance estimate of the optimized data processing link. The performance estimate may include one or more of the following: runtime estimate, resource consumption estimate, output quality estimate, and task success rate estimate.

[0038] Optionally, the reinforcement learning model can be implemented based on deep neural networks, such as Deep Q-Network (DQN). The core idea of ​​Deep Q-Network is to combine the Q-Learning algorithm in reinforcement learning with deep neural networks, using neural networks to approximate the value function of a high-dimensional state space. Neural networks containing multiple activation functions can achieve non-linear function approximation, making them very powerful function approximators. DQN uses a deep neural network Q(s,a) to estimate the value (Q-value) of each action, and then selects the better action based on the Q-value. It can well support the mapping between link states and optimized actions, and can handle such complex state spaces well. The Deep Q-Network includes a state encoding sub-network and a Q-value output sub-network. The state encoding sub-network encodes the model input into a fixed-dimensional state feature vector, and the Q-value output sub-network takes the aforementioned state feature vector as input and outputs the Q-value of each candidate action in the action space. The state encoding sub-network contains convolutional neural network (CNN) layers or recurrent neural network (RNN) layers, and the Q-value output sub-network contains fully connected neural network layers. During training, two independent sets of deep Q-network parameters are maintained simultaneously: one for the online network and one for the target network. Both have the same sub-network structure as described above. The online network is used for interactive sampling and real-time parameter updates; the target network provides a stable value estimation target, and its parameters are obtained by periodically synchronizing the online network parameters.

[0039] Optionally, feature extraction is performed on the first data processing link to be optimized to obtain its topological structure features. Based on the topological structure features, the current running state of the multiple operators, and the performance estimate of the first data processing link to be optimized, a trained reinforcement learning model is used to optimize the first data processing link to obtain a first target data processing link. For example, a graph neural network can be used to extract features from the first data processing link to be optimized to obtain its topological structure features. The embodiments in this specification, through topological structure feature extraction, help reduce data dimensionality and thus reduce computational complexity.

[0040] In some embodiments, a Graph Convolutional Network (GCN) can be used to extract features from the first data processing link to be optimized, thereby obtaining the topological features of the first data processing link to be optimized. Taking the first data processing link to be optimized as a Directed Acyclic Graph (DAG) as an example, a GCN is used to extract multi-level topological features from the DAG to obtain node embedding representations that contain task dependencies and structural semantics. Specifically, considering the directional characteristics of the DAG, the adjacency relationship of each node is decomposed into a predecessor adjacency matrix and a successor adjacency matrix, which aggregate the upstream dependency information and downstream influence information of each node, and combine them with the initial attributes of the node (i.e., operator attributes, such as operator type, computational cost, or resource requirements) for multi-level message passing. The predecessor adjacency matrix represents the input dependency of the node (who points to it), and the successor adjacency matrix represents the output influence of the node (who it points to), and the two together characterize the directionality of the edges in the directed graph. Each layer of graph convolution operation performs a linear transformation on the aggregated features through a learnable weight matrix, and generates a higher-order representation through a non-linear activation function. After multiple iterations, the dense vectors of each node are output as its topological features, fully characterizing its role and position in the global dependency graph.

[0041] In the embodiments of this specification, the data processing link is a directed data flow graph that transforms the data to be processed into the required data. It consists of multiple operators connected in a specific order, where each operator performs an independent and functionally defined processing operation on the input data and passes the output result to the next operator. The overall structure of the link is determined by the data dependencies between operators, ensuring the correctness of the data processing logic and the execution efficiency. Typically, the data processing link is represented as a directed acyclic graph (DAG), where nodes represent operators, directed edges represent the direction of data flow, and the acyclic nature ensures that the data processing flow will not fall into an infinite loop, allowing tasks to start and eventually end.

[0042] In the technical solution provided in the embodiments of this specification, multiple first operators are matched from multiple candidate operators according to task requirements, and a data processing link to be optimized is generated based on the multiple first operators. Then, based on the data processing link to be optimized, its performance estimate, and the current running status of the multiple candidate operators, a trained reinforcement learning model is used to optimize the data processing link to be optimized, thereby obtaining the optimized data processing link. It can be seen that this solution, based on reinforcement learning technology, achieves intelligent optimization of the data processing link, thereby improving link performance.

[0043] In some embodiments, the reinforcement learning module can iteratively output multiple optimization actions to optimize the first data processing link to be optimized, thereby obtaining the final first target data processing link. Specifically, step 104 above, "based on the first data processing link to be optimized, its performance estimate, and the current running state of the multiple candidate operators, optimize the first data processing link to be optimized using the trained reinforcement learning model to obtain the first target data processing link," can be implemented using the following steps: 1040. Determine the first state vector based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators.

[0044] 1042. Input the state vector into the reinforcement learning model.

[0045] 1044. Execute the optimization action output by the reinforcement learning model to optimize the first data processing link to be optimized.

[0046] 1046. When the optimization process does not meet the termination condition, the first data processing link to be optimized after this optimization is taken as the new first data processing link to be optimized, and the process returns to step 1040.

[0047] 1048. When the optimization process meets the termination condition, the first data processing link to be optimized after this optimization is determined as the first target data processing link.

[0048] In some optional implementations of the above 1040, the first state vector may include: the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running state of the plurality of candidate operators.

[0049] In other alternative implementations, feature extraction can be performed on the first data processing link to be optimized to obtain its topological structure features. Thus, the first state vector may include: the topological structure features of the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running state of the plurality of candidate operators. In this embodiment, by extracting features from the link, key information about the link's topological structure can be extracted, which helps improve the optimization efficiency of the reinforcement learning model.

[0050] In step 1042 above, the first state vector is input into the reinforcement learning model so that the reinforcement learning model can estimate the value of multiple candidate optimization actions in the action space, and then select and output the optimization action based on the value.

[0051] In practical applications, based on the preset action type, for each operator in the first data processing link to be optimized, the alternative optimization action corresponding to the operator can be determined. Then, the alternative optimization actions corresponding to each of the multiple operators in the first data processing link to be optimized are used as the action space corresponding to the first data processing link to be optimized.

[0052] Optionally, the multiple action types may include one or more of the following preset action types: adding an operator, deleting an operator, adjusting the order of adjacent operators, and replacing an operator. In practical applications, to avoid the uncertainty caused by order adjustment, the order adjustment of adjacent operators can be limited to the order adjustment between two adjacent operators. In some embodiments, the order adjustment of adjacent operators is limited to two adjacent operators that have no data dependency. Assume: the multiple action types include adding an operator, deleting an operator, adjusting the order of adjacent operators, and replacing an operator, and the first data processing link to be optimized is A->B->C, then the alternative optimization actions corresponding to operator A include: adding operator 1 before operator A, adding operator 2 before operator A, ..., adding operator n before operator A, deleting operator A, swapping the positions of A and B, replacing operator A with operator 1, replacing operator A with operator 2, ..., replacing operator A with operator n. Here, operator 1, operator 2, ..., operator n are multiple candidate operators stored in the operator library. The candidate optimization actions corresponding to operator B include: adding operator 1 before operator B, adding operator 2 before operator B, ..., adding operator n before operator B, deleting operator B, swapping the positions of A and B, swapping the positions of B and C, replacing operator B with operator 1, replacing operator B with operator 2, ..., replacing operator B with operator n. The candidate optimization actions corresponding to operator C include: adding operator 1 before operator C, adding operator 2 before operator C, ..., adding operator n before operator C, deleting operator B, swapping the positions of B and C, replacing operator C with operator 1, replacing operator C with operator 2, ..., replacing operator C with operator n. Where n is a positive integer, the action space corresponding to the first data processing link to be optimized includes: the candidate optimization action corresponding to operator A, the candidate optimization action corresponding to operator B, and the candidate optimization action corresponding to operator C.

[0053] In step 1044 above, the first data processing link to be optimized is optimized according to the optimization action output by the reinforcement learning model.

[0054] Using the previous example, the optimization action is: replace operator A with operator 1, then the optimized link is: 1->B->C.

[0055] In steps 1046 and 1048 above, after each optimization of the first data processing link to be optimized, it can be determined whether the current optimization process meets the termination condition. If the termination condition is not met, the optimized first data processing link to be optimized is taken as the new first data processing link to be optimized, and the process returns to step 1040. If the termination condition is met, the optimized first data processing link to be optimized is determined as the first target data processing link.

[0056] The termination conditions may include: the optimization process has converged and / or the number of actions performed by the optimization process has reached a preset action threshold. Here, "optimization process converged" means that the performance estimates of the data processing link obtained from the most recent optimizations by the optimization process tend to stabilize. The preset action threshold can be set according to actual conditions; this embodiment does not specifically limit it, for example, 5 times.

[0057] In practice, when the optimization process has converged or the number of actions performed by the optimization process has reached a preset action threshold, the optimization process ends, and the first data processing link to be optimized obtained from the last optimization is determined as the first target data processing link. When the optimization process has not converged and the number of actions performed by the optimization process has not reached the preset action threshold, the first data processing link to be optimized after this optimization is taken as the new first data processing link to be optimized, and the process returns to step 1040. In the embodiments of this specification, the reinforcement learning model optimizes the link multiple times through iteration to improve the link optimization effect and thus improve the link performance.

[0058] The following describes a method for link performance estimation. This method includes the following steps: 106. Sample the data source to be processed in the first data processing task to obtain a data subset.

[0059] 108. Run the first data processing link to be optimized to process the data subset.

[0060] 110. Based on the observed performance indicators, determine the performance estimate of the first data processing link to be optimized.

[0061] In the above 106, the data source to be processed can be sampled according to a preset sampling ratio to obtain a data subset. The sampling ratio can be set as needed, and this specification does not limit it in detail in the embodiments, for example: 1%.

[0062] Of the 108 mentioned above, the first data processing link to be optimized can be run in a real environment to process a subset of data.

[0063] In point 110 above, the observed performance metrics refer to the performance metrics obtained by observing the process of running the first data processing link to be optimized in a real environment to process a subset of data. These performance metrics may include: runtime, resource consumption, output quality, task success rate, etc.

[0064] Based on the observed performance metrics, the performance estimate of the first data processing link to be optimized when processing the data source to be processed can be inferred.

[0065] In the embodiments described in this specification, the link is run in a real environment based on small batches of data, and the performance of the link is estimated by observing the performance indicators, which has high accuracy.

[0066] Alternatively, a link performance prediction model can be trained based on the training data. In this way, by inputting the data volume of the first data processing link to be optimized and the data volume of the data source to be processed into the link performance prediction model, the performance estimate output by the link performance prediction model can be obtained.

[0067] The reinforcement learning process of the above-described reinforcement learning model will be described below. In the embodiments of this specification, the reinforcement learning process of the reinforcement learning model includes multiple rounds. Each round includes the following steps: 112. At the start of this round, obtain the second data processing link to be optimized corresponding to the second data processing task.

[0068] 114. Based on the second data processing link to be optimized, the performance estimate of the second data processing link to be optimized, and the current running status of the multiple candidate operators, the optimization action is determined using the reinforcement learning model to be trained.

[0069] 116. Perform the optimization action to optimize the second data processing link to be optimized.

[0070] 118. Optimize the reinforcement learning model to be trained based on the environmental reward after the optimized action.

[0071] 120. If the termination condition is not met in this round, the second data processing link to be optimized after this optimization will be used as the new second data processing link to be optimized, and the process will return to step 114.

[0072] 122. When the end condition is met in this round, the current round ends.

[0073] In some embodiments, the reinforcement learning model may be trained using only tasks of a specific task type, so that the trained reinforcement learning model can handle tasks of that task type well. Therefore, in the embodiments of this specification, the second data processing task processed in different rounds may be the same or may be different tasks under that specific task type.

[0074] In other embodiments, at the start of each round, a data processing task can be randomly selected from the task set as the second data processing task, and then steps 112 to 122 described above are executed. The task set may contain data processing tasks of various different task types. Optionally, there may be multiple data processing tasks of each task type in the task set. The reinforcement learning model trained using the training method provided in this embodiment is more versatile, can adapt to various known task types, and also has a certain degree of adaptability to newly emerging task types. In practical applications, any data processing task in the task set may be selected multiple times; that is, the same data processing task has a probability of being selected in multiple rounds.

[0075] In the above 112, the second data processing link to be optimized is determined based on the second task requirements of the second data processing task and the plurality of candidate operators. Specifically, based on the second task requirements of the second data processing task, multiple required second operators can be matched from the plurality of candidate operators, and a second data processing link to be optimized can be generated based on the multiple second operators. The second task requirements are used to indicate the data source to be processed and its data source parameters, data processing objectives, performance expectations, and other information.

[0076] The generation method of the second data processing link to be optimized can refer to the generation method of the first data processing link to be optimized described above, and will not be repeated in this embodiment. The second data processing link to be optimized obtained in step 112 is the link that has not been optimized using a reinforcement learning model.

[0077] In the above 114, a second state vector can be constructed based on the second data processing link to be optimized, the performance estimate of the second data processing link to be optimized, and the current running status of multiple candidate operators.

[0078] In some optional embodiments, the second state vector may include: a second data processing link to be optimized, a performance estimate of the second data processing link to be optimized, and the current running state of a plurality of candidate operators.

[0079] In some alternative implementations, feature extraction can be performed on the second data processing link to be optimized to obtain its topological features. Thus, the second state vector may include: the topological features of the second data processing link to be optimized, a performance estimate of the second data processing link to be optimized, and the current operating states of multiple candidate operators.

[0080] The action space corresponding to the reinforcement learning model includes one or more of the following preset action types: adding operators, deleting operators, adjusting the order of adjacent operators, and replacing operators. Different action spaces correspond to different second data processing links to be optimized. Based on the preset action types, for each operator in the second data processing link to be optimized, candidate optimization actions are determined. Then, the candidate optimization actions corresponding to each of the multiple operators in the second data processing link to be optimized are used as the action space corresponding to the second data processing link to be optimized. The obtained second state vector is input into the reinforcement learning model so that the reinforcement learning model can estimate the value of multiple candidate optimization actions in the action space corresponding to the second data processing link to be optimized, and then select and output the optimization action based on the value.

[0081] In step 116 above, the second data processing link to be optimized is optimized according to the optimization action output by the reinforcement learning model.

[0082] In step 118 above, the reinforcement learning model to be trained is optimized based on the environmental reward after the optimized action.

[0083] In reinforcement learning, the core objective of the optimization process is to iteratively improve the agent's decision-making strategy based on rewards from environmental feedback. Essentially, this process involves continuously updating the model's parameters through trial and error, enabling it to achieve higher cumulative rewards in the future.

[0084] Specifically, after each action is performed at each time step, an immediate reward signal and a new environmental state are received. This interaction data (state, action, reward, new state) is stored as experience. During optimization, the algorithm samples from this experience and uses temporal difference error or Monte Carlo estimation to calculate the difference between the expected reward and the actual reward of the current policy. For value function methods (such as Q-learning), this error is used to update the action value function, making it more accurate in predicting long-term returns. The entire optimization is a closed-loop incremental learning process: the model continuously refines its value judgment preferences using new experience, gradually internalizing successful action sequences into stable policies while reducing behaviors that lead to low rewards, ultimately converging to a decision model that performs well on a given task.

[0085] In steps 120 and 122 above, it is determined whether the current round meets the termination condition. The termination condition may include: the current round has converged and / or the number of actions performed in the current round has reached a preset action count threshold. Convergence in the current round means that the environmental reward after the most recent optimized actions in the current round tends to stabilize. The size of the preset action count threshold can be set according to actual needs; this embodiment does not specifically limit it.

[0086] In some embodiments, the current round ends when the current round has converged or the number of actions performed in the current round has reached a preset action threshold. If the current round has not converged and the number of actions performed in the current round has not reached the preset action threshold, the optimized second data processing link is used as the new second data processing link to be optimized, and the process returns to step 114.

[0087] In the embodiments described in this specification, the reinforcement learning model is trained through multiple rounds, continuously attempting to optimize the data processing pipeline, thereby enabling the model to optimize the pipeline. This allows the trained reinforcement learning model to be applied to a production environment to quickly build a data processing pipeline with performance requirements. The upper limit for the number of rounds can be preset; once this upper limit is reached, training is considered complete.

[0088] During the reinforcement learning phase, an Experience Replay Buffer is created to store transition samples generated by the agent's interaction with the environment. This buffer is represented as <s, a, r, s', done>, where s is the current state, a is the action performed in state s, r is the environmental reward, and s' is the next state entered after performing the action. Done is a boolean value indicating whether the current episode has ended (e.g., task completion or failure). Subsequently, batches of data are randomly sampled from the replay buffer, and the network parameters are updated using a pre-defined loss function (e.g., mean squared Bellman error loss function).

[0089] During the reinforcement learning phase, an ε-greedy exploration strategy can be adopted. For example, the initial value of ε is set to 1.0 to encourage full exploration in the initial stage; after every N (e.g., 1000) decisions, ε decreases linearly by 0.05 until it reaches a minimum value of 0.1, which is used to ensure sufficient exploration in the early stage and convergence of the strategy in the later stage.

[0090] In some optional implementations, the above method may further include: 124. Based on the performance estimate of the new second data processing link to be optimized, determine the environmental reward after the optimization action.

[0091] In the above 124, the data source to be processed of the second data processing task to be optimized can be sampled to obtain a data subset; the second data processing link to be optimized can be run to process the data subset; and the performance estimate of the second data processing link to be optimized can be determined based on the observed performance indicators.

[0092] Optionally, the amount of data from the data source to be processed in the second data processing task to be optimized and the second data processing link to be optimized can be input into a trained link performance prediction model so that the model can output a performance estimate of the second data processing link to be optimized.

[0093] In some embodiments, the aforementioned environmental rewards may include multiple rewards, including at least two of the following: a reward determined based on the estimated runtime of the new second data processing link to be optimized (hereinafter referred to as runtime reward), a reward determined based on the estimated resource consumption of the new second data processing link to be optimized (hereinafter referred to as resource consumption reward), a reward determined based on the estimated output (output result) quality of the new second data processing link to be optimized (hereinafter referred to as data quality reward), and a reward determined based on the estimated task success rate of the new second data processing link to be optimized (hereinafter referred to as stability reward). The shorter the runtime, the higher the runtime reward; the lower the resource consumption (e.g., CPU, GPU, memory), the higher the resource consumption reward; the more the output data of the link conforms to the quality standards in the task requirements, the higher the data quality reward; the higher the link execution success rate, the better the stability, and the higher the stability reward.

[0094] When there are multiple environmental rewards, the step 118 above, "optimizing the reinforcement learning model to be trained based on the environmental rewards after the optimized action," can be implemented using the following steps: 1180. Obtain weight configuration information.

[0095] 1182. Based on the weight configuration information, the multiple environmental rewards are weighted and summed to obtain the target reward.

[0096] 1184. Based on the target reward, optimize the reinforcement learning model to be trained.

[0097] In the above 1180, the weight configuration information can be entered by the user or configured automatically by the system.

[0098] In the aforementioned 1182, for example, multiple environmental rewards include: runtime rewards, resource consumption rewards, data quality rewards, and stability rewards. The environmental rewards are calculated using the following reward function formula: (1) Where T represents the normalized representation of runtime reward, C represents the normalized representation of resource efficiency reward, Q represents the normalized representation of data quality reward, S represents the normalized representation of stability reward, and a, b, c, and d are weights that can be dynamically adjusted according to actual needs (such as user requirements).

[0099] Based on the second data processing link to be optimized, the performance estimate of the second data processing link to be optimized, and the current running status of the multiple candidate operators, the optimization action is determined using the reinforcement learning model to be trained.

[0100] Figure 2 This is a flowchart illustrating the reinforcement learning method provided in the embodiments of this specification. Figure 2 As shown, the method includes the following steps: 200. At the start of this round, obtain the second data processing link to be optimized corresponding to the second data processing task.

[0101] The second data processing link to be optimized is determined based on the second task requirements of the second data processing task and multiple alternative operators.

[0102] 202. Based on the current running status of the second data processing link to be optimized, the multiple candidate operators, and the performance estimate of the second data processing link to be optimized, the optimization action is determined using the reinforcement learning model to be trained.

[0103] 204. Based on the optimization action, optimize the second data processing link to be optimized to obtain a new second data processing link to be optimized.

[0104] 206. Optimize the reinforcement learning model to be trained based on the environmental reward after the optimized action.

[0105] 208. If the termination condition is not met in this round, the second data processing link to be optimized after this optimization will be used as the new second data processing link to be optimized, and the process will return to step 202.

[0106] 210. When the end condition is met in this round, the current round ends.

[0107] The specific implementation of steps 200 to 210 can be found in the corresponding content of the above embodiments, and will not be repeated here.

[0108] In the technical solution provided in the embodiments of this specification, multiple first operators are matched from multiple candidate operators according to task requirements, and a data processing link to be optimized is generated based on the multiple first operators. Then, based on the data processing link to be optimized, its performance estimate, and the current running status of the multiple candidate operators, a trained reinforcement learning model is used to optimize the data processing link to be optimized, thereby obtaining the optimized data processing link. It can be seen that this solution, based on reinforcement learning technology, achieves intelligent optimization of the data processing link, thereby improving link performance.

[0109] In some embodiments, the reinforcement learning process of the reinforcement learning model to be trained includes multiple rounds.

[0110] In some embodiments, the step 200 above, "at the start of this round, obtain the second data processing link to be optimized corresponding to the second data processing task", can be implemented by the following steps: 2000. At the start of this round, select the second data processing task from the preset task set according to the preset selection strategy.

[0111] 2002. Obtain the second data processing link to be optimized corresponding to the second data processing task.

[0112] In the aforementioned 2000, the preset task set includes data processing tasks of various task types, and under the selection strategy, each type of data processing task has a probability of being selected.

[0113] For example, the selection strategy is a random selection strategy.

[0114] In the above 2002, the second data processing link to be optimized corresponding to the selected data processing task is obtained.

[0115] The reinforcement learning model trained using the training method provided in this embodiment is more versatile, can adapt to a variety of known task types, and can also have a certain degree of adaptability to newly emerging task types.

[0116] It should be noted that any steps in the methods provided in the embodiments of this specification that are not fully described in detail can be found in the corresponding content of the above embodiments, and will not be repeated here. Furthermore, the methods provided in the embodiments of this specification may include other parts or all of the steps in the above embodiments in addition to the steps described above; for details, please refer to the corresponding content of the above embodiments, and will not be repeated here.

[0117] Figure 3 This is a flowchart illustrating the reinforcement learning method provided in the embodiments of this specification. Figure 3 As shown, the method includes the following steps: 300. Obtain the first data processing task.

[0118] The first data processing task includes a first task requirement.

[0119] 302. Based on the first task requirements, match multiple first operators from multiple candidate operators and generate a first data processing link to be optimized based on the multiple first operators.

[0120] 304. Based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, the first data processing link to be optimized is optimized using a trained reinforcement learning model to obtain the first target data processing link.

[0121] 306. Run the first target data processing link to process the data source to be processed corresponding to the first data processing task.

[0122] The specific implementation of steps 300 to 304 above can be found in the corresponding contents of the above embodiments, and will not be repeated here.

[0123] In some embodiments of the above 306, the first target data processing link can be submitted (also referred to as deployed) to the computing engine. The computing engine starts the first data processing task according to the first target data processing link to process the data from the data source to be processed corresponding to the first data processing task and obtain the required result. In this process, the computing engine first reads data from the target data source, and then performs cleaning, calculation and other operations step by step like an assembly line. The downstream task can only be started after the upstream task is completed, and the target data source is processed to obtain the required result.

[0124] In the technical solution provided in the embodiments of this specification, multiple first operators are matched from multiple candidate operators according to task requirements, and a data processing link to be optimized is generated based on the multiple first operators. Then, based on the data processing link to be optimized, its performance estimate, and the current running status of the multiple candidate operators, a trained reinforcement learning model is used to optimize the data processing link to be optimized, thereby obtaining the optimized data processing link. It can be seen that this solution, based on reinforcement learning technology, achieves intelligent optimization of the data processing link, thereby improving link performance.

[0125] Optionally, before step 306 above, the performance of the first target data processing link can be verified. If the verification passes, step 306 is executed; if the verification fails, the first target data processing link is not run, and it can be sent to staff for manual optimization. Optionally, the first target data processing link can be run to process a subset of data from the data source to be processed. Based on the observed performance indicators, it can be determined whether the link meets the performance requirements, such as whether the link meets the requirements for running time, resource consumption, and data quality.

[0126] It should be noted that any steps in the methods provided in the embodiments of this specification that are not fully described in detail can be found in the corresponding content of the above embodiments, and will not be repeated here. Furthermore, the methods provided in the embodiments of this specification may include other parts or all of the steps in the above embodiments in addition to the steps described above; for details, please refer to the corresponding content of the above embodiments, and will not be repeated here.

[0127] The following will combine Figure 4 and Figure 5 The reinforcement learning method provided in the embodiments of this specification is illustrated with examples: Figure 4 This describes the training architecture involved in the reinforcement learning method provided in the embodiments of this specification. The architecture includes: a requirement deconstruction module 41, a link generation module 42, an operator library 43, a reinforcement learning optimization module 44, and a link execution and detection module 45.

[0128] Among them, the requirement deconstruction module 41 can help users define data processing tasks. Users can input specific requirements (corresponding to) through the structured input system. Figure 4 In S1, user input may include: data source identifier, data source type (such as database table, JSON file, CSV data, etc.); data source format, i.e., fields and field types, descriptions, lineage, etc.; and expected data processing goals, such as data cleaning standards, feature transformation, grouping methods, and quality indicators of the final output data.

[0129] The requirement deconstruction module can provide the link generation module with system-identifiable task requirements (corresponding to...). Figure 4 (S2 in the middle).

[0130] The operator library 43 provides various types of operators to meet the data processing needs of different scenarios and stages. These operators form the foundation of the data processing chain, and can be flexibly combined through the chain generation module 42 to construct a data processing chain that can effectively complete specific data processing tasks.

[0131] For example, the operators provided by operator library 43 include, but are not limited to: data cleaning operators, feature engineering operators, aggregation operators, and machine learning operators.

[0132] Data cleaning is a crucial step in the data preprocessing stage, aiming to improve data quality and usability. Data cleaning operators can include the following specific operators: 1. Redundant data cleanup operator, used to remove duplicate data, invalid fields and empty rows.

[0133] 2. Missing value handling operators are used to repair or complete missing values ​​in the data, employing various imputation methods such as mean imputation, median imputation, and Lagrange interpolation.

[0134] 3. Data format unification operator, used to convert raw data into structured or semi-structured formats from the user's perspective, to ensure data standard consistency.

[0135] Feature engineering is a core part of AI and machine learning tasks, used to generate high-quality feature inputs to improve model training and inference performance. This operator library provides a variety of feature engineering operators, such as: 1. Feature extraction operators are used to extract key information from raw data. For example, extracting n-grams in text processing and edge features in image processing.

[0136] 2. Feature standardization and normalization operators are used to scale features to a uniform scale, such as standardizing data to zero mean and unit variance, or normalizing to the range of 0 to 1, to prevent feature differences from interfering with model optimization.

[0137] 3. Feature selection operators are used to select features that have a significant impact through correlation analysis or statistical methods.

[0138] This operator library provides a variety of aggregation operators, such as: Grouping operators are used to group data, for example, based on one or more columns, and perform subsequent processing within each group.

[0139] Statistical aggregation operators are used to perform statistical processing on the field data of each group, such as summation, mean, count, maximum / minimum values, etc.

[0140] Sorting and filtering operators are used to sort data based on specific metrics (such as ascending or descending order) and filter out subsets that meet certain conditions (such as selecting the top N largest values).

[0141] The time window aggregation operator is used to perform sliding window operations on time series data, analyze data patterns within a certain time segment, and output results that meet the target requirements.

[0142] Machine learning operators are designed for higher-level tasks, and in addition to performing traditional data processing operations, they can directly support model training and inference. Furthermore, interfaces to these operators are provided on various leading AI cloud platforms. This operator library offers a variety of machine learning operators, such as: Model training operators are used to support the entire process from data input to model optimization, including the efficient training of traditional machine learning algorithms and deep learning frameworks, meeting the needs of various scenarios.

[0143] Model inference operators are used to perform data prediction and classification on trained models, supporting model loading in various formats and accelerating the inference process.

[0144] Model evaluation operators are used to evaluate the performance and accuracy of models, providing comprehensive metric analysis, such as key performance parameters like precision, recall, and error.

[0145] Link generation module 42 deconstructs the task requirements output by the requirement deconstruction module and uses the rule engine to match operators (corresponding to...) Figure 4 In S3, the initial data processing link is constructed. The link generated by this module is expressed using DAG (Directed Acyclic Graph), where each node represents an operator and each edge represents the dependency relationship between operators.

[0146] Link generation module 42 passes the initial link to reinforcement learning optimization module 44 for further optimization (corresponding to...). Figure 4 (S3). The specific optimization process will be combined below. Figure 5 Let's go into detail. The link generation module 42 can pass an initial link to the reinforcement learning optimization module 44 at the beginning of each round. Each initial link can be constructed based on the same task requirement or based on different task requirements. The task types described in the different task requirements can be the same or different.

[0147] The link execution and detection module 45 is used to take the better link output by the reinforcement learning model as the execution input through the underlying scheduling, call the operators in sequence, and allocate CPU / GPU / memory resources to ensure the effective and stable completion of the task; it is also responsible for resource detection, cost calculation, task status detection, etc.

[0148] Figure 5 This illustrates one round in the optimization process of the reinforcement learning optimization module. For example... Figure 5 As shown, a round includes the following steps: 51. Receive initial link.

[0149] Receive the initial link provided by the link generation module.

[0150] 52. Obtain the initial link performance estimate.

[0151] The initial link and a subset of data from the corresponding data source can be submitted to the link execution and detection module 45 (corresponding to...). Figure 4 In S6), the link execution and detection module 45 runs the initial link to process a subset of data and determines the performance estimate of the initial link based on the observed performance metrics.

[0152] 53. Optimize the link.

[0153] Read the running status of each operator in the operator library (corresponding to) Figure 4In step S5, a state vector is constructed based on the topological characteristics of the initial link, the performance estimate of the initial link, and the running states of multiple operators in the operator library. The state vector is then input into the reinforcement learning model to be trained, so that the reinforcement learning model outputs optimization actions to optimize the initial link.

[0154] 54. Obtain the performance estimate of the optimized link.

[0155] The optimized link and a subset of data from the corresponding data source can be submitted to the link execution and detection module 45, which will then run the optimized link to process the data subset and feed back the observed performance metrics to the reinforcement learning optimization module (corresponding to...). Figure 4 In S7 of the code, the reinforcement learning optimization module determines the performance estimate of the optimized link based on the feedback performance metrics. The link execution and detection module 45 can detect the real-time running status of operators in the operator library and report the real-time running status to the operator library (corresponding to...). Figure 4 (S8 in the middle).

[0156] 55. Calculate the reward.

[0157] The reinforcement learning optimization module can calculate the environmental reward after the optimized action based on the performance estimate of the optimized link and the reward function.

[0158] 56. Does the termination condition meet?

[0159] If the current round does not converge and the number of actions performed in the current round does not reach the preset number of actions, the optimized link will be used as the new initial link, and the process will return to step 53.

[0160] The current round ends when the current round has converged or the number of actions performed in the current round has reached the preset action threshold.

[0161] In practical applications, after the current round ends, it can be determined whether the entire training process meets the termination conditions. For example, if the number of rounds executed so far has reached the preset round number threshold or the average reward of the most recent rounds is greater than or equal to the preset threshold, then the entire training process is determined to end. If the number of rounds executed so far has not reached the preset round number threshold and the average reward of the most recent rounds is less than the preset threshold, then the entire training process is determined not to end and a new round is started.

[0162] The average reward for the most recent rounds refers to the average environmental reward after the last optimization action in each of the most recent rounds.

[0163] After the entire training process is complete, the trained reinforcement learning module can be deployed. During the inference phase: based on the user-provided task requirements, an initial data production pipeline is generated and dynamically optimized using the trained reinforcement learning model to quickly construct a superior data production pipeline. After optimization, the pipeline undergoes a performance evaluation module to verify whether it meets performance metrics and resource requirements, ultimately producing an optimized data production pipeline (DAG) for execution of data processing tasks in a real-world production environment. Unlike the training phase, the inference phase does not calculate reward values ​​or update the reinforcement learning model; instead, it directly applies the output of the reinforcement learning model.

[0164] The technical solutions provided in the embodiments of this specification aim to solve the problem of automating the design and optimization of data production links in big data platforms. By introducing an intelligent optimization method based on reinforcement learning, the system can balance indicators such as task execution cost, time, success rate and data quality, thereby effectively generating a better link, improving data processing efficiency and meeting the ever-growing needs of artificial intelligence and data analysis.

[0165] The technical solution provided in this specification, through the introduction of reinforcement learning technology, creates a dynamic optimization link scheme based on state modeling, action selection, and reward function calculation. The reinforcement learning module can automatically explore optimal link combinations and adjustment strategies within a complex state space (such as operator execution performance and resource consumption). During link optimization, the reinforcement learning algorithm continuously collects feedback information such as resource usage and performance indicators through multiple rounds of training, dynamically adjusting the link structure to gradually achieve optimal performance (such as shorter execution time and lower resource overhead). For the link optimization process, this solution defines a flexible set of action operations based on the reinforcement learning strategy, supporting dynamic link generation and real-time optimization, greatly improving the degree of freedom of actions and adapting to complex and ever-changing task requirements. This solution uses a reinforcement learning state modeling module to monitor the resource usage of each operator task in the link in real time (including CPU utilization, GPU load, memory usage, etc.) and integrates the resource consumption status into a multi-dimensional reward function. The reinforcement learning module in this solution dynamically adjusts the operator execution order based on the current operator's running state, achieving a balanced optimization of resource usage and link performance.

[0166] Furthermore, this solution implements a modularly designed data processing system, providing an automated management platform for the entire process from user input to data processing output. Moreover, through the implementation of plug-in strategy modules for link generation and optimization, it supports dynamic expansion and offers highly flexible task processing.

[0167] One embodiment of this specification also provides a link generation apparatus, comprising: The acquisition module is used to acquire the first task requirements of the first data processing task. The generation module is used to match multiple required first operators from multiple candidate operators according to the first task requirements and generate a first data processing link to be optimized based on the multiple first operators; The optimization module is used to optimize the first data processing link to be optimized based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, using a trained reinforcement learning model to obtain the first target data processing link. The first target data processing link is used to process the first data processing task.

[0168] One embodiment of this specification also provides a reinforcement learning device, comprising: The acquisition module is used to acquire the second data processing link to be optimized corresponding to the second data processing task at the beginning of this round. The second data processing link to be optimized is determined based on the second task requirements of the second data processing task and multiple alternative operators. The determination module is used to determine the optimization action based on the second data processing link to be optimized, the current running status of the multiple candidate operators, and the performance estimate of the second data processing link to be optimized, using the reinforcement learning model to be trained. The link optimization module is used to optimize the second data processing link to be optimized based on the optimization action to obtain a new second data processing link to be optimized. The model optimization module is used to optimize the reinforcement learning model to be trained based on the environmental reward after the optimization action. The decision module is used to, when the current round does not meet the termination condition, take the optimized second data processing link as the new second data processing link to be optimized, and return to execute the step of determining the optimization action based on the current running status of the second data processing link to be optimized, the multiple candidate operators, and the performance estimate of the second data processing link to be optimized, using the reinforcement learning model to be trained; the acquisition module is used to end the current round when the current round meets the termination condition.

[0169] One embodiment of this specification also provides a data processing apparatus, including: The acquisition module is used to acquire a first data processing task, wherein the first data processing task includes a first task requirement; The generation module is used to match multiple required first operators from multiple candidate operators according to the first task requirements and generate a first data processing link to be optimized based on the multiple first operators; The optimization module is used to optimize the first data processing link to be optimized based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, using a trained reinforcement learning model to obtain the first target data processing link. The running module is used to run the first target data processing link to process the data source to be processed corresponding to the first data processing task.

[0170] It should be noted that the devices provided in the above embodiments can implement the technical solutions described in the corresponding method embodiments above. The specific implementation principles and corresponding beneficial effects of the above modules or units can be found in the corresponding content of the above method embodiments, and will not be repeated here.

[0171] This specification also provides an electronic device according to one embodiment. For example... Figure 6 As shown, the electronic device includes a processor 42 and a memory 41. The memory 41 stores one or more computer programs (or instructions); the processor 42 is coupled to the memory 41 and is used for the at least one or more computer programs to implement the steps in the methods provided in the embodiments of this specification.

[0172] Furthermore, the electronic device also includes other components such as a communication component 43, a display 44, a power supply component 45, and an audio component 46. Only some components are shown here for illustrative purposes, and it is not intended that the electronic device contains only these components.

[0173] The methods in this specification can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as a computer program product. Therefore, this specification also provides a computer program product. This computer program product includes a computer program / instructions that, when executed by an electronic component such as a processor, can perform, in whole or in part, the steps or functions of the methods provided in the embodiments of this specification. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, or other programmable device.

[0174] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0175] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0176] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0177] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0178] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium.

[0179] Accordingly, embodiments of this specification also provide a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above-described method embodiments. It should be understood that each step or combination of steps in the above-described method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above-described method embodiments.

[0180] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0181] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for embodiments such as devices, electronic devices, storage media, and program products, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments.

[0182] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0183] The above are merely embodiments of this specification and are not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this application.

Claims

1. A link generation method, characterized in that, include: Obtain the primary task requirements for the first data processing task; According to the first task requirements, multiple first operators are matched from multiple candidate operators and a first data processing link to be optimized is generated based on the multiple first operators; Based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, the first data processing link to be optimized is optimized using a trained reinforcement learning model to obtain the first target data processing link. The first target data processing link is used to process the first data processing task.

2. The method according to claim 1, characterized in that, Based on the first data processing link to be optimized, its performance estimate, and the current running status of the multiple candidate operators, the first data processing link to be optimized is optimized using a trained reinforcement learning model to obtain a first target data processing link, including: Based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, a first state vector is determined. The state vector is input into the reinforcement learning model; The optimization action output by the reinforcement learning model is executed to optimize the first data processing link to be optimized; When the optimization process does not meet the termination condition, the first data processing link to be optimized after this optimization is taken as the new first data processing link to be optimized, and the process returns to the step of determining the first state vector based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running state of the multiple candidate operators. When the optimization process meets the termination conditions, the first data processing link to be optimized after this optimization is determined as the first target data processing link.

3. The method according to claim 1, characterized in that, The reinforcement learning process of the reinforcement learning model includes multiple rounds, wherein one round includes the following steps: At the start of this round, the second data processing link to be optimized corresponding to the second data processing task is obtained. The second data processing link to be optimized is determined based on the second task requirements of the second data processing task and the multiple alternative operators. Based on the second data processing link to be optimized, the performance estimate of the second data processing link to be optimized, and the current running status of the multiple candidate operators, the optimization action is determined using the reinforcement learning model to be trained. Perform the optimization action to optimize the second data processing link to be optimized; The reinforcement learning model to be trained is optimized based on the environmental reward after the optimized action. If the current round does not meet the termination condition, the second data processing link to be optimized after this round is taken as the new second data processing link to be optimized, and the process returns to the step of determining the optimization action based on the current running status of the second data processing link to be optimized, the multiple candidate operators, and the performance estimate of the second data processing link to be optimized, using the reinforcement learning model to be trained. The current turn ends when the end condition is met.

4. The method according to claim 3, characterized in that, Also includes: Based on the performance estimate of the new second data processing link to be optimized, the environmental reward after the optimization action is determined.

5. The method according to claim 3, characterized in that, The environmental rewards are multiple, including at least two of the following: a reward determined based on the runtime estimate of the new second data processing link to be optimized, a reward determined based on the resource consumption estimate of the new second data processing link to be optimized, a reward determined based on the output quality estimate of the new second data processing link to be optimized, and a reward determined based on the task success rate estimate of the new second data processing link to be optimized.

6. The method according to claim 5, characterized in that, Based on the environmental reward after the optimized action, the reinforcement learning model to be trained is optimized, including: Obtain weight configuration information; Based on the weight configuration information, the multiple environmental rewards are weighted and summed to obtain the target reward; Based on the target reward, the reinforcement learning model to be trained is optimized.

7. The method according to any one of claims 2 to 6, characterized in that, The termination conditions include: the current round has converged and / or the number of actions performed in the current round has reached a preset action count threshold.

8. The method according to any one of claims 2 to 6, characterized in that, The action space corresponding to the reinforcement learning model includes one or more of the following action types: adding operators, deleting operators, adjusting the order of adjacent operators, and replacing operators.

9. The method according to any one of claims 1 to 6, characterized in that, Also includes: The data source to be processed in the first data processing task is sampled to obtain a data subset; Run the first data processing link to be optimized to process the data subset; Based on the observed performance metrics, the performance estimate of the first data processing link to be optimized is determined.

10. The method according to any one of claims 1 to 6, characterized in that, Based on the first task requirement, multiple first operators are matched from multiple candidate operators, and a first data processing link to be optimized is generated based on the multiple first operators, including: Using a heuristic algorithm, based on the requirements of the first task, multiple first operators are matched from multiple candidate operators, and a first data processing link to be optimized is generated based on the multiple first operators.

11. The method according to any one of claims 1 to 6, characterized in that, Based on the first data processing link to be optimized, its performance estimate, and the current running status of the multiple candidate operators, the first data processing link to be optimized is optimized using a trained reinforcement learning model to obtain a first target data processing link, including: Feature extraction is performed on the first data processing link to be optimized to obtain the topological structure features of the first data processing link to be optimized; Based on the topological features, the current operating status of the multiple operators, and the performance estimate of the first data processing link to be optimized, the first data processing link to be optimized is optimized using a trained reinforcement learning model to obtain the first target data processing link.

12. A reinforcement learning method, characterized in that, include: At the start of this round, the second data processing link to be optimized corresponding to the second data processing task is obtained. The second data processing link to be optimized is determined based on the second task requirements of the second data processing task and multiple alternative operators. Based on the current running status of the second data processing link to be optimized, the multiple candidate operators, and the performance estimate of the second data processing link to be optimized, the optimization action is determined using the reinforcement learning model to be trained. Based on the optimization action, the second data processing link to be optimized is optimized to obtain a new second data processing link to be optimized; The reinforcement learning model to be trained is optimized based on the environmental reward after the optimized action. If the current round does not meet the termination condition, the second data processing link to be optimized after this round is taken as the new second data processing link to be optimized, and the process returns to the step of determining the optimization action based on the current running status of the second data processing link to be optimized, the multiple candidate operators, and the performance estimate of the second data processing link to be optimized, using the reinforcement learning model to be trained. The current turn ends when the end condition is met.

13. A data processing method, characterized in that, include: Obtain a first data processing task, wherein the first data processing task includes a first task requirement; According to the first task requirements, multiple first operators are matched from multiple candidate operators and a first data processing link to be optimized is generated based on the multiple first operators; Based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, the first data processing link to be optimized is optimized using a trained reinforcement learning model to obtain the first target data processing link. Run the first target data processing link to process the data source to be processed corresponding to the first data processing task.

14. A link generation device, characterized in that, include: The acquisition module is used to acquire the first task requirements of the first data processing task. The generation module is used to match multiple required first operators from multiple candidate operators according to the first task requirements and generate a first data processing link to be optimized based on the multiple first operators; The optimization module is used to optimize the first data processing link to be optimized based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, using a trained reinforcement learning model to obtain the first target data processing link. The first target data processing link is used to process the first data processing task.

15. A reinforcement learning device, characterized in that, include: The acquisition module is used to acquire the second data processing link to be optimized corresponding to the second data processing task at the beginning of this round. The second data processing link to be optimized is determined based on the second task requirements of the second data processing task and multiple alternative operators. The determination module is used to determine the optimization action based on the second data processing link to be optimized, the current running status of the multiple candidate operators, and the performance estimate of the second data processing link to be optimized, using the reinforcement learning model to be trained. The link optimization module is used to optimize the second data processing link to be optimized based on the optimization action to obtain a new second data processing link to be optimized. The model optimization module is used to optimize the reinforcement learning model to be trained based on the environmental reward after the optimization action. The decision module is used to take the optimized second data processing link as the new second data processing link when the current round does not meet the termination condition, and return to the step of determining the optimization action based on the second data processing link to be optimized, the current running status of the multiple candidate operators and the performance estimate of the second data processing link to be optimized, using the reinforcement learning model to be trained. The acquisition module is used to end the current round when the end condition is met.

16. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire a first data processing task, wherein the first data processing task includes a first task requirement; The generation module is used to match multiple required first operators from multiple candidate operators according to the first task requirements and generate a first data processing link to be optimized based on the multiple first operators; The optimization module is used to optimize the first data processing link to be optimized based on the first data processing link to be optimized, the performance estimate of the first data processing link to be optimized, and the current running status of the multiple candidate operators, using a trained reinforcement learning model to obtain the first target data processing link. The running module is used to run the first target data processing link to process the data source to be processed corresponding to the first data processing task.

17. An electronic device, characterized in that, include: Memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method of any one of claims 1 to 13.

18. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a computer, it can implement the method of any one of claims 1 to 13.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 13.

Images