Task processing method and device and storage medium

Abstract

The invention discloses a task processing method, and the method comprises the steps: enabling a task identifier of a data processing task to be matched with a predefined task template under the condition that the data processing task belongs to a heterogeneous data platform task, and further determining a plurality of target function nodes defined in a target template, according to the task processing method, the target function nodes and the node dependency relationship among the plurality of function nodes are combined, so that task processing can be performed based on the node dependency relationship among the target function nodes and the task processing strategies defined by the plurality of target function nodes respectively; the problem of data processing conflict or repetition during heterogeneous data platform task processing can be automatically avoided, and the processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a task processing method and apparatus, and a storage medium. Background Technology

[0002] Against the backdrop of rapid development in the field of big data processing, data platforms face the challenge of handling massive amounts of heterogeneous data. Heterogeneous data refers to data sets originating from different systems and possessing different data formats, storage media, and processing methods. This data is typically distributed across multiple platforms such as relational databases, file systems, and message queues, each with its unique functions and limitations.

[0003] Traditional data processing methods often lack effective unified scheduling and orchestration mechanisms when dealing with heterogeneous platforms. In most cases, data processing relies on manually written scripts or customized solutions, which is inefficient and prone to introducing data processing conflicts and duplication issues. Although some existing technologies (such as ETL tools and data warehouse hierarchical scheduling methods) attempt to solve these problems, these technologies still reveal limitations in practical applications, especially when dealing with highly complex and dynamically changing heterogeneous environments. Summary of the Invention

[0004] This application provides a task processing method to at least solve the problem of low task processing efficiency in related technologies.

[0005] According to one embodiment of this application, a task processing method is provided, comprising: when the data processing task to be processed belongs to a heterogeneous data platform task, determining task attribute parameters of the data processing task; querying a predefined template database based on the task attribute parameters to determine a target task template matching the data processing task; wherein the target task template defines multiple functional nodes and node dependencies between the multiple functional nodes; each of the multiple functional nodes defines a task processing strategy; and processing the data processing task based on the node dependencies and the task processing strategies defined by the multiple functional nodes respectively.

[0006] In an exemplary embodiment, the template database includes: multiple task templates; querying a predefined template database based on the task attribute parameters to determine a target task template matching the data processing task includes: encoding the task attribute parameters to determine a first encoded representation of the task attribute parameters; extracting key elements from the multiple task templates to determine key content corresponding to each of the multiple task templates; encoding the key content corresponding to each of the multiple task templates to determine a second encoded representation corresponding to each of the multiple key contents; calculating the similarity between the first encoded representation and each of the multiple second encoded representations; determining a target encoded representation from the multiple second encoded representations that satisfies a preset similarity condition, and determining the task template corresponding to the target encoded representation as the target task template.

[0007] In one exemplary embodiment, key element extraction is performed on the plurality of task templates to determine the key content corresponding to each of the plurality of task templates, including: extracting key elements from the plurality of task templates based on a predefined element extraction dimension to obtain description parameters of the plurality of task templates under the element extraction dimension; constructing key performance matrices corresponding to the plurality of task templates based on the description parameters extracted under the element extraction dimension; and converting the format of the key performance matrices corresponding to the plurality of task templates to obtain the key content corresponding to the plurality of task templates.

[0008] In an exemplary embodiment, before querying a predefined template database based on the task attribute parameters to determine a target task template matching the data processing task, the method further includes: determining a first functional node from the plurality of functional nodes; determining a target functional node satisfying a dependency condition from the second functional nodes based on a first processing time of the first functional node and a second processing time of a second functional node other than the first functional node; wherein the processing of the target functional node depends on the processing result of the first functional node; re-determining a new first functional node from the plurality of functional nodes, and returning to the step of determining a target functional node satisfying a dependency condition from the second functional nodes based on the first processing time of the first functional node and the second processing time of the second functional node other than the first functional node, until a preset iteration stop condition is reached, thereby obtaining the node dependency relationship between the plurality of functional nodes.

[0009] In an exemplary embodiment, determining a target functional node that satisfies the dependency relationship condition from the second functional nodes based on the first processing time of the first functional node and the second processing time of the second functional nodes other than the first functional node includes: calculating a plurality of time differences between the first processing time of the first functional node and the second processing time of the second functional node; selecting a target time difference that satisfies the time difference condition from the plurality of time differences; and determining the second functional node corresponding to the target time difference as the target functional node of the first functional node.

[0010] In an exemplary embodiment, the data processing task is processed based on the node dependencies and the task processing strategies defined by the plurality of functional nodes, including: selecting a third functional node to be executed from the plurality of functional nodes based on the node dependencies between the plurality of functional nodes; and performing data processing on the data processing task based on the task processing strategy defined by the third functional node to be executed.

[0011] In an exemplary embodiment, when the data processing task to be processed belongs to a heterogeneous data platform task, before determining the task attribute parameters of the data processing task, the method further includes: when the data processing task to be processed belongs to a heterogeneous data platform task, extracting key elements from the data processing task to determine the key content of the data processing task; when the statement length of the key content is greater than a set statement length threshold, filtering the key content of the data processing task to obtain filtered key content; and determining the task attribute parameters of the data processing task based on the filtered key content.

[0012] According to another embodiment of this application, a task processing apparatus is also provided, comprising: a first determining module, configured to determine task attribute parameters of the data processing task when the data processing task to be processed belongs to a heterogeneous data platform task; a second determining module, configured to query a predefined template database based on the task attribute parameters to determine a target task template matching the data processing task; wherein the target task template defines multiple functional nodes and node dependencies between the multiple functional nodes; each of the multiple functional nodes defines a task processing strategy; and a processing module, configured to process the data processing task based on the node dependencies and the task processing strategies defined by the multiple functional nodes respectively.

[0013] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described method at runtime.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described method through the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0016] In this embodiment of the application, when the data processing task belongs to the heterogeneous data platform task, the task identifier of the data processing task is matched with the predefined task template, thereby determining the multiple target functional nodes defined in the target template, as well as the node dependencies between the multiple functional nodes. This allows the task to be processed based on the node dependencies between the target functional nodes and the task processing strategies defined by the multiple target functional nodes. This task processing method can automatically avoid data processing conflicts or duplication problems when processing heterogeneous data platform tasks, thereby improving processing efficiency. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal for a task processing method according to an embodiment of this application.

[0019] Figure 2 This is a flowchart of a task processing method according to an embodiment of this application;

[0020] Figure 3 This is a structural block diagram of a task processing apparatus according to an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus; "a plurality" means two or more.

[0023] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device, cloud platform, independent physical server, or software platform, wherein the aforementioned software platform runs through one or more servers. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a task processing method according to an embodiment of this application. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1 (Only one is shown in the image) A processor 102 and a memory 104 for storing data. In one exemplary embodiment, the computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. The processor 102 may include, but is not limited to, a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA). Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0024] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0025] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0026] This embodiment provides a task processing method, applied to the aforementioned computer device. Figure 2 This is a flowchart of a task processing method according to an embodiment of this application, which includes the following steps:

[0027] Step S202: If the data processing task to be processed belongs to a heterogeneous data platform task, determine the task attribute parameters of the data processing task.

[0028] It should be noted that data processing tasks typically need to be performed across multiple different data platforms, including but not limited to SQL databases (such as Oracle and MySQL), NoSQL databases (such as MongoDB and Cassandra), Hadoop clusters, Spark environments, and message queues (such as Kafka and RabbitMQ). Each of these platforms has its own unique advantages and data processing methods.

[0029] Task attribute parameters are key characteristics or configuration information that describe and define a specific data processing task. They are crucial for task execution, scheduling, monitoring, and optimization. For example, task attribute parameters may include the type of data processing task, the source and target platform types, data format, execution frequency, expected processing time, and any specific business rules and requirements.

[0030] Step S204: Query the predefined template database based on task attribute parameters to determine the target task template that matches the data processing task; wherein, the target task template defines multiple functional nodes and the node dependencies between the multiple functional nodes; each of the multiple functional nodes defines a task processing strategy.

[0031] In an exemplary embodiment, the template database includes: multiple task templates; querying the predefined template database based on task attribute parameters to determine a target task template matching the data processing task includes: encoding the task attribute parameters to determine a first encoded representation of the task attribute parameters, and extracting key elements from the multiple task templates to determine key content corresponding to each of the multiple task templates; encoding the key content corresponding to each of the multiple task templates to determine a second encoded representation corresponding to each of the multiple key contents; calculating the similarity between the first encoded representation and each of the multiple second encoded representations; determining the target encoded representation corresponding to the similarity that satisfies a preset similarity condition from the multiple second encoded representations, and determining the task template corresponding to the target encoded representation as the target task template.

[0032] Understandably, the task attribute parameters of the data processing task can first be encoded to obtain a digital first coded representation. The encoding process can involve converting the task attribute parameters into a numerical vector, where each dimension represents a different level or type of a parameter. For example, suppose one dimension of the encoded vector represents the data processing type (e.g., TRS, ULD, SYN), another dimension represents the source platform type (e.g., SQL, NoSQL), yet another dimension represents the target platform type, and so on. In this way, the complex attributes of the task can be transformed into a compact numerical representation, facilitating subsequent automated processing.

[0033] Furthermore, key elements are extracted from multiple task templates in the predefined template database to identify the core functions and applicable scope of each template. This process is also coded, converting the key information of each template into a secondary coded representation. This encoding can be similar to the encoding method of task attribute parameters to ensure that the two can be directly compared.

[0034] It should be noted that key elements may include the task type of the task template, supported source / target platforms, data format, processing strategy, and expected resource consumption and execution time. For each task template, these elements are extracted and encoded into a numerical vector. The dimension of this vector is consistent with the dimension of the task attribute parameter vector to facilitate similarity calculation.

[0035] Specifically, the similarity between the first encoded representation (task attribute parameter encoding) and the second encoded representation (task template encoding) is calculated. This can be achieved through various mathematical methods, such as Euclidean distance, Manhattan distance, and cosine similarity. Cosine similarity is a commonly used method; it measures the similarity between two vectors by calculating the cosine of the angle between them. The closer the value is to 1, the more similar the two vectors are. Assuming the first encoded representation of the task attribute parameters is vector A, and the second encoded representation of the task templates is vectors B1, B2, and Bn (where n is the number of templates), a set of similarity values ​​can be obtained by calculating the cosine similarity between A and each B. Finally, from the calculated similarities, the task template corresponding to the highest similarity that meets the preset similarity criteria is selected. The preset similarity criteria can be set according to specific needs, such as selecting a task template with a cosine similarity greater than 0.8. Once the highest similarity is determined and meets the preset criteria, the target task template that best matches the current data processing task attributes is considered to have been found. Using this template as the basis for subsequent task processing can significantly reduce the time spent on manual configuration and debugging, and improve the accuracy and efficiency of task processing.

[0036] In the above embodiments, abstract business requirements and task configuration problems are transformed into mathematical problems through parameter encoding and similarity calculation, thereby achieving automated and intelligent template matching. This method is particularly suitable for scenarios involving massive heterogeneous data processing tasks, and can greatly improve the management and execution efficiency of data processing tasks.

[0037] In an exemplary embodiment, key element extraction is performed on multiple task templates to determine the key content corresponding to each task template. This includes: extracting key elements from multiple task templates based on predefined element extraction dimensions to obtain description parameters for each task template under the element extraction dimensions; constructing key performance matrices corresponding to each task template based on the description parameters extracted under the element extraction dimensions; and converting the format of the key performance matrices corresponding to each task template to obtain the key content corresponding to each task template.

[0038] Among them, the feature extraction dimensions refer to the key attributes or features considered when evaluating and comparing task templates. These dimensions should cover the main aspects of the data processing task, including but not limited to: data source type, data target type, data format, data processing type (such as transformation, cleaning, aggregation), execution frequency, expected processing time, resource requirements (CPU, memory, storage), dependencies (prerequisite tasks, data dependencies), anomaly handling strategies, and data security and privacy protection requirements.

[0039] In some embodiments, for each predefined task template, relevant descriptive parameters are extracted based on the aforementioned feature extraction dimensions. For example, for a data transformation task template, the descriptive parameters may include: data source type (SQL database, NoSQL database, file system, etc.), data format (JSON, XML, CSV, etc.), processing rules (field mapping, data type conversion, etc.), and execution frequency (daily, weekly). For a data unloading task template, the descriptive parameters may include: target system type (external database, cloud storage, FTP server, etc.), file format (such as compression format, delimiter type), and whether data encryption and authentication mechanisms are required. Constructing a key performance matrix means creating a matrix for each task template, where rows represent feature extraction dimensions and columns represent specific descriptive parameters or values.

[0040] Furthermore, the format conversion of the key performance matrices is for further encoding of these matrices to enable computer-processable matching algorithms. Format conversion typically involves converting the descriptive parameters into a numerical vector, also known as a second encoded representation, which allows mathematical operations (such as vector distance calculation) to be used to evaluate the similarity between different templates.

[0041] In the above embodiments, a suitable task template can be automatically found for each data processing task, thereby simplifying the task definition and configuration process and improving the flexibility and efficiency of the overall data processing system.

[0042] Step S206: Based on node dependencies and the task processing strategies defined by multiple functional nodes, the data processing task is processed.

[0043] In steps S202-S206 above, when the data processing task belongs to a heterogeneous data platform task, the task identifier of the data processing task is matched with a predefined task template. This allows the determination of multiple target functional nodes defined in the target template, as well as the node dependencies between these functional nodes. This enables task processing to be carried out based on the node dependencies between the target functional nodes and the task processing strategies defined for each target functional node. This task processing method can automatically avoid data processing conflicts or duplication problems when processing heterogeneous data platform tasks, thereby improving processing efficiency.

[0044] In an exemplary embodiment, before querying a predefined template database based on task attribute parameters to determine the target task template matching the data processing task, the method further includes: obtaining scheduling version files corresponding to multiple scheduling platforms respectively; each scheduling version file defines the job configuration information of the scheduling platform; analyzing the multiple scheduling version files to determine the key configuration information in the multiple scheduling version files; and generating multiple task templates for multiple scheduling platforms based on the key configuration information.

[0045] The scheduling platform is used to manage and coordinate the execution of tasks in a computer system, and is particularly crucial in large-scale data processing, business process automation, and multi-system integration environments. Its core responsibilities include arranging the execution order of data processing tasks according to predefined rules and priorities, managing dependencies between data processing tasks, ensuring that data processing tasks start and complete on time and as needed, monitoring task execution status, and handling fault recovery. The scheduling platform can be applied to various scenarios, including but not limited to ETL (Extract, Transform, Load) processes in data warehouses, big data processing (such as Hadoop MapReduce or Apache Spark jobs), distributed computing tasks, and inter-microservice calls.

[0046] The scheduling platform is responsible for configuring and executing data processing tasks on the data platform. It not only specifies when and where these tasks are executed (scheduled, periodic, or triggered by specific conditions), but also defines their execution logic and data flow, ensuring the accuracy and efficiency of data processing. Through intelligent algorithms, the scheduling platform optimizes resource allocation, coordinates multiple data processing tasks on the data platform, avoids resource conflicts, and ensures that each task receives the optimal resources required for execution.

[0047] In the field of big data processing, data platforms are often heterogeneous, including various databases, data warehouses, and data analysis tools. A scheduling platform can uniformly manage tasks on these heterogeneous data platforms, enabling collaborative operations and data flow between multiple platforms. Simply put, a data platform is the physical carrier of data processing, responsible for data storage, processing, and analysis; while a scheduling platform is the logical controller of data processing, ensuring the efficient and orderly execution of the data processing flow through task scheduling and resource coordination. The organic combination of these two forms the core component of a modern data processing architecture, jointly driving data-driven business decision-making and improved operational efficiency.

[0048] Understandably, scheduling version files can be collected from multiple scheduling platforms, typically in XML, JSON, or other parsable formats. Each scheduling platform may have its own specific job configuration syntax and rules. The collected scheduling version files are then analyzed in depth to extract key configuration information. This includes job name, execution time, frequency, dependencies, resource allocation, execution path, and parameter settings. Based on this key configuration information, task templates for multiple scheduling platforms are generated. Each template defines a set of common job configuration attributes, as well as the specific representation of these attributes on different platforms.

[0049] In the above embodiments, not only can task templates conforming to the specifications of different scheduling platforms be generated, but the templates can also be continuously optimized through intelligent analysis and user feedback to ensure that they are both universal and specific, meeting the needs of cross-platform job scheduling. At the same time, the job configuration process is simplified, reducing the risk of configuration errors. This method not only improves the efficiency of job scheduling but also promotes the standardization and automation of scheduling jobs.

[0050] In an exemplary embodiment, generating multiple task templates for the multiple scheduling platforms based on the key configuration information includes: for any one of the scheduling platforms, obtaining configuration requirement information and a script file from a target object; inputting the key configuration information, the configuration requirement information, and the script file into a pre-trained template generation model; generating an initial task template based on the key configuration information and the configuration requirement information; determining the dependencies between multiple functional nodes based on the script file; and processing the initial task template according to the dependencies between the multiple functional nodes to obtain a task template for any one of the scheduling platforms.

[0051] The target object can provide specific configuration information for its job requirements through an interface or command line, such as the job name, execution frequency, dependent jobs, data source and destination, and specific job processing logic. Correspondingly, the target object can also upload job-related script files, such as SQL scripts, Python scripts, and Shell scripts, which contain the core logic for data processing.

[0052] Using a pre-trained template generation model, the model is capable of understanding and generating job templates for the scheduling platform. The training dataset includes successful job configuration instances from multiple scheduling platforms, along with their corresponding script files and configuration information. The template generation model takes the key configuration information and configuration requirements provided by the user, as well as the uploaded script files, as input. Through natural language processing and code analysis techniques, it understands all the requirements and logic of the job.

[0053] The template generation model generates an initial task template based on the input information and its understanding of the scheduling platform's job configuration. This template may not yet contain all the details, especially job dependencies, but it lays the foundation for the basic structure and most of the configuration parameters.

[0054] The template generation model further performs in-depth analysis of the script file, identifying upstream and downstream data tables, files, and other resources involved in the job, as well as the data flow and logical dependencies between them. Based on the script analysis results, the job dependencies can be determined, including direct dependencies (such as tables referenced in SQL scripts) and indirect dependencies (such as tasks connected via data flow). Based on the identified dependencies, the template generation model configures the job dependencies in the initial task template. This involves correctly setting the execution order, waiting conditions, and data transfer mechanisms of upstream and downstream jobs within the template.

[0055] The template generation model integrates job dependency configuration information into the initial task template, populates or updates the fields and attributes related to dependencies in the template, and generates the final, complete task template. The generated final task template will be adjusted according to the scheduling platform's specifications to ensure that all configuration information, syntax format, and interface calls comply with the requirements of the scheduling platform so that it can be directly imported and executed.

[0056] In the above embodiments, the template generation model can not only automatically generate task templates for the scheduling platform based on the configuration requirements of the target object and the script file, but also intelligently identify and configure job dependencies, which greatly simplifies the complexity of job configuration and improves the efficiency and accuracy of job configuration.

[0057] In an exemplary embodiment, before querying a predefined template database based on task attribute parameters to determine a target task template matching the data processing task, the task processing method further includes: determining a first functional node from multiple functional nodes; determining a target functional node satisfying a dependency condition from the second functional nodes based on a first processing time of the first functional node and a second processing time of a second functional node other than the first functional node; wherein the processing of the target functional node depends on the processing result of the first functional node; re-determining a new first functional node from multiple functional nodes, and returning to the step of determining a target functional node satisfying a dependency condition from the second functional nodes based on the first processing time of the first functional node and the second processing time of the second functional node other than the first functional node, until a preset iteration stop condition is reached, thereby obtaining the node dependency relationship between multiple functional nodes.

[0058] The first functional node is the starting point of the entire data processing task. It may directly read data from the data source and perform preliminary data loading or preprocessing operations. Specifically, the selection of the first functional node can be determined by analyzing the data source information and preliminary processing requirements in the task attribute parameters. For example, if the task attribute parameters indicate that the data needs to be loaded from an SQL database and preliminarily cleaned, then the node for SQL data loading and cleaning functions can be selected as the first functional node.

[0059] It's important to note that each functional node has its predetermined processing time. This time can be statically defined (e.g., a fixed time each day) or dynamically determined (e.g., when the previous task is completed). The selection of the target functional node depends on whether its processing depends on the processing result of the first functional node. For example, a data transformation node depends on the output of the data cleaning node; therefore, the transformation node can only begin processing after the data cleaning node has finished. By examining all second functional nodes (all other functional nodes excluding the first functional node), the nodes whose processing time follows the first functional node and whose processing depends on the processing result of the first functional node are identified as the target functional nodes. This process of determining a new first functional node from multiple functional nodes and checking the dependencies of other nodes is repeated until the dependencies of all nodes are determined or a preset iteration stopping condition is met.

[0060] In some embodiments, the preset iteration stopping condition is typically used to avoid infinite loops or to ensure that the process covers all necessary nodes. The iteration stopping condition may include all functional nodes being assigned a dependency, the complexity limit of a dependency network being reached, or the dependency network of all nodes forming a Directed Acyclic Graph (DAG). As the iteration proceeds, a dependency network between functional nodes is gradually built. This network should be a directed acyclic graph, where each node represents a functional unit, and directed edges represent dependencies between nodes.

[0061] In the above embodiments, the dependencies between functional nodes in data processing tasks can be systematically determined and optimized, ensuring that the task execution flow conforms to the business logic of data processing while maximizing the utilization of system resources and improving processing efficiency. This process is particularly suitable for handling large-scale, heterogeneous data processing tasks, providing a solid foundation for building efficient and reliable data processing pipelines.

[0062] In an exemplary embodiment, determining a target functional node that satisfies the dependency relationship condition from the second functional nodes based on the first processing time of the first functional node and the second processing time of the second functional nodes other than the first functional node includes: calculating the difference between the first processing time of the first functional node and the second processing time of the second functional node to obtain multiple time differences; selecting a target time difference that satisfies the time difference condition from the multiple time differences; and determining the second functional node corresponding to the target time difference as the target functional node of the first functional node.

[0063] Understandably, the first step is to clearly define the scheduled processing time of the first functional node. This can be given in the definition of the data processing task or determined based on information such as the availability of the data source and the data loading cycle in the task attribute parameters. The first processing time of the first functional node can serve as the baseline for the processing time scheduling of subsequent nodes, providing an important time reference point for determining the execution time of other nodes in the entire task flow.

[0064] For each second functional node other than the first functional node, its respective processing time can be determined. These times can be specified when the data processing task is defined, or they can be calculated based on specific node attributes (such as data availability and processing frequency). The difference between the second processing time of the second functional node and the first processing time of the first functional node is calculated, resulting in a series of time differences. These time differences reflect the processing time interval of each second functional node relative to the first functional node.

[0065] It should be noted that, based on the business logic and execution efficiency requirements of the data processing task, one or more time difference conditions should be defined. For example, it might be necessary to ensure that the data transformation function node (second function node) can only begin processing after the data cleaning function node (first function node) has completed its processing. In this case, the time difference condition can be set to a positive number greater than a certain time threshold. From the calculated time differences, the time differences that meet the predetermined time difference conditions are selected. This process may involve comparing and selecting multiple time differences to find the difference that best meets the business logic and execution efficiency requirements.

[0066] The second functional node corresponding to the target time difference is identified as the target functional node of the first functional node. This means that the processing of this functional node depends on the processing result of the first functional node and satisfies the pre-defined dependency conditions in time. Repeat the above steps, using the target functional node as the new first functional node, and recalculate its processing time difference with other functional nodes (now called the new second functional nodes) until the dependencies of all functional nodes are clearly defined. This continues until all functional nodes are correctly assigned dependencies, or the entire dependency network forms an acyclic graph, ensuring no circular dependencies.

[0067] The above embodiments not only ensure that the dependencies between functional nodes in the data processing task meet business logic requirements, but also enable optimization based on time differences, thereby improving the efficiency and reliability of the entire data processing flow. This time difference-based dependency determination method is particularly suitable for handling large-scale, heterogeneous data processing tasks, providing strong technical support for building efficient and reliable data processing pipelines.

[0068] In an exemplary embodiment, the data processing task is processed based on node dependencies and task processing strategies defined by multiple functional nodes, including: selecting a third functional node to be executed from multiple functional nodes based on node dependencies; and performing data processing on the data processing task based on the task processing strategy defined by the third functional node to be executed.

[0069] Specifically, the base can create an empty execution queue to track which functional nodes are ready to be executed. Initially, the queue may contain the first functional node without any predecessor node dependencies. It analyzes the dependencies between multiple functional nodes, which could be a directed acyclic graph (DAG). Each node depends on the results of one or more predecessor nodes. Dependencies have already been determined in the earlier planning phase; here, an existing dependency network is used. Based on the node states and dependencies in the current execution queue, one or more third functional nodes that can be executed immediately are selected. The predecessor nodes of these nodes have all completed, and their own execution conditions are met. The selected third functional nodes are added to the execution plan, ensuring data is processed in the correct order. Simultaneously, the execution queue is updated, removing nodes already selected for execution and adding new, ready nodes.

[0070] In some embodiments, each functional node has a predefined task processing strategy, which may include data processing methods, algorithms used, resource requirements (such as computing power, memory, I / O), error handling mechanisms, and details of interaction with specific data sources or target systems. Based on the resource requirements and task processing strategy of the third functional node, a corresponding execution environment is configured. For example, if the node's task strategy requires running deep learning algorithms on a GPU, then the execution environment should include appropriate GPU resources. Necessary data is obtained from its predecessor node based on the node's input dependencies. If the data requires preprocessing (such as format conversion or data cleaning), the corresponding preprocessing steps are executed according to the task strategy. The core logic of the data processing task is executed according to the predefined task processing strategy. This may involve operations such as data transformation, algorithm application, and calculation of statistical indicators. After data processing is complete, data quality checks are performed according to the data validation rules in the task strategy. For example, the integrity and consistency of the data are checked to ensure that the processed data meets expectations. The processed data is saved to a specified location or database according to the requirements of the task strategy. Furthermore, if the result of this node is the input to a subsequent node, the result also needs to be passed to the next functional node to be executed. Finally, the execution status, time, and any abnormal situations of the third functional node are recorded.

[0071] In the above embodiments, the data processing task process based on node dependency and task processing strategy can ensure the orderliness, efficiency and quality of data processing, and is suitable for processing various complex, cross-platform data processing tasks.

[0072] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0073] This application also provides a structural block diagram of a task processing device in its embodiments. Figure 3 This is a structural block diagram of a task processing apparatus according to an embodiment of this application; as shown below. Figure 3 As shown, it includes:

[0074] The first determining module 32 is used to determine the task attribute parameters of the data processing task when the data processing task to be processed belongs to a heterogeneous data platform task.

[0075] The second determining module 34 is used to query a predefined template database based on the task attribute parameters to determine a target task template that matches the data processing task; wherein, the target task template defines multiple functional nodes and node dependencies between the multiple functional nodes; each of the multiple functional nodes defines a task processing strategy;

[0076] The processing module 36 is used to process the data processing task based on the node dependency relationship and the task processing strategies defined by the multiple functional nodes respectively.

[0077] Using the aforementioned device, when the data processing task belongs to a heterogeneous data platform task, the task identifier of the data processing task is matched with a predefined task template. This allows the determination of multiple target functional nodes defined in the target template, as well as the node dependencies between these functional nodes. This enables task processing to be carried out based on the node dependencies between the target functional nodes and the task processing strategies defined for each target functional node. This task processing method can automatically avoid data processing conflicts or duplication when processing heterogeneous data platform tasks, thereby improving processing efficiency.

[0078] In one exemplary embodiment, the apparatus further includes a template generation module; the template generation module is configured to obtain scheduling version files corresponding to multiple scheduling platforms respectively; each of the scheduling version files defines job configuration information of the scheduling platform; analyze the multiple scheduling version files to determine key configuration information in the multiple scheduling version files; and generate multiple task templates for the multiple scheduling platforms based on the key configuration information.

[0079] In an exemplary embodiment, the template generation module is further configured to: acquire configuration requirement information and script file from a target object for any of the scheduling platforms; input the key configuration information, the configuration requirement information, and the script file into a pre-trained template generation model; generate an initial task template based on the key configuration information and the configuration requirement information; determine the dependencies between multiple functional nodes based on the script file; and process the initial task template according to the dependencies between the multiple functional nodes to obtain a task template for any of the scheduling platforms.

[0080] In an exemplary embodiment, the template database includes: a plurality of task templates; a second determining module 34, further configured to encode the task attribute parameters to determine a first encoded representation of the task attribute parameters, and to extract key elements from the plurality of task templates to determine key content corresponding to each of the plurality of task templates; to encode the key content corresponding to each of the plurality of task templates to determine a second encoded representation corresponding to each of the plurality of key content; to calculate the similarity between the first encoded representation and each of the plurality of second encoded representations; to determine a target encoded representation from the plurality of second encoded representations that satisfies a preset similarity condition, and to determine the task template corresponding to the target encoded representation as the target task template.

[0081] In an exemplary embodiment, the second determining module 34 is further configured to extract key elements from the plurality of task templates based on a predefined element extraction dimension to obtain description parameters of the plurality of task templates under the element extraction dimension; construct key performance matrices corresponding to the plurality of task templates based on the description parameters extracted under the element extraction dimension; and perform format conversion on the key performance matrices corresponding to the plurality of task templates to obtain key content corresponding to the plurality of task templates.

[0082] In an exemplary embodiment, the apparatus further includes an analysis module; the analysis module is configured to: determine a first functional node from the plurality of functional nodes; determine a target functional node satisfying a dependency condition from the second functional nodes based on a first processing time of the first functional node and a second processing time of a second functional node other than the first functional node; wherein the processing of the target functional node depends on the processing result of the first functional node; re-determine a new first functional node from the plurality of functional nodes, and return to the step of determining a target functional node satisfying a dependency condition from the second functional nodes based on the first processing time of the first functional node and the second processing time of the second functional node other than the first functional node, until a preset iteration stop condition is reached, thereby obtaining the node dependency relationship between the plurality of functional nodes.

[0083] In an exemplary embodiment, the analysis module is further configured to calculate the difference between the first processing time of the first functional node and the second processing time of the second functional node to obtain a plurality of time differences; select a target time difference that satisfies the time difference condition from the plurality of time differences; and determine the second functional node corresponding to the target time difference as the target functional node of the first functional node.

[0084] In an exemplary embodiment, the processing module 36 is further configured to process the data processing task based on the node dependency relationship and the task processing strategies defined by the plurality of functional nodes respectively, including: selecting a third functional node to be executed from the plurality of functional nodes based on the node dependency relationship between the plurality of functional nodes; and performing data processing on the data processing task based on the task processing strategy defined by the third functional node to be executed.

[0085] In one exemplary embodiment, the apparatus further includes an extraction module; the extraction module is configured to, when the data processing task to be processed belongs to a heterogeneous data platform task, extract key elements of the data processing task to determine the key content of the data processing task; when the sentence length of the key content is greater than a set sentence length threshold, filter the key content of the data processing task to obtain filtered key content; and determine the task attribute parameters of the data processing task based on the filtered key content.

[0086] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0087] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0088] S1, when the data processing task to be processed belongs to a heterogeneous data platform task, determine the task attribute parameters of the data processing task;

[0089] S2, based on the task attribute parameters, query a predefined template database to determine a target task template that matches the data processing task; wherein, the target task template defines multiple functional nodes and the node dependencies between the multiple functional nodes; each of the multiple functional nodes defines a task processing strategy;

[0090] S3. Based on the node dependencies and the task processing strategies defined by the multiple functional nodes, the data processing task is processed.

[0091] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0092] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0093] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0094] S1, when the data processing task to be processed belongs to a heterogeneous data platform task, determine the task attribute parameters of the data processing task;

[0095] S2, based on the task attribute parameters, query a predefined template database to determine a target task template that matches the data processing task; wherein, the target task template defines multiple functional nodes and the node dependencies between the multiple functional nodes; each of the multiple functional nodes defines a task processing strategy;

[0096] S3. Based on the node dependencies and the task processing strategies defined by the multiple functional nodes, the data processing task is processed.

[0097] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0098] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0099] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0100] Embodiments of this application also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0101] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0102] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0103] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing a task, characterized in that, include: When the data processing task to be processed belongs to a heterogeneous data platform task, determine the task attribute parameters of the data processing task; Based on the task attribute parameters, a query is performed in a predefined template database to determine a target task template that matches the data processing task; wherein, the target task template defines multiple functional nodes and the node dependencies between the multiple functional nodes; each of the multiple functional nodes defines a task processing strategy; Based on the node dependencies and the task processing strategies defined by the multiple functional nodes, the data processing task is processed.

2. The method according to claim 1, characterized in that, Before querying a predefined template database based on the task attribute parameters to determine a target task template matching the data processing task, the method further includes: Obtain scheduling version files corresponding to multiple scheduling platforms; each of the aforementioned scheduling version files defines the job configuration information of the scheduling platform. Analyze multiple scheduling version files to determine the key configuration information in the multiple scheduling version files; Based on the key configuration information, multiple task templates are generated for the multiple scheduling platforms.

3. The method according to claim 2, characterized in that, The generation of multiple task templates for the multiple scheduling platforms based on the key configuration information includes: For any of the aforementioned scheduling platforms, obtain the configuration requirement information and script files from the target object; The key configuration information, the configuration requirement information, and the script file are input into a pre-trained template generation model to generate an initial task template based on the key configuration information and the configuration requirement information, and to determine the dependencies between multiple functional nodes based on the script file. The initial task template is processed according to the dependencies between the multiple functional nodes to obtain a task template for any of the scheduling platforms.

4. The method according to claim 1, characterized in that, The template database includes: multiple task templates; querying the predefined template database based on the task attribute parameters to determine the target task template matching the data processing task, including: The task attribute parameters are encoded to determine a first encoded representation of the task attribute parameters, and key elements are extracted from the multiple task templates to determine the key content corresponding to each of the multiple task templates. Encode the key content corresponding to the multiple task templates respectively, and determine the second encoding representation corresponding to the multiple key contents respectively; Calculate the similarity between the first encoding representation and each of the plurality of second encoding representations; From the plurality of second encoding representations, a target encoding representation corresponding to the similarity that satisfies the preset similarity conditions is determined, and the task template corresponding to the target encoding representation is determined as the target task template.

5. The method according to claim 4, characterized in that, Key elements are extracted from the multiple task templates to determine the key content corresponding to each task template, including: Based on a predefined feature extraction dimension, key features are extracted from the multiple task templates to obtain the description parameters of the multiple task templates under the feature extraction dimension. Based on the descriptive parameters extracted under the aforementioned feature extraction dimension, construct the key performance matrices corresponding to the multiple task templates respectively; The key performance matrices corresponding to the multiple task templates are converted into their respective formats to obtain the key content corresponding to the multiple task templates.

6. The method according to claim 1, characterized in that, Before querying a predefined template database based on the task attribute parameters to determine a target task template matching the data processing task, the method further includes: The first functional node is determined from the plurality of functional nodes; Based on the first processing time of the first functional node and the second processing time of the second functional nodes other than the first functional node among the plurality of functional nodes, a target functional node that satisfies the dependency relationship condition is determined from the second functional nodes; wherein, the processing of the target functional node depends on the processing result of the first functional node. The process involves re-determining a new first functional node from the plurality of functional nodes, returning a first processing time based on the first functional node, and a second processing time based on a second functional node other than the first functional node from the plurality of functional nodes. The process continues until a preset iteration stop condition is met, thereby obtaining the node dependency relationship between the plurality of functional nodes.

7. The method according to claim 6, characterized in that, Based on the first processing time of the first functional node and the second processing time of the second functional nodes other than the first functional node among the plurality of functional nodes, the target functional node that satisfies the dependency condition is determined from the second functional node, including: Based on the first processing time of the first functional node, the difference between the first processing time of the second functional node and the second processing time of the second functional node are calculated to obtain multiple time differences; Select a target time difference that meets the time difference condition from the plurality of time difference values; The second functional node corresponding to the target time difference is determined as the target functional node of the first functional node.

8. A task processing apparatus, characterized in that, The device includes: The first determining module is used to determine the task attribute parameters of the data processing task when the data processing task to be processed belongs to a heterogeneous data platform task. The second determining module is used to query a predefined template database based on the task attribute parameters to determine a target task template that matches the data processing task; wherein, the target task template defines multiple functional nodes and node dependencies between the multiple functional nodes; each of the multiple functional nodes defines a task processing strategy; The processing module is used to process the data processing task based on the node dependencies and the task processing strategies defined by the multiple functional nodes.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images