Distributed task scheduling execution method and device, computer device, and storage medium

Abstract

The application belongs to the field of big data processing, and relates to a distributed task scheduling execution method, comprising the following steps: obtaining a target service instance and a target executor group corresponding to task information; determining a first task execution instance according to execution state information of each executor instance in the target executor group through a pre-trained executor optimal solution algorithm model; executing the task instance through the first task execution instance to obtain a first execution result; when the first execution result is task execution failure, reacquiring a second task execution instance through the executor optimal solution algorithm model until the task is successfully executed or a preset retry number is reached. The application also provides a distributed task scheduling execution device, a computer device and a storage medium. The application can improve the concurrency and reliability of task execution and ensure that all tasks are executed successfully and efficiently on time.

Description

Technical Field

[0001] This application relates to the fields of task scheduling and big data processing technology, and in particular to a distributed task scheduling execution method, apparatus, computer equipment and storage medium. Background Technology

[0002] In the field of task scheduling technology, the most widely used scheduling frameworks on the market today include Elastic-Job and XXL-Job, both based on Quartz. However, these frameworks all have some significant drawbacks in practical applications. Quartz, as a representative of the first generation of task scheduling frameworks, suffers from limitations in user experience due to historical legacy issues. Specifically, Quartz does not provide an intuitive web interface; users can only configure tasks through an API, which undoubtedly increases the difficulty of operation and reduces ease of use and flexibility. Furthermore, Quartz only supports single-machine execution mode, failing to fully utilize the computing power of a cluster environment, thus limiting its application potential in large-scale task scheduling scenarios. More importantly, Quartz's scheduling and execution functions are coupled within the same application, lacking platform-based service capabilities, which further limits its scalability and maintainability.

[0003] In contrast, the XXL-Job framework, as a representative of second-generation task scheduling frameworks, has overcome some of the aforementioned shortcomings of the Quartz framework. However, with continuous technological advancements and increasingly diverse application requirements, XXL-Job has gradually revealed some new deficiencies. For example, XXL-Job's database support is relatively limited, restricting its applicability in multi-database environments. Furthermore, while XXL-Job possesses some distributed computing capabilities, its distributed nature remains relatively limited, failing to meet the high-performance requirements of complex scenarios. In addition, XXL-Job lacks workflow support, which to some extent limits its application in complex business process management. Summary of the Invention

[0004] The purpose of this application is to propose a distributed task scheduling and execution method, apparatus, computer equipment and storage medium to solve the technical problems of existing task scheduling algorithms having limited data support, poor adaptability, parallelism and scalability, and difficulty in adapting to constantly changing system environments and task requirements.

[0005] To address the aforementioned technical problems, this application provides a distributed task scheduling and execution method, employing the following technical solution:

[0006] The system periodically obtains task information from the scheduling center, determines the target service instance and target executor group for executing the task based on the task information, and initializes the task instance based on the task information.

[0007] The execution status information of each executor instance in the target executor group is obtained, and the execution status information is input into the pre-trained executor optimal solution algorithm model for scoring calculation. The executor instance with the highest score is selected as the first task execution instance.

[0008] The task instance is distributed to the first task execution instance through the target service instance, and the first execution result of the first task execution instance executing the task according to the task instance is obtained, and it is determined whether the first execution result is that the task was executed successfully;

[0009] If the first execution result is that the task is executed successfully, the task status is updated through the target service instance. If the first execution result is that the task is executed unsuccessfully, the latest execution status information of other executor instances in the target executor group is obtained again through the target service instance. The other executor instances are all executor instances other than the first task execution instance.

[0010] The latest execution status information is input into the executor optimal solution algorithm model for scoring calculation, and the other executor instance with the highest score is selected as the second task execution instance;

[0011] The task instance is distributed to the second task execution instance through the target service instance, and the second execution result of the second task execution instance executing the task according to the task instance is obtained, and it is determined whether the second execution result is that the task was executed successfully;

[0012] If the second execution result is that the task is executed successfully, the task status is updated through the target service instance. If the second execution result is that the task is executed unsuccessfully, the executor instance is reselected through the target service instance until the task is executed successfully or the preset number of retries is reached.

[0013] To address the aforementioned technical problems, this application also provides a distributed task scheduling and execution device, which employs the following technical solution:

[0014] The acquisition module is used to periodically acquire task information from the scheduling center, determine the target service instance and target executor group for executing the task based on the task information, and initialize the task instance based on the task information.

[0015] The first selection module is used to obtain the execution status information of each executor instance in the target executor group, input the execution status information into the pre-trained executor optimal solution algorithm model for scoring calculation, and select the executor instance with the highest score as the first task execution instance;

[0016] The first task execution module is used to distribute the task instance to the first task execution instance through the target service instance, obtain the first execution result of the first task execution instance executing the task according to the task instance, and determine whether the first execution result is that the task was executed successfully;

[0017] The first execution result analysis module is used to update the task status through the target service instance if the first execution result is that the task was successfully executed, and to re-obtain the latest execution status information of other executor instances in the target executor group through the target service instance if the first execution result is that the task was unsuccessful. The other executor instances are all executor instances other than the first task execution instance.

[0018] The second selection module is used to input the latest execution status information into the executor optimal solution algorithm model for scoring calculation, and select the other executor instance with the highest score as the second task execution instance.

[0019] The second task execution module is used to distribute the task instance to the second task execution instance through the target service instance, obtain the second execution result of the second task execution instance executing the task according to the task instance, and determine whether the second execution result is that the task was successfully executed;

[0020] The second execution result analysis module is used to update the task status through the target service instance if the second execution result is that the task was successfully executed, and to continue to reselect an executor instance through the target service instance if the second execution result is that the task was unsuccessful, until the task is successfully executed or the preset number of retries is reached.

[0021] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0022] The computer device includes a memory and a processor. The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the distributed task scheduling and execution method described above.

[0023] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0024] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the distributed task scheduling and execution method described above.

[0025] Compared with the prior art, this application has the following main advantages:

[0026] This application provides a distributed task scheduling and execution method. It obtains the target service instance and target executor group corresponding to the task information. A pre-trained executor optimal solution algorithm model determines a first task execution instance based on the execution status information of each executor instance in the target executor group. The first task execution instance executes the task instance to obtain a first execution result. If the first execution result is a task execution failure, a second task execution instance is obtained again using the executor optimal solution algorithm model, until the task execution is successful or a preset number of retries is reached. This application improves the concurrency of task execution by determining the target service instance based on task information. Secondly, by selecting the best executor instance to execute the task using the optimal solution algorithm model, tasks can be dynamically allocated to the most suitable executor instance for stable execution, ensuring that all tasks are executed on time, efficiently, and successfully. Furthermore, when the executor group includes multiple executor instances, if one executor instance fails, the task can be dynamically transferred to other executor instances for continued execution, improving the reliability and stability of task execution. Attached Figure Description

[0027] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0029] Figure 2 This is a flowchart of an embodiment of the distributed task scheduling and execution method according to this application;

[0030] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S202;

[0031] Figure 4 This is a schematic diagram of the architecture of a specific embodiment of the distributed task scheduling and execution method according to this application;

[0032] Figure 5 This is a schematic diagram of a structure of an embodiment of the distributed task scheduling and execution device according to this application;

[0033] Figure 6 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0035] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0036] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0037] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0038] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0039] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0040] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0041] It should be noted that the distributed task scheduling and execution method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the distributed task scheduling and execution method apparatus is generally set in the server / terminal device.

[0042] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0043] Continue to refer to Figure 2 The flowchart illustrates an embodiment of the distributed task scheduling and execution method according to this application, including the following steps:

[0044] Step S201: Periodically obtain task information through the scheduling center, determine the target service instance and target executor group for executing the task based on the task information, and initialize the task instance based on the task information.

[0045] In this embodiment, a distributed, highly available, and horizontally scalable task scheduling cluster is constructed. This cluster includes a scheduling center (i.e., the server) and executor nodes. The scheduling center contains one or more service instances (i.e., Server nodes) responsible for managing the registration, status tracking, heartbeats, task management and distribution, and task instance management of all executors. The executor nodes consist of multiple isolated executor groups, each containing one or more executor instances (Workers). Each executor group belongs to the same Server managed by the scheduling center and is responsible for receiving and executing tasks dispatched from the scheduling center. The scheduling center and executor nodes communicate remotely. Communication between Workers and Servers includes heartbeat ping detection, Server task information distribution, Worker task execution information reporting, and Worker status monitoring.

[0046] Before executing the task, both the scheduling center and the executor are initialized. The specific steps for initializing the scheduling center include:

[0047] Start the remote communication system and run the Actor container on the server side;

[0048] Various types of Actor instances are created in the Actor container. Actor instances are used to process information reported from Workers, such as logs, task status, heartbeats, etc., as well as to receive tasks for scheduled task scheduling and distribution, and distribute tasks to the managed Workers.

[0049] Initialize Server context information, which includes database connection information and other environment information, such as email notification service and a distributed unique service ID created using the Server's IP address.

[0050] Start the timed scheduling service thread pool to schedule and distribute tasks;

[0051] Initialization complete, service is now available.

[0052] The specific steps for initializing the executor include:

[0053] Start the remote communication system and run the Actor container on the executor side;

[0054] Various types of Actor instances are created in the Actor container. Among them, the Actor instances on the executor side are used to receive tasks issued by the server, process related tasks within the worker, and receive information related to the worker and report it to the server, such as logs, heartbeats, and task execution status.

[0055] Initialize Worker context information, including clearing Worker historical data such as local database, tasks, logs, etc., and initializing local storage and other information;

[0056] Request the corresponding Server and obtain the executor group ID to which the Worker belongs;

[0057] Perform Server service discovery; after successful service discovery, update the relevant Worker information.

[0058] Start the scheduled task thread pool to periodically report heartbeats, logs, and task information executed by this Worker instance to the Server;

[0059] Initialization complete, service is now available.

[0060] In this embodiment, the server application integrates the AKKA framework (RPC remote communication framework) to automatically initialize the ActorSystem when the service instance starts, thereby managing the lifecycle of all Actor containers.

[0061] The client creates a task and sets the number of retries for each failure (i.e., the preset number of retries). It also sets the executor group for task execution and submits the task to the most suitable service instance of the scheduling center. After receiving the task request from the client, the scheduling center creates relevant task information in the background database to facilitate data storage and management, task tracking and monitoring, and data sharing and collaboration, ensuring data security and integrity.

[0062] In this embodiment, the scheduling center performs task scheduling and distribution periodically, obtains task information periodically through the scheduling center, determines the target executor group for the corresponding task execution based on the task information, determines the corresponding target service instance based on the target executor group, and then starts initializing the task instance based on the task information.

[0063] In some optional implementations of this embodiment, the step of determining the target service instance and target executor group for executing the task based on the task information includes:

[0064] Parse the task information to obtain task parameters, determine the target executor group based on the task parameters, and obtain the initial service instance associated with the target executor group;

[0065] Determine if the initial service instance is available;

[0066] If the initial service instance is available, then the initial service instance will be identified as the target service instance;

[0067] If the initial service instance is unavailable, obtain the service status information of all available service instances;

[0068] The service status information is input into the trained server optimal solution algorithm model to obtain the status score of all available service instances;

[0069] The available service instance with the highest status score is identified as the target service instance.

[0070] The task parameters include task identifier, task name, task type, execution logic, execution parameters, executor group identifier, retry policy (including preset number of retries, retry interval, etc.), log records, and task priority information.

[0071] Parse the task parameters to obtain the target executor group associated with the task. Based on the executor group identifier of the target executor group, obtain the initial service instance registered by the target executor group. Send a Ping signal to the initial service instance to detect heartbeat through the executor instance. If a response to the Ping signal is received from the initial service instance, it means that the initial service instance is available, and the initial service instance is identified as the target service instance. If no response is received from the initial service instance, it means that the initial service instance is unavailable. Then, obtain the service status information of all currently available service instances. Calculate and decide on the service status information using a trained server optimal solution algorithm model to obtain the status score of all available service instances. The available service instance with the highest status score is selected as the target service instance, and the target executor group is re-registered or bound to the target service instance.

[0072] The service status information includes the current running data of each service instance, including the current service's memory usage, CPU utilization, network I / O, number of tasks to be dispatched, and the number of currently managed executor groups and executors.

[0073] In this embodiment, the initial service instance is also selected through the server optimal solution algorithm model, which is trained using decision tree algorithms such as ID3, C5 or CART.

[0074] Determining the availability of a service instance to identify the target service instance can improve the success rate of task execution. Assigning tasks to available servers can avoid resource waste, while ensuring that tasks are completed on time and efficiently, enhancing the stability and reliability of the entire system and improving the user experience.

[0075] Step S202: Obtain the execution status information of each executor instance in the target executor group, input the execution status information into the pre-trained executor optimal solution algorithm model for scoring calculation, and select the executor instance with the highest score as the first task execution instance.

[0076] In this embodiment, the target executor group contains multiple executor instances. The execution status information of each executor instance is obtained. The execution status information includes the memory utilization, CPU utilization, network I / O, number of tasks to be executed, total GC (Garbage Collection) time, longest GC time, and average GC time of the executor.

[0077] The collected execution status information is input into the trained executor optimal solution algorithm model for calculation, and the first score corresponding to each executor instance is obtained. Based on the first score, the executor instance with the highest score is determined as the first task execution instance to execute the corresponding task.

[0078] In some alternative implementations, see [link to relevant documentation]. Figure 3 As shown, the step of inputting the execution state information into the pre-trained executor optimal solution algorithm model further includes:

[0079] Step S301: Obtain the first running state dataset for each executor instance, and extract the first target variable and the first feature variable based on the first running state dataset;

[0080] Step S302: Calculate the first contribution of each first feature variable to the first target variable using the Gini index, and determine the first preset number of first feature variables most relevant to the first target variable as the root node of the executor of the decision tree based on the first contribution.

[0081] Step S303: Obtain the executor feature value of the executor root node, and divide the first running state dataset into a first feature subset of a first preset number based on the executor feature value;

[0082] Step S304: Using a recursive partitioning method, the executor instance is assigned to different child nodes based on the feature values ​​of each first feature subset, until the leaf node is reached, thereby generating the executor decision model.

[0083] Step S305: Optimize the executor decision model through pruning operations to obtain the optimal solution algorithm model for the executor.

[0084] In this embodiment, the historical running status of each executor is periodically collected to form a first running status dataset. The historical running status includes the memory usage, CPU usage, CPU load, network I / O status, total GC time / longest GC time / average GC time of virtual machines, number of tasks that succeeded / failed / blocked, longest task time / total time / average time, number of task retries, etc.

[0085] The acquired first running state dataset is preprocessed to obtain a preprocessed first running state dataset. Preprocessing includes data cleaning and normalization. Data cleaning removes outliers, missing values, and other noisy data to ensure accuracy and completeness. For example, Python's Pandas library can be used for data cleaning. Normalization refers to normalizing the performance metric data across different dimensions, scaling the data to a uniform range, such as between 0 and 1. Common normalization methods include min-max normalization and Z-score normalization.

[0086] A decision tree is recursively constructed using the preprocessed first running state dataset. In this embodiment, the CART decision tree algorithm is used to generate multiple decision trees to form a decision model.

[0087] Extract the first target variable and multiple first feature variables related to the business from the first running state dataset, and calculate the importance measure of each first feature variable to the first target variable. The importance measure is calculated using the Gini index.

[0088] The Gini index for each first characteristic variable is calculated using the following formula:

[0089]

[0090] Where P is the first running state dataset, k is the number of predicted categories for the first feature variable, and p i This represents the proportion of samples in different predicted categories under the same primary feature variable.

[0091] The larger the Gini index, the greater the uncertainty of the sample; the smaller the Gini index, the greater the correlation of the features and the purer the classification.

[0092] In this embodiment, the Gini index of each first feature variable is used as the first contribution of the first feature variable to the first target variable. The smaller the Gini index, the more correlated the first feature variable is with the first target variable. All first feature variables are sorted according to the Gini index from smallest to largest. The first preset number of first feature variables at the top of the sorted list are selected as the root node of the current decision tree. The feature value corresponding to each root node is obtained. The first running state dataset is divided into a first feature subset with the same number of root nodes according to the feature value. Steps S302 and S303 are recursively executed for each first feature subset until a preset stopping condition is met, such as the number of samples in the leaf node is less than a threshold or the depth of the tree reaches a limit, to obtain the executor decision model. At the leaf node, the probability distribution of the corresponding first target variable is determined according to the samples contained in the leaf node, and the probability distribution is used as the scoring result.

[0093] After training, the structural parameters of the executor decision model are obtained, including tree depth and the number of leaf node samples. These structural parameters are compared with the set pruning parameters to determine if the structural complexity of the decision tree exceeds a preset threshold. If it does, pruning is performed. The set pruning parameters include the maximum tree depth parameter and the minimum sample number parameter. The maximum depth parameter limits the tree depth and reduces the number of branches, while the minimum sample number parameter limits the minimum number of samples per leaf node to reduce overfitting. Based on the set pruning parameters, the structure of the executor decision model is pruned to obtain a simplified executor decision model. The simplified executor decision model is then used to predict test set data, and the model's accuracy, precision, and recall on the test set are calculated to evaluate generalization performance. If the generalization performance meets the requirements, the final decision tree model is output as the optimal solution algorithm model for the executor; otherwise, the pruning parameters are adjusted, and the above steps are repeated until the requirements are met.

[0094] By training and optimizing the executor decision model, we can improve the model's generalization ability, accuracy, and robustness. This helps to accurately select executor instances with higher scores to execute tasks during task scheduling, thereby improving task execution efficiency and resource utilization.

[0095] Step S203: Distribute the task instance to the first task execution instance through the target service instance, obtain the first execution result of the first task execution instance executing the task according to the task instance, and determine whether the first execution result indicates that the task was successfully executed.

[0096] When scheduling tasks, each service instance periodically retrieves all task information associated with all executor groups it holds from the database. It then calculates the next trigger time for each task and updates the database with the latest next trigger time, simultaneously pushing the task into the task scheduling queue for triggering. When distributing tasks in the queue, the service instance's task scheduling thread first queries the list of all executor workers associated with the task's executor group, then calls the executor optimal solution algorithm model to calculate the instantaneous state score of each executor instance. Finally, it selects the executor instance with the highest score for communication and distributes the task to that instance for execution.

[0097] In this embodiment, the first task execution instance executes the task according to the task instance distributed by the target service instance, generates a first execution result, and returns the first execution result to the target service instance. The target service instance determines whether the task was executed successfully based on the first execution result.

[0098] Step S204: If the first execution result is that the task is executed successfully, the task status is updated through the target service instance. If the first execution result is that the task is executed unsuccessfully, the latest execution status information of other executor instances in the target executor group is obtained again through the target service instance. The other executor instances are all executor instances other than the first task execution instance.

[0099] In this embodiment, the task ends when it is successfully executed. The target service instance writes the successful execution status of the task to the task instance record in the database and marks the task as completed. If the task fails, the target service instance retrieves the latest execution status information of other executor instances in the target executor group according to the task retry policy. The other executor instances are all executor instances other than the first task execution instance.

[0100] Step S205: Input the latest execution status information into the executor optimal solution algorithm model for score calculation, and select the other executor instance with the highest score as the second task execution instance.

[0101] The latest execution status information collected is input into the trained executor optimal solution algorithm model for calculation to obtain the second score corresponding to each other executor instance. Based on the second score, the other executor instance with the highest score is determined as the second task execution instance to execute the corresponding task.

[0102] The latest execution status information includes the memory utilization, CPU utilization, network I / O, number of tasks currently pending execution, total GC time / longest GC time / average GC time of other executors, etc.

[0103] Step S206: Distribute the task instance to the second task execution instance through the target service instance, obtain the second execution result of the second task execution instance executing the task according to the task instance, and determine whether the second execution result indicates that the task was successfully executed.

[0104] In this embodiment, the second task execution instance executes the task according to the task instance distributed by the target service instance, generates a second execution result, and returns the second execution result to the target service instance. The target service instance determines whether the task was executed successfully based on the second execution result.

[0105] Step S207: If the second execution result is that the task is executed successfully, the task status is updated through the target service instance. If the second execution result is that the task is executed unsuccessfully, the executor instance is reselected through the target service instance until the task is executed successfully or the preset number of retries is reached.

[0106] In this embodiment, if the second execution result is that the task is executed successfully, the task execution is completed and the target service instance writes the successful execution status of the task to the task instance record in the database and marks the task execution as completed. If the second execution result is that the task is executed unsuccessfully, the target service instance, according to the task retry policy, re-obtains the latest execution status information of the remaining other executor instances in the target executor group and repeats steps S205 to S207 until the task is executed successfully or the preset number of retries is reached.

[0107] This application can be applied to the finance, internet, and big data industries. By determining the target service instance for task execution based on task information, the concurrency of task execution can be improved. Secondly, by selecting the best executor instance to execute the task through the optimal solution algorithm model, tasks can be dynamically allocated to the most suitable executor instance for stable execution, ensuring that all tasks are executed on time, efficiently, and successfully. In addition, when the executor group includes multiple executor instances, if one executor instance fails, the task can be dynamically transferred to other executor instances to continue execution, improving the reliability and stability of task execution.

[0108] In some optional implementations of this embodiment, the method further includes the following step before inputting the service status information into the trained server optimal solution algorithm model:

[0109] Step A: Obtain the second running status dataset for each server, and extract the second target variable and the second feature variable based on the second running status dataset;

[0110] Step B: Calculate the second contribution of each second feature variable to the second target variable using information gain, and determine the second preset number of second feature variables most relevant to the second target variable as the server root node of the decision tree based on the second contribution.

[0111] Step C: Obtain the server feature values ​​of the server root node, and divide the second running state dataset into a second preset number of second feature subsets based on the server feature values;

[0112] Step D: Using a recursive partitioning method, the server is assigned to different child nodes based on the feature values ​​of each second feature subset, until the leaf node is reached, thus generating the server decision model.

[0113] Step E involves optimizing the server decision model through pruning operations to obtain the optimal solution algorithm model for the server.

[0114] In this embodiment, the historical running status of each service instance is periodically collected to form a second running status dataset. The historical status includes the memory usage, CPU usage, CPU load, network I / O status, total GC duration / longest GC duration / average GC duration of virtual machines, number of successful / failed / blocked task distributions, number of managed executor group instances, and number of managed executors, etc.

[0115] The acquired second running state dataset is preprocessed to obtain a preprocessed second running state dataset. Preprocessing includes data cleaning and normalization. A decision tree is recursively constructed using the preprocessed second running state dataset. In this embodiment, the C5 decision tree algorithm is used to generate multiple decision trees, forming a decision model.

[0116] Extract the second target variable and multiple second feature variables related to the service from the second running state dataset, and calculate the importance measure of each second feature variable to the second target variable. The importance measure is calculated using information gain.

[0117] The Gini index for each second characteristic variable is calculated using the following formula for information gain:

[0118] Gain(second objective variable) = Ent(D1) - Ent(D1|second objective variable);

[0119] In the formula, Ent(D1|second objective variable) represents the conditional information entropy of a certain second objective variable. The formula for calculating the conditional information entropy is as follows:

[0120]

[0121] In the formula, p j For the j-th second objective variable C j The probability of this second target variable appearing in all samples, Ent(D1|C j ) represents the second objective variable C j The entropy of attribute information, Wherein, it represents the i-th second objective variable in the second objective variable C. j The probability of occurrence in all samples.

[0122] It should be noted that the greater the information gain of the second target variable, the higher the homogeneity of the second feature subset based on the second target variable, which is more conducive to classification.

[0123] In this embodiment, the information gain of each second feature variable is used as the second contribution of the second feature variable to the second target variable. The larger the information gain, the more relevant the second feature variable is to the second target variable. All second feature variables are sorted from largest to smallest according to their information gain. The second preset number of second feature variables at the top of the sorted list are selected as the root nodes of the current decision tree. The feature values ​​corresponding to each root node are obtained. The second running state dataset is divided into second feature subsets with the same number of root nodes according to the feature values. Steps B and C are recursively executed for each second feature subset until the preset stopping conditions are met, such as the number of samples in the leaf nodes being less than a threshold or the depth of the tree reaching a limit, to obtain the server decision model.

[0124] After training, obtain the structural parameters of the server decision model, including tree depth and the number of leaf node samples. Based on the structural parameters and the set pruning parameters, prune the server decision model to obtain a simplified model. Use the simplified server decision model to predict test set data and calculate the model's accuracy, precision, recall, and other metrics on the test set to evaluate generalization performance. If the generalization performance meets the requirements, output the final decision tree model as the optimal solution algorithm model for the server; otherwise, return to adjust the pruning parameters and repeat the above steps until the requirements are met.

[0125] By training and optimizing the server decision model, we can improve the model's generalization ability, accuracy, and robustness. This helps to accurately select the best service instance to manage and distribute tasks during task scheduling, thereby improving task distribution efficiency and resource utilization.

[0126] In some optional implementations, the step of obtaining task information from the scheduling center at regular intervals is further included:

[0127] Start the service instance and executor instance;

[0128] Obtain the service instance corresponding to the executor instance, and determine whether the service instance is available;

[0129] When a service instance is unavailable, a candidate service instance is obtained through the server optimal solution algorithm model;

[0130] Using the executor instance, service discovery is performed on the candidate service instance based on the distributed lock to determine whether the candidate service instance is available;

[0131] When a candidate service instance is available, the candidate service instance is selected as the optimal service instance, and the executor instance is associated with the optimal service instance.

[0132] If a candidate service instance is unavailable, a new service instance is acquired until the optimal service instance is found, and then the distributed lock is released.

[0133] In this embodiment, the executor instance is pre-registered with the server and associated with the service instance. Before the task begins execution, both the service instance and the executor instance are started. After the service instance starts, it registers relevant instance information in the database. When the executor instance starts, service discovery is performed. Specifically, after the executor instance starts, it queries the associated service instance, which is the initial service instance. The executor instance sends a Ping signal to this initial service instance to detect its heartbeat. If a response to the Ping signal is received from the initial service instance, it indicates that the initial service instance is available, and this initial service instance is identified as the target service instance. If no response to the Ping signal is received from the initial service instance, it indicates that the initial service instance is unavailable.

[0134] When the initial service instance is unavailable, the system queries the Server list configured in the initialization process. The optimal server solution algorithm selects the best candidate service instance. The executor instance uses a distributed lock to discover the candidate service instance and sends a Ping signal to it for heartbeat detection. If the executor instance receives a response to the Ping signal, it indicates that the candidate service instance is available, and it is designated as the optimal service instance for that executor instance. The executor instance is then migrated from the unavailable service instance to the selected optimal service instance to continue providing service. If the selected candidate service instance is unavailable, the optimal server solution algorithm continues to evaluate and select from the Server list until the optimal service instance is found, at which point the distributed lock is released. If no available service instance is found in the Server list, it indicates that the current executor instance is offline, and error handling is performed. The distributed lock is a synchronization mechanism to control multiple executor instances from simultaneously initiating service discovery, primarily to prevent multiple executor instances from selecting the same service instance at the same time.

[0135] At this point, before the task begins execution, all executor instances in an executor group are registered to the best available service instance.

[0136] By performing service discovery before task execution, system availability can be improved, faults can be detected and handled in a timely manner, resource allocation can be optimized, and system scalability and stability can be improved, thereby ensuring the smooth and efficient execution of tasks and the overall performance of the system.

[0137] In some optional implementations of this embodiment, the step of determining whether a candidate service instance is available by performing service discovery on the candidate service instance based on a distributed lock using an executor instance includes:

[0138] The executor instance sends a distributed lock acquisition request to the candidate service instance;

[0139] Upon receiving a successful locking response from a candidate service instance, the executor instance sends a response signal to the candidate service instance.

[0140] The availability of a candidate service instance is determined based on the response result of the response signal.

[0141] In this embodiment, if multiple executor instances simultaneously discover that previously registered service instances cannot communicate, they will simultaneously initiate server service discovery. Each executor instance first queries the database to find the service instance corresponding to its current executor instance. At this point, one executor instance in the list of Worker instances awaiting service discovery acquires a distributed lock to perform service discovery, while other executor instances continue to wait and retry. The executor instance that acquires the distributed lock then selects a service instance and updates the service instance information of its executor group. After the update, other executor instances will then perform service discovery and will find new available service instances.

[0142] Specifically, after obtaining the optimal service instance through the server's optimal solution algorithm model, the executor instance sends a distributed lock acquisition request to the database. The database locks the corresponding optimal service instance based on the received lock request and returns a lock acquisition feedback result to the executor instance. If the lock acquisition feedback result indicates successful locking, the executor instance sends a ping signal to the locked optimal service instance. If a response to the ping signal is received from the optimal service instance, the service instance association information of the group (executor group) corresponding to this executor instance is updated to the new service instance. After a successful update, the distributed lock is released, and service discovery continues for the next executor instance until all executor instances in the Worker instance list have found their associated service instances.

[0143] By locking service instances during the service discovery process on the executor instance, duplicate service instance discovery can be avoided, ensuring data consistency and thus improving the reliability and stability of the entire distributed task scheduling framework.

[0144] In some alternative implementations, the step of reaching the preset number of retries further includes:

[0145] Get the current running data of all service instances;

[0146] The optimal service instance is selected as the target service instance based on the current running data using the server optimal solution algorithm model.

[0147] In this embodiment, when the task reaches the preset number of retries of the retry policy, service discovery can be re-executed. The current running data of all service instances is calculated using the server optimal solution algorithm model, and the current optimal service instance is selected as the target service instance. Steps S201 to S207 are repeated until the task is successfully executed.

[0148] By re-performing service discovery and obtaining new target service instances, resource allocation can be optimized, while improving system scalability and stability, ensuring the smooth and efficient execution of tasks.

[0149] The following specific embodiment will be used to illustrate the content of this application in more detail and to further elaborate on this application, but these embodiments are by no means intended to limit this application.

[0150] See Figure 4 As shown, the cluster deployment in this embodiment includes three service instances and nine executor instances. The three service instances are Server1, Server2, and Server3. Each group of three executor instances forms an executor group. For example, worker11, worker12, and worker13 form executor group 1 and register with Server1; worker21, worker22, and worker23 form executor group 2 and register with Server2; and worker31, worker32, and worker33 form executor group 3 and register with Server3.

[0151] In this embodiment, the above-mentioned cluster is used to execute a data statistics task of SQL type, including the following steps:

[0152] Step 1: The client creates an SQL statistics task, sets the default number of retries for the task to 3, and specifies that the task is executed on executor group 1. The task is then submitted to Server1, which is associated with executor group 1. After receiving the task request, Server1 creates the relevant task information in the background database.

[0153] Step 2: Run the SQL statistics task and submit the task to the scheduling center.

[0154] Step 3: The scheduling center discovers through task query that the sub-task needs to be executed on executor group 1, starts to initialize the task instance, and queries the three instances worker11, worker12 and worker13 associated with executor group 1. According to the executor optimal solution algorithm model, the best executor instance is selected to execute the task. Let's assume that worker12 instance is selected.

[0155] Step 4: worker12 starts executing the task. If the task fails, the task status is submitted to the server1 registered with worker12.

[0156] Step 5: If server1 finds that the SQL statistics task has failed, it will continue to select worker11 or worker13 to execute the task according to the task retry strategy and the executor optimal solution algorithm model. Let's assume that the task is executed on worker13 at this time.

[0157] Step 6: worker13 receives the task from server1 and starts executing the SQL statistics task. Assuming the task is executed successfully, worker13 reports the success status of the task to server1, which then writes it to the task instance record in the database and marks the task as completed.

[0158] Step 7: After the task is completed, when the client queries the task again, it will return a message indicating that the task was successfully executed.

[0159] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0160] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0161] Further reference Figure 5 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a distributed task scheduling and execution device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0162] like Figure 5 As shown, the distributed task scheduling and execution device 500 described in this embodiment includes: an acquisition module 501, a first selection module 502, a first task execution module 503, a first execution result analysis module 504, a second selection module 505, a second task execution module 506, and a second execution result analysis module 507. Wherein:

[0163] The acquisition module 501 is used to periodically acquire task information through the scheduling center, determine the target service instance and target executor group for executing the task based on the task information, and initialize the task instance based on the task information.

[0164] The first selection module 502 is used to obtain the execution status information of each executor instance in the target executor group, input the execution status information into the pre-trained executor optimal solution algorithm model for scoring calculation, and select the executor instance with the highest score as the first task execution instance;

[0165] The first task execution module 503 is used to distribute the task instance to the first task execution instance through the target service instance, obtain the first execution result of the first task execution instance executing the task according to the task instance, and determine whether the first execution result is that the task was executed successfully;

[0166] The first execution result analysis module 504 is used to update the task status through the target service instance if the first execution result is that the task was successfully executed, and to re-obtain the latest execution status information of other executor instances in the target executor group through the target service instance if the first execution result is that the task was unsuccessful. The other executor instances are all executor instances other than the first task execution instance.

[0167] The second selection module 505 is used to input the latest execution status information into the executor optimal solution algorithm model for scoring calculation, and select the other executor instance with the highest score as the second task execution instance;

[0168] The second task execution module 506 is used to distribute the task instance to the second task execution instance through the target service instance, obtain the second execution result of the second task execution instance executing the task according to the task instance, and determine whether the second execution result is that the task was successfully executed;

[0169] The second execution result analysis module 507 is used to update the task status through the target service instance if the second execution result is that the task was successfully executed, and to continue to reselect the executor instance through the target service instance if the second execution result is that the task was unsuccessful, until the task is successfully executed or the preset number of retries is reached.

[0170] Based on the aforementioned distributed task scheduling and execution device 500, the concurrency of task execution can be improved by determining the target service instance for task execution according to task information. Secondly, by selecting the best executor instance to execute the task through the optimal solution algorithm model, tasks can be dynamically allocated to the most suitable executor instance for stable execution, ensuring that all tasks are executed on time, efficiently and successfully. In addition, when the executor group includes multiple executor instances, if one executor instance fails, the task can be dynamically transferred to other executor instances to continue execution, improving the reliability and stability of task execution.

[0171] In some optional implementations of this embodiment, the acquisition module 501 includes:

[0172] The parsing submodule is used to parse the task information to obtain task parameters, determine the target executor group based on the task parameters, and obtain the initial service instance associated with the target executor group;

[0173] The judgment submodule is used to determine whether the initial service instance is available;

[0174] The first determining submodule is used to determine the initial service instance as the target service instance if the initial service instance is available;

[0175] The acquisition submodule is used to acquire the service status information of all available service instances if the initial service instance is unavailable.

[0176] The service selection submodule is used to input the service status information into the trained server optimal solution algorithm model to obtain the status scores of all available service instances;

[0177] The second determination submodule is used to determine the available service instance with the highest status score as the target service instance.

[0178] Determining the availability of a service instance to identify the target service instance can improve the success rate of task execution. Assigning tasks to available servers can avoid resource waste, while ensuring that tasks are completed on time and efficiently, enhancing the stability and reliability of the entire system and improving the user experience.

[0179] In some alternative implementations, the distributed task scheduling and execution device 500 further includes a first training module, used for:

[0180] Obtain a first running state dataset for each executor instance, and extract a first target variable and a first feature variable based on the first running state dataset;

[0181] The first contribution of each first feature variable to the first target variable is calculated using the Gini index, and a first preset number of first feature variables most relevant to the first target variable are determined as the root nodes of the executor of the decision tree based on the first contribution.

[0182] Obtain the executor feature value of the executor root node, and divide the first running state dataset into a first feature subset of a first preset number based on the executor feature value;

[0183] A recursive partitioning method is adopted, which divides the executor instance into different child nodes according to the feature value of each first feature subset, until the leaf node is reached, thereby generating the executor decision model.

[0184] The executor decision model is optimized by pruning operations to obtain the optimal solution algorithm model for the executor.

[0185] By training and optimizing the executor decision model, we can improve the model's generalization ability, accuracy, and robustness. This helps to accurately select executor instances with higher scores to execute tasks during task scheduling, thereby improving task execution efficiency and resource utilization.

[0186] In some alternative implementations, the distributed task scheduling and execution device 500 further includes a second training module, used for:

[0187] Obtain a second running status dataset for each of the servers, and extract a second target variable and a second feature variable based on the second running status dataset;

[0188] The second contribution of each second feature variable to the second target variable is calculated by information gain, and a second preset number of second feature variables most relevant to the second target variable are determined as the server root node of the decision tree based on the second contribution.

[0189] Obtain the server feature value of the server root node, and divide the second running status dataset into a second feature subset of a second preset number based on the server feature value;

[0190] A recursive partitioning method is adopted, which divides the server according to the feature value of each second feature subset and assigns the server to different child nodes until the leaf node is reached, thereby generating a server decision model.

[0191] The server decision-making model is optimized by pruning operations to obtain the optimal solution algorithm model for the server.

[0192] By training and optimizing the server decision model, we can improve the model's generalization ability, accuracy, and robustness. This helps to accurately select the best service instance to manage and distribute tasks during task scheduling, thereby improving task distribution efficiency and resource utilization.

[0193] In some alternative implementations, the aforementioned distributed task scheduling and execution device 500 further includes a service discovery module, comprising:

[0194] The startup submodule is used to start service instances and executor instances;

[0195] The judgment submodule is used to obtain the service instance corresponding to the executor instance and determine whether the service instance is available.

[0196] The selection submodule is used to obtain candidate service instances through the server optimal solution algorithm model when the service instance is unavailable.

[0197] The service discovery submodule is used to perform service discovery on the candidate service instances based on the distributed lock using the executor instance to determine whether the candidate service instances are available;

[0198] The association submodule is used to, when the candidate service instance is available, use the candidate service instance as the optimal service instance and associate the executor instance with the optimal service instance;

[0199] The reselection submodule is used to reacquire a service instance when the candidate service instance is unavailable, until the optimal service instance is acquired, and then release the distributed lock.

[0200] By performing service discovery before task execution, system availability can be improved, faults can be detected and handled in a timely manner, resource allocation can be optimized, and system scalability and stability can be improved, thereby ensuring the smooth and efficient execution of tasks and the overall performance of the system.

[0201] In some optional implementations of this embodiment, the service discovery submodule is further configured to:

[0202] The executor instance sends a distributed lock acquisition request to the candidate service instance;

[0203] Upon receiving successful locking feedback from the candidate service instance, the executor instance sends a response signal to the candidate service instance.

[0204] The availability of the candidate service instance is determined based on the response result of the response signal.

[0205] By locking service instances during the service discovery process on the executor instance, duplicate service instance discovery can be avoided, ensuring data consistency and thus improving the reliability and stability of the entire distributed task scheduling framework.

[0206] In some alternative implementations, the aforementioned distributed task scheduling and execution device 500 further includes a service reselection module, used for:

[0207] Get the current running data of all service instances;

[0208] Based on the current running data, the optimal service instance is selected as the target service instance using the server optimal solution algorithm model.

[0209] By re-performing service discovery and obtaining new target service instances, resource allocation can be optimized, while improving system scalability and stability, ensuring the smooth and efficient execution of tasks.

[0210] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0211] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only a computer device 6 with a memory 61, a processor 62, and a network interface 63 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0212] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0213] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may include both the internal storage unit and its external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as the program code of a distributed task scheduling execution method. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or will be output.

[0214] In some embodiments, the processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run program code stored in the memory 61 or process data, for example, to run the program code of the distributed task scheduling execution method.

[0215] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 6 and other electronic devices.

[0216] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the distributed task scheduling execution method described above.

[0217] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0218] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A distributed task scheduling and execution method, characterized in that, Includes the following steps: The system periodically obtains task information from the scheduling center, determines the target service instance and target executor group for executing the task based on the task information, and initializes the task instance based on the task information. The execution status information of each executor instance in the target executor group is obtained, and the execution status information is input into the pre-trained executor optimal solution algorithm model for scoring calculation. The executor instance with the highest score is selected as the first task execution instance. The task instance is distributed to the first task execution instance through the target service instance, and the first execution result of the first task execution instance executing the task according to the task instance is obtained, and it is determined whether the first execution result is that the task was executed successfully; If the first execution result is that the task is executed successfully, the task status is updated through the target service instance. If the first execution result is that the task is executed unsuccessfully, the latest execution status information of other executor instances in the target executor group is obtained again through the target service instance. The other executor instances are all executor instances other than the first task execution instance. The latest execution status information is input into the executor optimal solution algorithm model for scoring calculation, and the other executor instance with the highest score is selected as the second task execution instance; The task instance is distributed to the second task execution instance through the target service instance, and the second execution result of the second task execution instance executing the task according to the task instance is obtained, and it is determined whether the second execution result is that the task was executed successfully; If the second execution result is that the task is executed successfully, the task status is updated through the target service instance. If the second execution result is that the task is executed unsuccessfully, the executor instance is reselected through the target service instance until the task is executed successfully or the preset number of retries is reached.

2. The distributed task scheduling and execution method according to claim 1, characterized in that, Before the step of inputting the execution state information into the pre-trained executor optimal solution algorithm model, the following is also included: Obtain a first running state dataset for each executor instance, and extract a first target variable and a first feature variable based on the first running state dataset; The first contribution of each first feature variable to the first target variable is calculated using the Gini index, and a first preset number of first feature variables most relevant to the first target variable are determined as the root nodes of the executor of the decision tree based on the first contribution. Obtain the executor feature value of the executor root node, and divide the first running state dataset into a first feature subset of a first preset number based on the executor feature value; A recursive partitioning method is adopted, which divides the executor instance into different child nodes according to the feature value of each first feature subset, until the leaf node is reached, thereby generating the executor decision model. The executor decision model is optimized by pruning operations to obtain the optimal solution algorithm model for the executor.

3. The distributed task scheduling and execution method according to claim 1, characterized in that, The step of determining the target service instance and target executor group for executing the task based on the task information includes: The task information is parsed to obtain task parameters, the target executor group is determined based on the task parameters, and the initial service instance associated with the target executor group is obtained. Determine whether the initial service instance is available; If the initial service instance is available, then the initial service instance is determined as the target service instance; If the initial service instance is unavailable, then obtain the service status information of all available service instances; The service status information is input into the trained server optimal solution algorithm model to obtain the status score of all available service instances; The available service instance with the highest status score is identified as the target service instance.

4. The distributed task scheduling and execution method according to claim 3, characterized in that, Before the step of inputting the service status information into the trained server optimal solution algorithm model, the following is also included: Obtain a second running status dataset for each of the servers, and extract a second target variable and a second feature variable based on the second running status dataset; The second contribution of each second feature variable to the second target variable is calculated by information gain, and a second preset number of second feature variables most relevant to the second target variable are determined as the server root node of the decision tree based on the second contribution. Obtain the server feature value of the server root node, and divide the second running status dataset into a second feature subset of a second preset number based on the server feature value; A recursive partitioning method is adopted, which divides the server according to the feature value of each second feature subset and assigns the server to different child nodes until the leaf node is reached, thereby generating a server decision model. The server decision-making model is optimized by pruning operations to obtain the optimal solution algorithm model for the server.

5. The distributed task scheduling and execution method according to claim 1, characterized in that, The step of obtaining task information from the scheduling center at regular intervals also includes: Start the service instance and executor instance; Obtain the service instance corresponding to the executor instance, and determine whether the service instance is available; If the service instance is unavailable, a candidate service instance is obtained through the server optimal solution algorithm model; Using the executor instance, service discovery is performed on the candidate service instance based on the distributed lock to determine whether the candidate service instance is available; When the candidate service instance is available, the candidate service instance is selected as the optimal service instance, and the executor instance is associated with the optimal service instance. If the candidate service instance is unavailable, a new service instance is obtained until the optimal service instance is found, and then the distributed lock is released.

6. The distributed task scheduling and execution method according to claim 5, characterized in that, The step of performing service discovery on the candidate service instance based on the distributed lock using the executor instance to determine whether the candidate service instance is available includes: The executor instance sends a distributed lock acquisition request to the candidate service instance; Upon receiving successful locking feedback from the candidate service instance, the executor instance sends a response signal to the candidate service instance. The availability of the candidate service instance is determined based on the response result of the response signal.

7. The distributed task scheduling and execution method according to any one of claims 1 to 6, characterized in that, After the step of reaching the preset number of retries, the method further includes: Get the current running data of all service instances; Based on the current running data, the optimal service instance is selected as the target service instance using the server optimal solution algorithm model.

8. A distributed task scheduling and execution device, characterized in that, include: The acquisition module is used to periodically acquire task information from the scheduling center, determine the target service instance and target executor group for executing the task based on the task information, and initialize the task instance based on the task information. The first selection module is used to obtain the execution status information of each executor instance in the target executor group, input the execution status information into the pre-trained executor optimal solution algorithm model for scoring calculation, and select the executor instance with the highest score as the first task execution instance; The first task execution module is used to distribute the task instance to the first task execution instance through the target service instance, obtain the first execution result of the first task execution instance executing the task according to the task instance, and determine whether the first execution result is that the task was executed successfully; The first execution result analysis module is used to update the task status through the target service instance if the first execution result is that the task was successfully executed, and to re-obtain the latest execution status information of other executor instances in the target executor group through the target service instance if the first execution result is that the task was unsuccessful. The other executor instances are all executor instances other than the first task execution instance. The second selection module is used to input the latest execution status information into the executor optimal solution algorithm model for scoring calculation, and select the other executor instance with the highest score as the second task execution instance. The second task execution module is used to distribute the task instance to the second task execution instance through the target service instance, obtain the second execution result of the second task execution instance executing the task according to the task instance, and determine whether the second execution result is that the task was successfully executed; The second execution result analysis module is used to update the task status through the target service instance if the second execution result is that the task was successfully executed, and to continue to reselect an executor instance through the target service instance if the second execution result is that the task was unsuccessful, until the task is successfully executed or the preset number of retries is reached.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the distributed task scheduling execution method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the distributed task scheduling and execution method as described in any one of claims 1 to 7.

Images