Distributed task scheduling execution method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of big data processing, and relates to a distributed task scheduling execution method, which comprises the following steps: acquiring a target service instance and a target actuator group corresponding to task information, determining a first task execution instance according to the execution state information of each executor instance in the target executor group through a pre-trained executor optimal solution algorithm model, executing the task instance through the first task execution instance to obtain a first execution result, and when the first execution result is task execution failure, executing the task instance through the first task execution instance to obtain a second execution result. And obtaining the second task execution instance again through the actuator optimal solution algorithm model until the task is successfully executed or the preset retry times are reached. The invention further provides a distributed task scheduling execution device, computer equipment and a storage medium. The task execution concurrency and reliability can be improved, and it is guaranteed that all tasks are efficiently and successfully executed on time.

Description

Technical Field

[0001] The present application relates to the technical field of task scheduling and big data processing, and in particular to a distributed task scheduling execution method, apparatus, computer equipment and storage medium. Background Art

[0002] In the field of task scheduling technology, the scheduling frameworks widely used in the current market mainly include Elastic-Job and xxl-job based on Quartz. However, these scheduling frameworks all have some significant shortcomings in practical applications. The Quartz framework, as a representative of the first generation of task scheduling frameworks, has certain limitations in user experience due to historical issues. Specifically, Quartz does not provide an intuitive web interface, and users can only configure tasks through the API, which undoubtedly increases the difficulty of operation and reduces the convenience and flexibility of use. In addition, Quartz only supports the stand-alone execution mode of tasks and cannot fully utilize the computing power of the cluster environment, thereby limiting its application potential in large-scale task scheduling scenarios. More importantly, Quartz's scheduling and execution functions are coupled in the same application, lacking the ability of platform services, which further limits its scalability and maintainability.

[0003] In contrast, as a representative of the second-generation task scheduling framework, the xxl-job framework has overcome the above-mentioned shortcomings of the Quartz framework to a certain extent. However, with the continuous advancement of technology and the increasing diversification of application requirements, xxl-job has gradually exposed some new defects. For example, xxl-job is relatively single in terms of database support, which limits its applicability in a multi-database environment. At the same time, although xxl-job has certain distributed computing capabilities, its distributed characteristics are still relatively limited and cannot meet the high-performance requirements in complex scenarios. In addition, xxl-job also lacks support for workflows, which to a certain extent limits its application in complex business process management. Summary of the invention

[0004] The purpose of this application is to propose a distributed task scheduling execution method, device, computer equipment and storage medium to solve the technical problems of the task scheduling algorithm in the prior art, such as single data support, poor adaptability, parallelism and scalability, and difficulty in adapting to the ever-changing system environment and task requirements.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a distributed task scheduling execution method, which adopts the following technical solution:

[0006] Obtaining task information from the dispatch center at regular intervals, determining a target service instance and a target executor group for executing the task according to the task information, and initializing the task instance according to the task information;

[0007] Obtaining execution status information of each executor instance in the target executor group, inputting the execution status information into a pre-trained executor optimal solution algorithm model for scoring calculation, and selecting the executor instance with the highest score as the first task execution instance;

[0008] Distribute the task instance to the first task execution instance through the target service instance, obtain a first execution result of the first task execution instance executing the task according to the task instance, and determine whether the first execution result indicates that the task is successfully executed;

[0009] If the first execution result is that the task execution is successful, the task status is updated through the target service instance; if the first execution result is that the task execution fails, the latest execution status information of other executor instances in the target executor group is re-acquired through the target service instance, wherein the other executor instances are executor instances other than the first task execution instance;

[0010] Inputting the latest execution state information into the executor optimal solution algorithm model for scoring calculation, and selecting other executor instances with the highest scores as the second task execution instances;

[0011] Distribute the task instance to the second task execution instance through the target service instance, obtain a second execution result of the second task execution instance executing the task according to the task instance, and determine whether the second execution result indicates successful task execution;

[0012] If the second execution result is that the task is executed successfully, the task status is updated through the target service instance. If the second execution result is that the task fails, the executor instance is continuously reselected through the target service instance until the task is executed successfully or the preset number of retries is reached.

[0013] In order to solve the above technical problems, the embodiment of the present application further provides a distributed task scheduling execution device, which adopts the following technical solution:

[0014] An acquisition module is used to periodically acquire task information from a scheduling center, determine a target service instance and a target executor group for executing the task according to the task information, and initialize the task instance according to the task information;

[0015] A first selection module is used to obtain execution status information of each executor instance in the target executor group, input the execution status information into a pre-trained executor optimal solution algorithm model for scoring calculation, and select the executor instance with the highest score as the first task execution instance;

[0016] A first task execution module is used to distribute the task instance to the first task execution instance through the target service instance, obtain a first execution result of the first task execution instance executing the task according to the task instance, and determine whether the first execution result is successful task execution;

[0017] A first execution result analysis module, configured to update the task status through the target service instance if the first execution result is that the task execution is successful, and to reacquire the latest execution status information of other executor instances in the target executor group through the target service instance if the first execution result is that the task execution fails, wherein the other executor instances are executor instances other than the first task execution instance;

[0018] A second selection module is used to input the latest execution state information into the executor optimal solution algorithm model for scoring calculation, and select other executor instances with the highest scores as the second task execution instance;

[0019] A second task execution module is used to distribute the task instance to the second task execution instance through the target service instance, obtain a second execution result of the second task execution instance executing the task according to the task instance, and determine whether the second execution result is a successful task execution;

[0020] The second execution result analysis module is used to update the task status through the target service instance if the second execution result is successful task execution, and to continue to reselect the executor instance through the target service instance if the second execution result is failed task execution until the task execution is successful or the preset number of retries is reached.

[0021] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0022] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned distributed task scheduling execution method when executing the computer-readable instructions.

[0023] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0024] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the distributed task scheduling execution method described above are implemented.

[0025] Compared with the prior art, this application has the following beneficial effects:

[0026] The present application provides a distributed task scheduling execution method, which obtains the target service instance and the target executor group corresponding to the task information, determines the first task execution instance according to the execution status information of each executor instance in the target executor group through a pre-trained executor optimal solution algorithm model, executes the task instance through the first task execution instance, and obtains a first execution result. When the first execution result is a task execution failure, the second task execution instance is re-acquired through the executor optimal solution algorithm model until the task execution is successful or a preset number of retries is reached; the present application can improve the concurrency of task execution by determining the target service instance for executing the task according to the task information, and secondly, by selecting the best executor instance to execute the task through the optimal solution algorithm model, the task can be dynamically assigned to the most suitable executor instance for stable execution of the task, thereby ensuring that all tasks are executed on time, efficiently and successfully. In addition, when the executor group includes multiple executor instances, if a problem occurs in one executor instance, the task can be dynamically transferred to other executor instances for continued execution, thereby improving the reliability and stability of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0029] Figure 2 is a flow chart of an embodiment of a distributed task scheduling execution method according to the present application;

[0030] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S202;

[0031] Figure 4 It is a schematic diagram of the architecture of a specific embodiment of the distributed task scheduling execution method according to the present application;

[0032] Figure 5 It is a structural diagram of an embodiment of a distributed task scheduling execution device according to the present application;

[0033] Figure 6 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0035] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0037] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0038] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0039] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0040] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0041] It should be noted that the distributed task scheduling execution method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the distributed task scheduling execution method device is generally set in the server / terminal device.

[0042] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0043] Continue to refer Figure 2 , shows a flow chart of an embodiment of a distributed task scheduling execution method according to the present application, comprising the following steps:

[0044] Step S201 , periodically obtain task information through the scheduling center, determine the target service instance and target executor group for executing the task according to the task information, and initialize the task instance according to the task information.

[0045] In this embodiment, a distributed, highly available, and horizontally scalable task scheduling cluster is constructed, which includes a scheduling center (i.e., a server) and an executor side. The scheduling center includes one to multiple service instances (i.e., Server nodes), which are responsible for managing the registration, status tracking, heartbeat, and task management and distribution, task instance management, etc. of all executors; the executor side includes multiple isolated executor groups, each executor group includes one to multiple executor instances (Worker), and an executor group is managed by the same Server of the scheduling center and is responsible for receiving the execution of tasks assigned by the scheduling center. The scheduling center and the executor side communicate data through remote communication, wherein the communication between the Worker and the Server includes Ping detection of the heartbeat, the sending of task information from the Server, the reporting of task execution information by the Worker, and the status monitoring of the Worker.

[0046] Before executing a task, the dispatch center and the executor are initialized respectively. The steps to initialize the dispatch center include:

[0047] Start the remote communication system and run the Actor container on the server side;

[0048] Create different types of Actor instances in the Actor container. Actor instances are used to process information reported by Workers, such as logs, task status, heartbeats, etc., receive tasks for scheduled task scheduling and distribution, and distribute tasks to managed Workers.

[0049] Initialize the server context information, which includes database connection information and other environment information, including email notification service, distributed unique service ID created by the server IP, etc.

[0050] Start the timer scheduling service thread pool to schedule and distribute tasks;

[0051] Initialization is complete and service starts.

[0052] The steps to initialize the executor side include:

[0053] Start the remote communication system and run the Actor container on the executor side;

[0054] Create different types of Actor instances in the Actor container. The Actor instance on the executor side is used to receive tasks from the Server, process related tasks within the Worker, and receive related information from the Worker and report it to the Server, such as logs, heartbeats, and task execution status.

[0055] Initialize Worker context information, including clearing Worker history-related data, such as local database, tasks, logs, etc., and initializing local storage and other information;

[0056] Request the corresponding Server and obtain the executor group ID to which the Worker belongs;

[0057] Execute Server service discovery. After the service discovery is successful, update the Worker related information.

[0058] Start the scheduled task thread pool and regularly report heartbeats, logs, and task information executed by this Worker instance to the Server;

[0059] Initialization is complete and service starts.

[0060] In this embodiment, the server application integrates the AKKA framework (RPC remote communication framework) to automatically initialize the ActorSystem when the service instance is started to manage the life cycle of all Actor containers.

[0061] The client creates a task and sets the number of failed retries corresponding to the task, that is, the preset number of retries. At the same time, it sets the executor group for task execution and submits the task to the most appropriate service instance of the scheduling center. After the scheduling center receives the task request from the client, it creates relevant task information in the background database to facilitate data storage and management, task tracking and monitoring, and data sharing and collaboration, to ensure data security and integrity.

[0062] In this embodiment, the scheduling center regularly schedules and distributes tasks, regularly obtains task information through the scheduling center, determines the target executor group for the corresponding task execution according to the task information, determines the corresponding target service instance according to the target executor group, and then starts initializing the task instance according to the task information.

[0063] In some optional implementations of this embodiment, the step of determining a target service instance and a target executor group for executing a task according to the task information includes:

[0064] Parse the task information to obtain task parameters, determine the target executor group according to the task parameters, and obtain the initial service instance associated with the target executor group;

[0065] Determine whether the initial service instance is available;

[0066] If the initial service instance is available, the initial service instance is determined as the target service instance;

[0067] If the initial service instance is unavailable, obtain the service status information of all available service instances;

[0068] Input the service status information into the trained server optimal solution algorithm model to obtain the status scores of all available service instances;

[0069] The available service instance with the highest status score is determined as the target service instance.

[0070] Among them, task parameters include task ID, task name, task type, execution logic, execution parameters, executor group ID, retry strategy (including preset retry times, retry interval, etc.), log records, and task priority information.

[0071] Parse the task parameters, obtain the target executor group associated with the task, obtain the initial service instance registered by the target executor group according to the executor group identifier of the target executor group, send a Ping signal for detecting the heartbeat to the initial service instance through the executor instance, if a response to the Ping signal is received from the initial service instance, indicating that the initial service instance is available, then the initial service instance is determined as the target service instance; if no response to the Ping signal is received from the initial service instance, indicating that the initial service instance is unavailable, then obtain the service status information of all currently available service instances, calculate and decide on the service status information through the trained server optimal solution algorithm model, obtain the status scores of all available service instances, and the available service instance with the highest status score is used as the target service instance, and re-register or bind the target executor group to the target service instance.

[0072] The service status information includes the current running data of each service instance, including the memory usage, CPU utilization, network IO, the current number of tasks to be dispatched, the current management executor group and the number of executors, etc.

[0073] In this embodiment, the initial service instance is also selected by using the server optimal solution algorithm model, and the server optimal solution algorithm model is trained by using a decision tree algorithm such as ID3, C5 or CART.

[0074] By judging whether the service instance is available to determine the target service instance, the success rate of task execution can be improved. Assigning tasks to available servers can avoid waste of resources, while ensuring that tasks are completed on time and efficiently, enhancing the stability and reliability of the entire system and improving user experience.

[0075] Step S202, obtaining the execution status information of each executor instance in the target executor group, inputting the execution status information into a pre-trained executor optimal solution algorithm model for score calculation, and selecting the executor instance with the highest score as the first task execution instance.

[0076] In this embodiment, the target executor group includes multiple executor instances, and the execution status information of each executor instance is obtained, wherein the execution status information includes the memory utilization, CPU utilization, network IO, the current number of tasks to be executed, the total GC (Garbage Collection) time / maximum GC time / average GC time of the executor's current service, etc.

[0077] The collected execution status information is input into the trained executor optimal solution algorithm model for calculation to obtain a first score corresponding to each executor instance. According to the first score, the executor instance with the highest score is determined as the first task execution instance to execute the corresponding task.

[0078] In some optional implementations, see Figure 3 As shown, before the step of inputting the execution state information into the pre-trained actuator optimal solution algorithm model, it also includes:

[0079] Step S301, obtaining a first operating status data set of each actuator instance, and extracting a first target variable and a first characteristic variable according to the first operating status data set;

[0080] Step S302, calculating a first contribution of each first characteristic variable to the first target variable by using the Gini index, and determining a first preset number of first characteristic variables that are most relevant to the first target variable as executor root nodes of the decision tree based on the first contribution;

[0081] Step S303, obtaining an actuator feature value of the actuator root node, and dividing the first operating state data set into a first preset number of first feature subsets according to the actuator feature value;

[0082] Step S304, using a recursive partitioning method, partitioning is performed according to the feature values ​​of each first feature subset, and the actuator instances are allocated to different child nodes until a leaf node is reached, thereby generating an actuator decision model;

[0083] Step S305, optimizing the actuator decision model through pruning operations to obtain the actuator optimal solution algorithm model.

[0084] In this embodiment, the historical operating status of each executor is regularly collected to form a first operating status data set. The historical operating status includes the memory usage rate, CPU usage rate, CPU load, network IO status, total GC time / maximum GC time / average GC time of the historical service, the number of successful / failed / blocked task executions, the longest task time / total time / average time, the number of task retries, etc.

[0085] The first operating status data set obtained is preprocessed to obtain the preprocessed first operating status data set, and the preprocessing includes data cleaning and normalization. Among them, data cleaning is to remove noise data such as outliers and missing values ​​to ensure the accuracy and completeness of the data. For example, the Python Pandas library can be used for data cleaning operations; normalization refers to normalizing performance indicator data of different dimensions and scaling the data to the same range, such as between 0 and 1. Common normalization methods include minimum-maximum normalization and Z-score normalization.

[0086] A decision tree is recursively constructed through the preprocessed first operating status data set. In this embodiment, a CART decision tree algorithm is used to generate multiple decision trees to form a decision model.

[0087] A first target variable and a plurality of first characteristic variables related to the business are extracted from the first operating status data set, and an importance measure of each first characteristic variable to the first target variable is calculated. The importance measure is calculated using the Gini index.

[0088] Among them, the Gini index of each first characteristic variable is calculated, and the Gini index calculation formula is as follows:

[0089]

[0090] Among them, P is the first operating status data set, k is the number of predicted categories of the first feature variable, and p i It is the sample proportion of different prediction categories under the same first feature variable.

[0091] The larger the Gini index is, the greater the uncertainty of the sample is. The smaller the Gini index is, the greater the correlation of the representative features is and the purer the classification is.

[0092] In this embodiment, the Gini index of each first characteristic variable is used as the first contribution of the first characteristic variable to the first target variable. The smaller the Gini index, the more relevant the first characteristic variable is to the first target variable. All the first characteristic variables are sorted from small to large according to the Gini index, and a first preset number of first characteristic variables in the front order are selected as the root nodes of the current decision tree. The feature value corresponding to each root node is obtained, and the first operating state data set is divided into first feature subsets consistent with the number of root nodes according to the feature value. Steps S302 and S303 are recursively executed for each first feature subset until a preset stop condition is met, such as the number of samples of a leaf node is less than a threshold, the depth of the tree reaches a limit, etc., to obtain an actuator decision model, wherein, at a leaf node, the probability distribution of the corresponding first target variable is determined according to the samples contained in the leaf node, and the probability distribution is used as a scoring result.

[0093] After the training is completed, the structural parameters of the actuator decision model are obtained, including the depth of the tree, the number of leaf node samples, etc.; the structural parameters are compared with the set pruning parameters to determine whether the structural complexity of the decision tree of the current actuator decision model exceeds the preset threshold. If it exceeds, a pruning operation is performed, wherein the set pruning parameters include the maximum depth parameter and the minimum sample number parameter of the decision tree. The maximum depth parameter is used to limit the depth of the tree and reduce the number of branches. The minimum sample number parameter is used to limit the minimum sample size of the leaf node and reduce overfitting; according to the set pruning parameters, the structure of the actuator decision model is pruned to obtain a simplified actuator decision model; the simplified actuator decision model is used to predict the test set data, and the accuracy, precision, recall rate and other indicators of the model on the test set are calculated to evaluate the generalization performance; if the generalization performance meets the requirements, the final decision tree model is output as the actuator optimal solution algorithm model, otherwise, the pruning parameters are adjusted and the above steps are repeated until the requirements are met.

[0094] By training and optimizing the executor decision model, the generalization ability of the model can be improved, and the accuracy and robustness of the model can be improved, which helps to accurately select executor instances with higher scores to perform tasks during task scheduling, thereby improving task execution efficiency and resource utilization.

[0095] Step S203: distribute the task instance to the first task execution instance through the target service instance, obtain a first execution result of the first task execution instance executing the task according to the task instance, and determine whether the first execution result indicates that the task is successfully executed.

[0096] When each service instance performs task scheduling, it periodically obtains all task information associated with all executor groups it holds from the database, then calculates the next task trigger time of each task and updates the next task trigger time in the database to the latest time, and pushes the task into the task scheduling queue for triggering. When the task scheduling thread of the service instance distributes tasks in the queue, it first queries the list of all executor workers of the executor group associated with the task, and calls the executor optimal solution algorithm model one by one to calculate the instantaneous state score of each executor instance, and finally selects the executor instance with the highest score to communicate and distributes the task to the executor instance for execution.

[0097] In this embodiment, the first task execution instance executes the task according to the task instance distributed by the target service instance, generates a first execution result, returns the first execution result to the target service instance, and the target service instance determines whether the task is executed successfully based on the first execution result.

[0098] Step S204: if the first execution result is that the task execution is successful, the task status is updated through the target service instance; if the first execution result is that the task execution fails, the latest execution status information of other executor instances in the target executor group is re-obtained through the target service instance, wherein the other executor instances are the executor instances except the first task execution instance.

[0099] In this embodiment, the task is completed when it is executed successfully, and the target service instance writes the successful execution status of the task to the task instance record in the database and marks the task execution completed; if the task execution fails, the target service instance re-acquires the latest execution status information of other executor instances in the target executor group according to the task retry strategy, where the other executor instances are the executor instances except the first task execution instance.

[0100] Step S205 , input the latest execution status information into the executor optimal solution algorithm model for score calculation, and select the other executor instance with the highest score as the second task execution instance.

[0101] The latest execution status information collected is input into the trained executor optimal solution algorithm model for calculation to obtain the second score corresponding to each other executor instance. According to the second score, the other executor instance with the highest score is determined as the second task execution instance to execute the corresponding task.

[0102] Among them, the latest execution status information includes the memory utilization, CPU utilization, network IO, number of tasks to be executed, total GC time / maximum GC time / average GC time of the virtual machine, etc.

[0103] Step S206: distribute the task instance to the second task execution instance through the target service instance, obtain a second execution result of the second task execution instance executing the task according to the task instance, and determine whether the second execution result indicates successful task execution.

[0104] In this embodiment, the second task execution instance executes the task according to the task instance distributed by the target service instance, generates a second execution result, returns the second execution result to the target service instance, and the target service instance determines whether the task is executed successfully based on the second execution result.

[0105] Step S207, if the second execution result is that the task is executed successfully, the task status is updated through the target service instance. If the second execution result is that the task fails, the executor instance is continuously reselected through the target service instance until the task is executed successfully or the preset number of retries is reached.

[0106] In this embodiment, if the second execution result is that the task execution is successful, the task execution is successful and the target service instance writes the successful execution status of the task to the task instance record in the database and marks the task execution as completed; if the second execution result is that the task execution fails, the target service instance re-acquires the latest execution status information of other executor instances remaining in the target executor group according to the task retry strategy, and repeats steps S205 to S207 until the task execution is successful or the preset number of retries is reached.

[0107] This application can be applied to the financial, Internet, and big data industries. By determining the target service instance for executing the task based on the task information, the concurrency of task execution can be improved. Secondly, by selecting the best executor instance to execute the task through the optimal solution algorithm model, the task can be dynamically allocated to the most suitable executor instance for stable execution, ensuring that all tasks are executed on time, efficiently, and successfully. In addition, when the executor group includes multiple executor instances, if a problem occurs in one executor instance, the task can be dynamically transferred to other executor instances for continued execution, thereby improving the reliability and stability of task execution.

[0108] In some optional implementations of this embodiment, before the step of inputting the service status information into the trained server optimal solution algorithm model, the following steps are also included:

[0109] Step A, obtaining a second operating status data set of each server, and extracting a second target variable and a second characteristic variable according to the second operating status data set;

[0110] Step B, calculating a second contribution of each second feature variable to the second target variable through information gain, and determining a second preset number of second feature variables most relevant to the second target variable as a server root node of the decision tree based on the second contribution;

[0111] Step C, obtaining a server characteristic value of the server root node, and dividing the second operating status data set into a second preset number of second characteristic subsets according to the server characteristic value;

[0112] Step D, using a recursive partitioning method, partitioning according to the feature values ​​of each second feature subset, assigning the server to different child nodes until a leaf node is reached, and generating a server decision model;

[0113] Step E: Optimize the server decision model through pruning operations to obtain the server optimal solution algorithm model.

[0114] In this embodiment, the historical operating status of each service instance is periodically collected to form a second operating status data set, wherein the historical status includes the memory usage, CPU usage, CPU load, network IO status, total GC time / maximum GC time / average GC time of the historical service, the number of distribution task success / failure / blocking times, the number of managed executor group instances, the number of managed executors, etc.

[0115] The acquired second operating state data set is preprocessed to obtain a preprocessed second operating state data set, wherein the preprocessing includes data cleaning and normalization. A decision tree is recursively constructed through the preprocessed second operating state data set. In this embodiment, a C5 decision tree algorithm is used to generate multiple decision trees to form a decision model.

[0116] A second target variable and a plurality of second characteristic variables related to the service are extracted from the second operating status data set, and an importance measure of each second characteristic variable to the second target variable is calculated, wherein the importance measure is calculated using information gain.

[0117] Among them, the Gini index of each second characteristic variable is calculated, and the information gain calculation formula is as follows:

[0118] Gain(second target variable)=Ent(D1)-Ent(D1|second target variable);

[0119] In the formula, Ent(D1|second target variable) represents the conditional information entropy of a second target variable. The calculation formula of conditional information entropy is as follows:

[0120]

[0121] In the formula, p j is the jth second target variable C j The probability of occurrence in all samples of the second target variable, Ent(D1|C j ) represents the second target variable C j The attribute information entropy of Where, represents the second target variable i in the second target variable C j The probability of occurrence in all samples.

[0122] It should be noted that, the greater the information gain of the second target variable, the higher the homogeneity of the second feature subsets divided according to the second target variable, which is more conducive to classification.

[0123] In this embodiment, the information gain of each second characteristic variable is used as the second contribution of the second characteristic variable to the second target variable. The larger the information gain, the more relevant the second characteristic variable is to the second target variable. All the first and second characteristic variables are sorted from large to small according to the information gain, and the second preset number of second characteristic variables in the front order are selected as the root node of the current decision tree. The feature value corresponding to each root node is obtained, and the second operating status data set is divided into second feature subsets consistent with the number of root nodes according to the feature value. Step B and step C are recursively executed for each second feature subset until the preset stop condition is met, such as the number of samples of the leaf node is less than the threshold, the depth of the tree reaches the limit, etc., to obtain the server decision model.

[0124] After the training is completed, the structural parameters of the server decision model are obtained, including the depth of the tree, the number of leaf node samples, etc.; the structure of the server decision model is pruned according to the structural parameters and the set pruning parameters to obtain a simplified server decision model; the simplified server decision model is used to predict the test set data, and the accuracy, precision, recall and other indicators of the model on the test set are calculated to evaluate the generalization performance; if the generalization performance meets the requirements, the final decision tree model is output as the server optimal solution algorithm model, otherwise, the pruning parameters are adjusted and the above steps are repeated until the requirements are met.

[0125] By training and optimizing the server decision model, the generalization ability of the model can be improved, and the accuracy and robustness of the model can be improved, which helps to accurately select the best service instance to manage and distribute tasks during task scheduling, thereby improving task distribution efficiency and resource utilization.

[0126] In some optional implementations, before the step of regularly acquiring task information through the dispatch center, the method further includes:

[0127] Start the service instance and executor instance;

[0128] Get the service instance corresponding to the executor instance and determine whether the service instance is available;

[0129] When a service instance is unavailable, a candidate service instance is obtained through the server optimal solution algorithm model;

[0130] Through the executor instance, service discovery is performed on the candidate service instance based on the distributed lock to determine whether the candidate service instance is available;

[0131] When the candidate service instance is available, the candidate service instance is used as the optimal service instance, and the executor instance is associated with the optimal service instance;

[0132] When the candidate service instance is unavailable, the service instance is re-acquired until the optimal service instance is obtained and the distributed lock is released.

[0133] In this embodiment, the executor instance is pre-registered in the Server and associated with the service instance. Before the task starts to execute, the service instance and the executor instance are started. After the service instance is started, the relevant instance information is registered in the database. When the executor instance is started, service discovery is performed. Specifically, after the executor instance is started, the associated service instance is queried, that is, the initial service instance. A Ping signal for detecting the heartbeat is sent to the initial service instance through the executor instance. If a response to the Ping signal from the initial service instance is received, it indicates that the initial service instance is available, and the initial service instance is determined as the target service instance; if no response to the Ping signal from the initial service instance is received, it indicates that the initial service instance is unavailable.

[0134] When the initial service instance is unavailable, the server list of the initial configuration is queried, and the best candidate service instance is selected through the server optimal solution algorithm model. The executor instance uses a distributed lock to discover the candidate service instance, and sends a Ping signal to the candidate service instance to detect the heartbeat. If the candidate service instance responds to the Ping signal, it means that the candidate service instance is available, then the candidate service instance is determined as the optimal service instance of the executor instance, and the executor instance is migrated from the unavailable service instance to the selected optimal service instance to continue to provide services; when the selected candidate service instance is unavailable, the server optimal solution algorithm model is continued to be used to judge and select in the server list until the optimal service instance is obtained and the distributed lock is released. If no available service instance is found in the server list, it means that the current executor instance is disconnected and error processing is performed. Among them, the distributed lock is a synchronization mechanism for controlling the simultaneous initiation of service discovery by multiple executor instances. The main purpose is to prevent multiple executor instances from selecting the same service instance at the same time.

[0135] At this point, before the task starts executing, all executor instances under an executor group are registered with the best available service instance.

[0136] By performing service discovery before task execution, system availability can be improved, faults can be discovered and handled in a timely manner, resource allocation can be optimized, and system scalability and stability can be improved, thereby ensuring the smooth and efficient execution of tasks and the overall performance of the system.

[0137] In some optional implementations of this embodiment, the step of performing service discovery on the candidate service instance according to the distributed lock through the executor instance to determine whether the candidate service instance is available includes:

[0138] Send a distributed lock request to the candidate service instance through the executor instance;

[0139] After receiving the locking success feedback from the candidate service instance, a response signal is sent to the candidate service instance through the executor instance;

[0140] Whether the candidate service instance is available is determined according to the response result of the response signal.

[0141] In this embodiment, if multiple executor instances simultaneously find that the previously registered service instances cannot communicate, they will simultaneously start the service discovery of the Server, and each executor instance will first query the service instance corresponding to the current executor instance from the database. At this time, an executor instance in the list of Worker instances to be service discovered acquires a distributed lock for service discovery, and other executor instances continue to wait and retry. The executor instance that obtains the distributed lock selects the service instance and updates the service instance information of the executor group to which it belongs. After the update, other executor instances will perform service discovery again and will find new available service instances.

[0142] Specifically, after obtaining the optimal service instance through the server optimal solution algorithm model, the executor instance sends a distributed lock lock request to the database. The database locks the corresponding optimal service instance according to the received lock request and returns the lock feedback result to the executor instance. If the lock feedback result is a successful lock, a ping signal is sent to the locked optimal service instance through the executor instance. If a response to the Ping signal is received from the optimal service instance, the service instance association information of the group (executor group) corresponding to this executor instance is updated to the new service instance. After the update is successful, the distributed lock is released and the service discovery of the next executor instance continues until all executor instances in the Worker instance list find the associated service instance.

[0143] By performing service instance locking during the service discovery process in the executor instance, it is possible to avoid duplicate discovery of service instances and ensure data consistency, thereby improving the reliability and stability of the entire distributed task scheduling framework.

[0144] In some optional implementations, after the step of reaching the preset number of retries, the method further includes:

[0145] Get the current running data of all current service instances;

[0146] Through the server optimal solution algorithm model, the current optimal service instance is selected as the target service instance according to the current running data.

[0147] In this embodiment, when the task reaches the preset number of retries of the retry strategy, service discovery can be re-executed, the current running data of all current service instances can be calculated through the server optimal solution algorithm model, and the current optimal service instance is selected as the target service instance, and steps S201 to S207 are repeated until the task is successfully executed.

[0148] By re-executing service discovery and obtaining new target service instances, resource allocation can be optimized, while improving system scalability and stability, ensuring smooth and efficient execution of tasks.

[0149] The following is a specific embodiment to more specifically illustrate the content of the present application and further elaborate on the present application, but these embodiments are by no means limiting the present application.

[0150] See also Figure 4 As shown, the cluster deployment of this embodiment includes three service instances and nine executor instances. The three service instances are Server1, Server2 and Server3, and every three executor instances form an executor group. For example, worker11, worker12, and worker13 constitute executor group 1 and are registered on Server1, worker21, worker22, and worker23 constitute executor group 2 and are registered on Server2, and worker31, worker32, and worker33 constitute executor group 3 and are registered on Server3.

[0151] In this embodiment, the above cluster is used to execute a SQL type data statistics task, including the following steps:

[0152] Step 1: The client creates a SQL statistics task, sets the preset number of retries for the task to 3 times, and specifies that the task is executed on executor group 1. The task is submitted to Server 1 associated with executor group 1. After receiving the task request, Server 1 creates relevant task information in the background database.

[0153] Step 2: Run the SQL statistics task and submit the task to the scheduling center.

[0154] Step 3: The dispatch center finds through task query that the task needs to be executed on executor group 1, starts to initialize the task instance, and queries executor group 1 to associate three instances of worker11, worker12, and worker13. According to the executor optimal solution algorithm model, select the best executor instance to execute the task. Assume that the worker12 instance is selected.

[0155] Step 4: worker12 starts to execute the task. If the task fails, the task status is submitted to server1 where worker12 is registered.

[0156] Step 5: When server1 finds that the SQL statistics task fails, it continues to select worker11 or woker13 to execute the task according to the task retry strategy and the executor optimal solution algorithm model. Assume that the task is executed on the worker13 instance at this time.

[0157] Step 6: After receiving the task from server1, worker13 starts to execute the SQL statistics task. Assuming that the task is successfully executed, worker13 reports the success status of the task to server1, which writes it to the task instance record in the database and marks the task execution completed.

[0158] Step 7: After the task is executed, when the client queries the task again, it returns that the task was successfully executed.

[0159] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0160] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0161] Further references Figure 5 , as a response to the above Figure 2 The present application provides an embodiment of a distributed task scheduling execution device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0162] like Figure 5 As shown, the distributed task scheduling execution device 500 described in this embodiment includes: an acquisition module 501, a first selection module 502, a first task execution module 503, a first execution result analysis module 504, a second selection module 505, a second task execution module 506 and a second execution result analysis module 507. Among them:

[0163] The acquisition module 501 is used to periodically acquire task information from the scheduling center, determine the target service instance and target executor group for executing the task according to the task information, and initialize the task instance according to the task information;

[0164] The first selection module 502 is used to obtain the execution status information of each executor instance in the target executor group, input the execution status information into the pre-trained executor optimal solution algorithm model for scoring calculation, and select the executor instance with the highest score as the first task execution instance;

[0165] The first task execution module 503 is used to distribute the task instance to the first task execution instance through the target service instance, obtain the first execution result of the first task execution instance executing the task according to the task instance, and determine whether the first execution result is successful task execution;

[0166] The first execution result analysis module 504 is used to update the task status through the target service instance if the first execution result is that the task execution is successful, and to re-acquire the latest execution status information of other executor instances in the target executor group through the target service instance if the first execution result is that the task execution fails, wherein the other executor instances are executor instances other than the first task execution instance;

[0167] The second selection module 505 is used to input the latest execution state information into the executor optimal solution algorithm model for scoring calculation, and select other executor instances with the highest scores as the second task execution instance;

[0168] The second task execution module 506 is used to distribute the task instance to the second task execution instance through the target service instance, and obtain the second execution result of the second task execution instance executing the task according to the task instance, and determine whether the second execution result is successful task execution;

[0169] The second execution result analysis module 507 is used to update the task status through the target service instance if the second execution result is successful task execution, and to continue to reselect the executor instance through the target service instance if the second execution result is failed task execution until the task execution is successful or the preset number of retries is reached.

[0170] Based on the above-mentioned distributed task scheduling execution device 500, by determining the target service instance for executing the task according to the task information, the concurrency of task execution can be improved. Secondly, by selecting the best executor instance to execute the task through the optimal solution algorithm model, the task can be dynamically allocated to the most suitable executor instance for stable execution, thereby ensuring that all tasks are executed on time, efficiently and successfully. In addition, in the case where the executor group includes multiple executor instances, if a problem occurs in one executor instance, the task can be dynamically transferred to other executor instances for continued execution, thereby improving the reliability and stability of task execution.

[0171] In some optional implementations of this embodiment, the acquisition module 501 includes:

[0172] A parsing submodule, configured to parse the task information to obtain task parameters, determine a target executor group according to the task parameters, and obtain an initial service instance associated with the target executor group;

[0173] A judgment submodule, used to judge whether the initial service instance is available;

[0174] A first determination submodule, configured to determine the initial service instance as a target service instance if the initial service instance is available;

[0175] An acquisition submodule, used for acquiring service status information of all available service instances if the initial service instance is unavailable;

[0176] A service selection submodule, used to input the service status information into a trained server optimal solution algorithm model to obtain status scores of all available service instances;

[0177] The second determining submodule is configured to determine the available service instance with the highest status score as the target service instance.

[0178] By judging whether the service instance is available to determine the target service instance, the success rate of task execution can be improved. Assigning tasks to available servers can avoid waste of resources, while ensuring that tasks are completed on time and efficiently, enhancing the stability and reliability of the entire system and improving user experience.

[0179] In some optional implementations, the distributed task scheduling execution device 500 further includes a first training module, which is used to:

[0180] Acquire a first operating status data set of each of the actuator instances, and extract a first target variable and a first characteristic variable according to the first operating status data set;

[0181] Calculate a first contribution of each of the first feature variables to the first target variable by using the Gini index, and determine a first preset number of the first feature variables that are most relevant to the first target variable as executor root nodes of a decision tree based on the first contribution;

[0182] Acquire an actuator feature value of the actuator root node, and divide the first operating status data set into a first preset number of first feature subsets according to the actuator feature value;

[0183] Using a recursive partitioning method, partitioning is performed according to the feature values ​​of each of the first feature subsets, and the actuator instances are assigned to different child nodes until a leaf node is reached, thereby generating an actuator decision model;

[0184] The actuator decision model is optimized through pruning operations to obtain the actuator optimal solution algorithm model.

[0185] By training and optimizing the executor decision model, the generalization ability of the model can be improved, and the accuracy and robustness of the model can be improved, which helps to accurately select executor instances with higher scores to perform tasks during task scheduling, thereby improving task execution efficiency and resource utilization.

[0186] In some optional implementations, the distributed task scheduling execution device 500 further includes a second training module, which is used to:

[0187] Acquire a second operating status data set of each of the servers, and extract a second target variable and a second characteristic variable according to the second operating status data set;

[0188] Calculate a second contribution of each second feature variable to the second target variable through information gain, and determine a second preset number of second feature variables most relevant to the second target variable as server root nodes of a decision tree based on the second contribution;

[0189] Obtaining a server characteristic value of the server root node, and dividing the second operating status data set into a second preset number of second characteristic subsets according to the server characteristic value;

[0190] Using a recursive partitioning method, partitioning is performed according to the feature values ​​of each of the second feature subsets, and the servers are assigned to different child nodes until a leaf node is reached, thereby generating a server decision model;

[0191] The server decision model is optimized through pruning operations to obtain the server optimal solution algorithm model.

[0192] By training and optimizing the server decision model, the generalization ability of the model can be improved, and the accuracy and robustness of the model can be improved, which helps to accurately select the best service instance to manage and distribute tasks during task scheduling, thereby improving task distribution efficiency and resource utilization.

[0193] In some optional implementations, the above-mentioned distributed task scheduling execution device 500 further includes a service discovery module, including:

[0194] The startup submodule is used to start the service instance and the executor instance;

[0195] A judgment submodule, used to obtain the service instance corresponding to the executor instance and judge whether the service instance is available;

[0196] A selection submodule, used for obtaining a candidate service instance through a server optimal solution algorithm model when the service instance is unavailable;

[0197] A service discovery submodule, configured to perform service discovery on the candidate service instance through the executor instance according to the distributed lock, and determine whether the candidate service instance is available;

[0198] an associating submodule, configured to use the candidate service instance as the optimal service instance when the candidate service instance is available, and associate the executor instance with the optimal service instance;

[0199] The reselection submodule is used to reacquire a service instance when the candidate service instance is unavailable, until the optimal service instance is acquired, and release the distributed lock.

[0200] By performing service discovery before task execution, system availability can be improved, faults can be discovered and handled in a timely manner, resource allocation can be optimized, and system scalability and stability can be improved, thereby ensuring the smooth and efficient execution of tasks and the overall performance of the system.

[0201] In some optional implementations of this embodiment, the service discovery submodule is further used to:

[0202] Sending a lock request for a distributed lock to the candidate service instance through the executor instance;

[0203] Receiving a locking success feedback from the candidate service instance, sending a response signal to the candidate service instance through the executor instance;

[0204] Whether the candidate service instance is available is determined according to a response result of the response signal.

[0205] By performing service instance locking during the service discovery process in the executor instance, it is possible to avoid duplicate discovery of service instances and ensure data consistency, thereby improving the reliability and stability of the entire distributed task scheduling framework.

[0206] In some optional implementations, the distributed task scheduling execution device 500 further includes a service reselection module, which is used to:

[0207] Get the current running data of all current service instances;

[0208] The server optimal solution algorithm model is used to select the current optimal service instance as the target service instance according to the current operation data.

[0209] By re-executing service discovery and obtaining new target service instances, resource allocation can be optimized, while improving system scalability and stability, ensuring smooth and efficient execution of tasks.

[0210] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0211] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 with a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0212] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0213] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 6. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as the program code of the distributed task scheduling execution method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0214] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the program code stored in the memory 61 or process data, such as running the program code of the distributed task scheduling execution method.

[0215] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0216] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the distributed task scheduling execution method as described above.

[0217] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0218] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A distributed task scheduling execution method, characterized in that: The steps include: Obtaining task information from the dispatch center at regular intervals, determining a target service instance and a target executor group for executing the task according to the task information, and initializing the task instance according to the task information; Obtaining execution status information of each executor instance in the target executor group, inputting the execution status information into a pre-trained executor optimal solution algorithm model for scoring calculation, and selecting the executor instance with the highest score as the first task execution instance; Distribute the task instance to the first task execution instance through the target service instance, obtain a first execution result of the first task execution instance executing the task according to the task instance, and determine whether the first execution result indicates that the task is successfully executed; If the first execution result is that the task execution is successful, the task status is updated through the target service instance; if the first execution result is that the task execution fails, the latest execution status information of other executor instances in the target executor group is re-acquired through the target service instance, wherein the other executor instances are executor instances other than the first task execution instance; Inputting the latest execution state information into the executor optimal solution algorithm model for scoring calculation, and selecting other executor instances with the highest scores as the second task execution instances; Distribute the task instance to the second task execution instance through the target service instance, obtain a second execution result of the second task execution instance executing the task according to the task instance, and determine whether the second execution result indicates successful task execution; If the second execution result is that the task is executed successfully, the task status is updated through the target service instance. If the second execution result is that the task fails, the executor instance is continuously reselected through the target service instance until the task is executed successfully or the preset number of retries is reached.

2. The distributed task scheduling execution method according to claim 1, characterized in that: The step of determining the target service instance and the target executor group for executing the task according to the task information comprises: Parsing the task information to obtain task parameters, determining a target executor group according to the task parameters, and acquiring an initial service instance associated with the target executor group; Determining whether the initial service instance is available; If the initial service instance is available, determining the initial service instance as the target service instance; If the initial service instance is unavailable, obtaining service status information of all available service instances; Inputting the service status information into a trained server optimal solution algorithm model to obtain status scores of all available service instances; The available service instance with the highest status score is determined as the target service instance.

3. The distributed task scheduling execution method according to claim 1, characterized in that: Before the step of inputting the execution state information into the pre-trained actuator optimal solution algorithm model, the method further includes: Acquire a first operating status data set of each of the actuator instances, and extract a first target variable and a first characteristic variable according to the first operating status data set; Calculate a first contribution of each of the first feature variables to the first target variable by using the Gini index, and determine a first preset number of the first feature variables that are most relevant to the first target variable as executor root nodes of a decision tree based on the first contribution; Acquire an actuator feature value of the actuator root node, and divide the first operating status data set into a first preset number of first feature subsets according to the actuator feature value; Using a recursive partitioning method, partitioning is performed according to the feature values ​​of each of the first feature subsets, and the actuator instances are assigned to different child nodes until a leaf node is reached, thereby generating an actuator decision model; The actuator decision model is optimized through pruning operations to obtain the actuator optimal solution algorithm model.

4. The distributed task scheduling execution method according to claim 2, characterized in that: Before the step of inputting the service status information into the trained server optimal solution algorithm model, the method further includes: Acquire a second operating status data set of each of the servers, and extract a second target variable and a second characteristic variable according to the second operating status data set; Calculate a second contribution of each second feature variable to the second target variable through information gain, and determine a second preset number of second feature variables most relevant to the second target variable as server root nodes of a decision tree based on the second contribution; Obtaining a server characteristic value of the server root node, and dividing the second operating status data set into a second preset number of second characteristic subsets according to the server characteristic value; Using a recursive partitioning method, partitioning is performed according to the feature values ​​of each of the second feature subsets, and the servers are assigned to different child nodes until a leaf node is reached, thereby generating a server decision model; The server decision model is optimized through pruning operations to obtain the server optimal solution algorithm model.

5. The distributed task scheduling execution method according to claim 1, characterized in that: Before the step of obtaining task information through the dispatch center at a fixed time, the method further includes: Start the service instance and executor instance; Obtain the service instance corresponding to the executor instance, and determine whether the service instance is available; When the service instance is unavailable, a candidate service instance is obtained through a server optimal solution algorithm model; Performing service discovery on the candidate service instance according to the distributed lock through the executor instance to determine whether the candidate service instance is available; When the candidate service instance is available, the candidate service instance is used as the optimal service instance, and the executor instance is associated with the optimal service instance; When the candidate service instance is unavailable, the service instance is re-acquired until the optimal service instance is acquired and the distributed lock is released.

6. The distributed task scheduling execution method according to claim 5, characterized in that: The step of performing service discovery on the candidate service instance according to the distributed lock through the executor instance to determine whether the candidate service instance is available includes: Sending a lock request for a distributed lock to the candidate service instance through the executor instance; Receiving a locking success feedback from the candidate service instance, sending a response signal to the candidate service instance through the executor instance; Whether the candidate service instance is available is determined according to a response result of the response signal.

7. The distributed task scheduling execution method according to any one of claims 1 to 6, characterized in that: After the step of reaching the preset number of retries, the method further includes: Get the current running data of all current service instances; The server optimal solution algorithm model is used to select the current optimal service instance as the target service instance according to the current operation data.

8. A distributed task scheduling execution device, characterized in that: include: An acquisition module is used to periodically acquire task information from a scheduling center, determine a target service instance and a target executor group for executing the task according to the task information, and initialize the task instance according to the task information; A first selection module is used to obtain execution status information of each executor instance in the target executor group, input the execution status information into a pre-trained executor optimal solution algorithm model for scoring calculation, and select the executor instance with the highest score as the first task execution instance; A first task execution module is used to distribute the task instance to the first task execution instance through the target service instance, obtain a first execution result of the first task execution instance executing the task according to the task instance, and determine whether the first execution result is successful task execution; A first execution result analysis module, configured to update the task status through the target service instance if the first execution result is that the task execution is successful, and to reacquire the latest execution status information of other executor instances in the target executor group through the target service instance if the first execution result is that the task execution fails, wherein the other executor instances are executor instances other than the first task execution instance; A second selection module is used to input the latest execution state information into the executor optimal solution algorithm model for scoring calculation, and select other executor instances with the highest scores as the second task execution instance; A second task execution module is used to distribute the task instance to the second task execution instance through the target service instance, obtain a second execution result of the second task execution instance executing the task according to the task instance, and determine whether the second execution result is a successful task execution; The second execution result analysis module is used to update the task status through the target service instance if the second execution result is successful task execution, and to continue to reselect the executor instance through the target service instance if the second execution result is failed task execution until the task execution is successful or the preset number of retries is reached.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the distributed task scheduling execution method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the distributed task scheduling execution method according to any one of claims 1 to 7 are implemented.

Images