Task scheduling method and system, medium, equipment and program product

Abstract

The invention provides a task scheduling method and system, a medium, equipment and a program product, and relates to the field of task scheduling, and the method comprises the steps: obtaining a task type and parameters through analyzing a user task request, and matching a proper artificial intelligence machine end or model. When it is ensured that the machine end is idle or the model load is lower than a threshold value, a task request is sent and the state is updated to be busy. According to the method, task requirements can be accurately identified, optimal matching of tasks and resources is realized, and the execution efficiency and the system utilization rate are improved. Meanwhile, load states of a machine end and the model are considered, overload is avoided, efficient completion of tasks is ensured, and system stability and reliability are improved. By tracking the resource use condition in real time, the scheduling strategy is optimized, repeated task allocation is avoided, efficient cyclic utilization of resources is achieved, and therefore the system operation efficiency and the user experience are improved.

Description

Technical Field

[0001] This application relates to the field of task scheduling, and in particular to a task scheduling method, system, medium, device and program product. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, enterprises are deploying various types of AI models, including but not limited to text generation, image generation, and translation, to meet diverse business needs. These AI services may be developed internally and communicate in real time via the WebSocket protocol, or they may be third-party services provided through HTTP / RESTful API interfaces. Faced with AI cluster management in this heterogeneous environment, existing task scheduling systems suffer from scalability and coupling issues, insufficient system robustness, and complex communication management.

[0003] Current task scheduling solutions are often tightly bound to the communication protocols and data formats used by specific AI nodes. This means that when new or different types of AI services are introduced, significant modifications must be made to the core scheduling logic, extending the development cycle and increasing maintenance costs. Furthermore, there are deficiencies in handling task submission, allocation, execution, and completion state transitions. For example, untimely state updates or a lack of effective countermeasures for anomalies directly impact the stable operation and reliability of the entire system. In addition, the system needs to simultaneously support requests from the user interface frontend and various other types of backend AI services, which may be based on different communication protocols (such as WebSocket or HTTP).

[0004] Therefore, how to efficiently and dynamically allocate resources to achieve task scheduling is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a task scheduling method, system, computer-readable storage medium, electronic device, and computer program product to solve the problems of poor scalability, low resource utilization, and insufficient system robustness in existing artificial intelligence task scheduling systems.

[0006] To address the aforementioned technical problems, this application provides a task scheduling method, the specific technical solution of which is as follows:

[0007] Obtain user task requests;

[0008] Parse the task type and task parameters of the user's task request;

[0009] All objects in the AI ​​machine group and AI model group that are adapted to the task type are selected as candidate nodes. The network transmission latency and inference latency of each candidate node are calculated according to the task parameters. The node priority score of each candidate node is calculated based on the network transmission latency, the inference latency and their respective latency weights. The target AI machine or target AI model is determined based on the node priority score.

[0010] When the AI ​​machine is idle, the user task request is sent to the AI ​​machine via the application layer communication protocol; or when the load of the target AI model is below a set threshold, the user task request is transmitted over the network to the application programming interface corresponding to the target AI model, and the target AI machine or the target AI model is updated to a busy state.

[0011] Optional, also includes:

[0012] Receive machine model registration information sent by the AI ​​machine terminal through the application layer communication protocol;

[0013] Record the machine information corresponding to the machine model registration information, and assign a unique identifier to the machine information; the unique identifier is used to maintain a heartbeat connection with the artificial intelligence machine terminal.

[0014] Optionally, if the user task request fails to be sent, the method further includes:

[0015] Record the number of retries and resend the task until the number of retries reaches the retry limit, and record the user task request that failed to schedule the task.

[0016] Optional, also includes:

[0017] The processing time of the user task request is periodically detected;

[0018] Update the task status of the user task request according to the processing time;

[0019] The task status is then fed back to the task initiator.

[0020] Optionally, invoking a predefined scheduling algorithm to determine the appropriate target AI machine or target AI model includes:

[0021] Calculate the task weight of the user channel corresponding to the user task request based on the task type and the task parameters;

[0022] The user channels are weighted hierarchically to obtain the task feature vector;

[0023] Calculate the similarity between the artificial intelligence features corresponding to each artificial intelligence model and each artificial intelligence machine and the task feature vector;

[0024] Based on the aforementioned similarity, suitable target AI machine tools or target AI models are selected.

[0025] Optionally, when selecting suitable target AI machine tools or target AI models based on the aforementioned similarities, the method further includes:

[0026] Determine the initial object;

[0027] Based on the initial object, calculate the equivalent channel gain of each of the artificial intelligence models and each of the artificial intelligence machine terminals;

[0028] Target clusters are selected based on the equivalent channel gain.

[0029] Within the target cluster, suitable target AI machine tools or target AI models are selected based on the task type and the task parameters.

[0030] This application also provides a task scheduling system, including:

[0031] The communication layer contains the communication protocol between the coordinator service layer and the processor layer;

[0032] The coordinator service layer is used to select all objects in the AI ​​machine group and AI model group that are adapted to the task type as candidate nodes, calculate the network transmission latency and inference latency of each candidate node according to the task parameters, calculate the node priority score of each candidate node based on the network transmission latency, the inference latency and their respective latency weights, and determine the target AI machine or target AI model based on the node priority score.

[0033] The processor layer is configured to send the user task request to the AI ​​machine terminal via an application layer communication protocol when the AI ​​machine terminal is idle, or to transmit the user task request over the network to the application programming interface corresponding to the target AI model when the load of the target AI model is below a set threshold, and update the target AI machine terminal or the target AI model to a busy state.

[0034] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0035] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described above when it invokes the computer program in the memory.

[0036] This application also provides a computer program product, including a computer program that, when executed, implements the steps of the method described above.

[0037] This application provides a task scheduling method, comprising: acquiring a user task request; parsing the task type and task parameters of the user task request; selecting all objects in an AI machine client group and an AI model group adapted to the task type as candidate nodes; calculating the network transmission latency and inference latency of each candidate node according to the task parameters; calculating a node priority score for each candidate node based on the network transmission latency, the inference latency, and their respective latency weights; and determining a target AI machine client or a target AI model based on the node priority score; when the AI ​​machine client is idle, sending the user task request to the AI ​​machine client through an application layer communication protocol, or when the load of the target AI model is lower than a set threshold, transmitting the user task request over the network to the application programming interface corresponding to the target AI model, and updating the target AI machine client or the target AI model to a busy state.

[0038] This application, through detailed analysis of user task requests, accurately obtains task types and parameters, precisely identifying the specific requirements of each task and providing a reliable basis for subsequent scheduling decisions. Based on the analyzed task types and parameters, and considering network transmission latency and inference latency, it determines the appropriate target AI machine or target AI model, fully taking into account the different requirements of various tasks for computing resources and model performance, achieving optimal matching between tasks and resources. This not only improves task execution efficiency but also ensures maximum utilization of the overall system. Secondly, when sending user task requests to the target device or model, this application considers the idle state of the AI ​​machine and the load of the target AI model. Task requests are only sent when the AI ​​machine is idle, avoiding task execution delays or failures due to machine overload; similarly, network transmission is only performed when the load of the target AI model is below a set threshold, ensuring the model has sufficient processing power to complete tasks efficiently. This effectively balances resource usage and task execution efficiency, improving the stability and reliability of the system. This application updates the target AI machine or target AI model to a busy state, enabling real-time tracking of resource usage. This provides accurate reference information for subsequent task scheduling, helps to further optimize scheduling strategies, avoids duplicate assignment of tasks to already busy resources, thereby achieving efficient resource recycling and improving the overall system's operating efficiency and user experience.

[0039] This application also provides a task scheduling system, a computer-readable storage medium, an electronic device, and a computer program product, which have the above-mentioned beneficial effects, and will not be repeated here. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0041] Figure 1 A flowchart illustrating a task scheduling method provided in an embodiment of this application;

[0042] Figure 2 This is a schematic diagram of a task scheduling system structure provided in an embodiment of this application;

[0043] Figure 3 This is a communication timing diagram provided for an embodiment of this application;

[0044] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] See Figure 1 , Figure 1 A flowchart of a task scheduling method provided in an embodiment of this application is included, the method comprising:

[0047] S101: Obtain user task request;

[0048] S102: Parse the task type and task parameters of the user task request;

[0049] S103: Select all objects in the AI ​​machine group and AI model group that are adapted to the task type as candidate nodes, calculate the network transmission latency and inference latency of each candidate node according to the task parameters, calculate the node priority score of each candidate node based on the network transmission latency, the inference latency and their respective latency weights, and determine the target AI machine or target AI model based on the node priority score.

[0050] S104: When the AI ​​machine is idle, the user task request is sent to the AI ​​machine via the application layer communication protocol, or when the load of the target AI model is lower than a set threshold, the user task request is transmitted over the network to the application interface corresponding to the target AI model, and the target AI machine or the target AI model is updated to a busy state.

[0051] In practical applications, user task requests can be submitted to the system in various ways. For example, in a web-based artificial intelligence service platform, users can submit tasks through a web interface, mobile application, or dedicated client software. User task requests may contain various forms of data, such as text, images, audio, or video, as well as specific requirements for the task, such as task priority and expected completion time. By providing multiple task submission methods, the system can meet the needs of different users in different scenarios and can broadly receive task requests from various user groups.

[0052] Upon receiving a user task request, the request is first parsed. There are no specific limitations on how the request is parsed; various technical means can be used to identify the task type and extract task parameters.

[0053] Task type identification can be based on the data format contained in the task request, the task category specified by the user, or semantic analysis of the request content. For example, if the user uploads an image file and specifies it as an image recognition task in the request, the task type can be directly identified as image recognition. If the user does not explicitly specify the task type, preliminary analysis of the image content can be performed to determine whether it is a specific type of image recognition task, such as object recognition, scene classification, or person recognition.

[0054] Extracting task parameters requires obtaining various information related to task execution from the user's task request. Task parameters may include, but are not limited to, the size of the input data, accuracy requirements, and processing time constraints. For example, in a text translation task, task parameters might include the number of words in the source text, the target language, and the required translation style (e.g., formal or conversational). In practical applications, task parameters can be matched and extracted using predefined parameter templates, or natural language processing techniques can be used to understand the user's description of task parameters in the request.

[0055] Subsequently, all objects in the AI ​​machine group and AI model group that are adapted to the task type are selected as candidate nodes. The network transmission latency and inference latency of each candidate node are calculated according to the task parameters. The node priority score of each candidate node is calculated based on the network transmission latency, inference latency and their respective latency weights. The target AI machine or target AI model is determined based on the node priority score.

[0056] The AI ​​machine-end group comprises various machine-end devices with different configurations, performance levels, and network connectivity. These AI machine-ends may be geographically distributed, possessing different hardware architectures (e.g., variations in CPU, GPU, TPU combinations), memory capacity, storage capabilities, and other hardware characteristics. Furthermore, their network access methods (wired, wireless, etc.) and network bandwidth also differ. This embodiment assumes that a comprehensive AI machine-end information database can be established before this step. This database records detailed hardware configuration parameters, network connectivity parameters (e.g., network access type, average bandwidth, network latency range), and current load status (e.g., CPU utilization, memory usage) for each machine-end. When determining candidate nodes, based on the hardware requirements of the task type (e.g., for image recognition tasks, machines with high-performance GPUs may be needed; for natural language processing tasks, the multi-threaded processing capabilities of the CPU may be more important) and network connectivity requirements (e.g., for large task data volumes, machines with higher network bandwidth and lower latency are preferred), machine-ends meeting the basic requirements are selected from the database as candidate objects. This process can be automated by writing specific filtering algorithms based on preset hardware and network parameter thresholds.

[0057] Artificial intelligence model groups can also encompass a variety of different types of models, including but not limited to deep learning models (such as convolutional neural networks, recurrent neural networks, and Transformer architectures), traditional machine learning models (such as support vector machines, decision trees, and random forests), and hybrid models. Each model has its unique structure, parameter size, training dataset characteristics, and inference performance features. Different models will handle the same task in different ways and with different efficiencies, and the model's deployment environment (such as whether model quantization has been performed or whether a specific inference framework has been used for optimization) will also affect its actual inference latency.

[0058] Similarly, this embodiment assumes that a comprehensive AI model information database can be pre-built, recording each model's type, structural parameters (such as number of layers, number of neurons, activation function type, etc.), parameter size (model size, measured by the number of parameters or storage usage), training dataset information (data type, data size, data distribution characteristics, etc.), inference performance metrics (such as average inference time and inference accuracy under specific hardware configurations, etc.), and deployment optimization status (whether it has been quantized, whether GPU acceleration has been used, etc.). Based on the specific requirements of the task type for the model (for example, for high-precision image classification tasks, it may be necessary to select a deep learning model trained on a large amount of high-quality image data and with a relatively complex structure; for speech recognition tasks with high real-time requirements, it may be more inclined to select a model that has been quantized and optimized and has a fast inference speed), models that match the task are selected as candidate objects from the model information database. The selection process can call automated algorithms to complete the process according to preset model performance and task-related parameter thresholds.

[0059] Network transmission latency refers to the time required for data to travel from the sender to the receiver, that is, the time required to forward a user task request to the target AI machine or target AI model. Network transmission latency is affected by various factors, including network bandwidth, network delay (composed of network device processing delay, signal transmission delay, etc.), network congestion, and packet size. In a network environment, data is typically transmitted in the form of packets, each packet traversing a different routing path within the network; therefore, network transmission latency is dynamic and uncertain.

[0060] For candidate AI machine tools and model groups, various network transmission latency measurement methods can be employed. One feasible implementation uses network latency testing tools (such as the ping command) to send a certain number of test data packets to the target node, record the time from packet transmission to return, calculate the round-trip time (RTT), and then take the average as an approximation of the network transmission latency. Furthermore, network performance monitoring tools can be used to analyze the latency of each link in the network path in more detail, including router processing latency and link transmission latency, thereby providing more accurate network transmission latency measurement results. In actual measurement, to reduce the influence of random factors, multiple measurements can be taken and the average value calculated. Measurements can also be performed at different time periods to reflect the dynamic changes in network transmission latency.

[0061] In addition to directly measuring network transmission latency, it is also possible to estimate network transmission latency based on known network parameters and models. Specifically, a network transmission latency estimation model can be established based on factors such as network topology, bandwidth of each link, latency characteristics, and packet size. For example, by utilizing the relationship between network bandwidth and packet size, combined with the statistical characteristics of network latency, a rough range of network transmission latency can be calculated using a certain algorithm. Furthermore, historical network transmission latency data can be referenced, combined with the current network load (e.g., by obtaining the current network traffic load through network traffic monitoring tools), to dynamically estimate network transmission latency. During the estimation process, a certain error range can be introduced to account for uncertainties in the network environment.

[0062] Inference latency refers to the time required from inputting a user task request to the target AI machine or model to outputting a result. It primarily depends on the model's complexity (e.g., the number of layers, neurons, and computational complexity), the model's deployment environment (e.g., whether model optimization has been performed, and whether suitable hardware acceleration devices are used), and the characteristics of the input data (e.g., data size and format). Different models may have significantly different inference latencies under the same hardware environment; even the same model may have different inference latencies under different hardware configurations.

[0063] For candidate AI models, inference latency can be measured in their deployed hardware environment using a test dataset similar to that of the real-world task. Specifically, this involves recording the time interval from inputting test data into the model to the model's output, taking multiple measurements, and averaging the results as the model's inference latency in that hardware environment. To ensure the accuracy of the measurement results, the test dataset should be representative, covering various data characteristics that might occur in real-world tasks. Simultaneously, it's crucial to ensure that other loads on the hardware are low during the measurement process to avoid interference from other tasks with the inference latency measurement results.

[0064] Latency weights quantify the relative importance of network transmission latency and inference latency during task execution. Different task types exhibit varying sensitivities to network transmission latency and inference latency; therefore, it is necessary to determine appropriate latency weights based on the specific characteristics of each task to more reasonably balance the impact of network transmission latency and inference latency when comprehensively evaluating candidate nodes. In practical applications, task types can be analyzed, and latency weights can be determined based on factors such as real-time requirements, data transmission volume, and task complexity. For example, for tasks with extremely high real-time requirements (such as obstacle detection in autonomous driving), network transmission latency is more critical and can therefore be assigned a higher weight. Conversely, for tasks with relatively lower real-time requirements but higher data processing complexity (such as offline analysis of large-scale data), inference latency may be more important, and its weight can be appropriately increased. The determination of latency weights can be accomplished through a combination of methods, including historical task data statistical analysis and user-defined approaches. By analyzing the impact of network transmission latency and inference latency on task execution time in historical task data, latency weights can be quantitatively determined. At the same time, users can also customize latency weights according to specific application scenarios and needs to meet the personalized requirements of different users.

[0065] Node priority scoring is a quantitative indicator that comprehensively evaluates the performance of candidate nodes based on network transmission latency, inference latency, and corresponding latency weights. It integrates two different performance indicators, network transmission latency and inference latency, and reflects the overall performance of candidate nodes under specific tasks by adjusting the latency weights.

[0066] When calculating node priority scores, a comprehensive calculation method can be used to obtain the node priority score based on the obtained network transmission delay and inference delay values, combined with determined delay weights. The calculation method can be a simple weighted summation or a more complex nonlinear combination, depending on the characteristics of the task and the required accuracy of the candidate node performance evaluation. For example, in a simple weighted summation, the network transmission delay is multiplied by its corresponding weight, and the inference delay is multiplied by its corresponding weight. Then, the two weighted delay values ​​are added together to obtain the node priority score. In a more complex nonlinear combination, an adjustment factor can be introduced. Based on the relationship between network transmission delay and inference delay, as well as the specific requirements of the task, the two delay values ​​are nonlinearly transformed before comprehensive calculation to more accurately reflect the overall performance of the candidate node.

[0067] Furthermore, based on the above, if a user task request fails to be sent, the number of retries can be recorded, and the task can be resent until the retries reach the maximum limit. The user task request that failed to schedule can then be recorded. At this point, attempts can be made to reschedule the failed user task request, or manual intervention can be implemented.

[0068] In step S104, after determining the target AI machine or model to be adapted, the system needs to send the user task request to the corresponding execution resource and update the status.

[0069] In the specific application of this step, the idle status of each AI machine is continuously monitored. When a machine is idle, the user task request is sent to that machine via an application layer communication protocol. The application layer communication protocol can be a protocol specifically designed for AI task scheduling, or a general network communication protocol, such as real-time communication based on WebSocket, or HTTP (Hypertext Transfer Protocol) or TCP / IP (Transmission Control Protocol / Internet Protocol).

[0070] While sending task requests, the status of the AI ​​machine is updated to "busy". This status update can be achieved by recording the machine's current task status in the system's internal resource management database. When other task requests arrive, the scheduling algorithm can make appropriate scheduling decisions based on the latest status information. By sending task requests when the AI ​​machine is idle and updating its status promptly, it is ensured that the machine can efficiently utilize idle time to process tasks, avoiding resource waste and guaranteeing the accuracy of scheduling decisions.

[0071] When scheduling AI models, a load threshold can be set to determine whether the target AI model is suitable for receiving new task requests. When the current load of the target AI model is below this threshold, the user task request can be transmitted over the network to the model's corresponding Application Programming Interface (API). The API is the interface through which the model interacts with external systems, allowing the system to send task requests to the model in a specific format and receive the model's processing results. Similar to the AI ​​machine side, after the task request is sent, the target AI model's status is updated to busy. Status updates can be accomplished by recording the model's current task load and processing status. By sending task requests when the target AI model's load is low and updating its status, model overload can be effectively avoided, ensuring that the model can run efficiently within a reasonable load range. This also facilitates real-time monitoring and management of model usage.

[0072] Furthermore, resources can be released promptly after the user task corresponding to the user task request is completed. If a local AI model or AI machine is called, the application status of the local AI model or AI machine can be updated promptly. If a third-party model is scheduled, the concurrency of the third-party API can be released. If a third-party AI model call fails, the call limit can be returned.

[0073] This application embodiment, through detailed parsing of user task requests, accurately obtains task types and parameters, precisely identifying the specific requirements of each task and providing a reliable basis for subsequent scheduling decisions. Based on the parsed task types and parameters, a pre-defined scheduling algorithm is invoked to determine the appropriate target AI machine or target AI model. This fully considers the different requirements of various tasks for computing resources and model performance, achieving optimal matching between tasks and resources. This not only improves task execution efficiency but also ensures maximum utilization of the overall system. Secondly, when sending user task requests to the target device or model, this application considers the idle state of the AI ​​machine and the load of the target AI model. Task requests are only sent when the AI ​​machine is idle, avoiding task execution delays or failures due to machine overload. Similarly, network transmission is only performed when the load of the target AI model is below a set threshold, ensuring the model has sufficient processing power to efficiently complete the task. This effectively balances resource usage and task execution efficiency, improving system stability and reliability. Furthermore, by updating the target AI machine or target AI model to a busy state, this application can track resource usage in real time, providing accurate reference information for subsequent task scheduling. This helps to further optimize scheduling strategies, avoid repeatedly assigning tasks to already busy resources, thereby achieving efficient resource recycling and improving the overall system's operating efficiency and user experience.

[0074] Based on the above embodiments, before executing task scheduling, machine model registration information sent by the AI ​​machine terminal via an application layer communication protocol can be received. At this time, machine information corresponding to the machine model registration information can be recorded, and a unique identifier can be assigned to the machine information. The unique identifier is used to maintain a heartbeat connection with the AI ​​machine terminal.

[0075] For example, when the AI ​​machine starts up, it registers its machine model information with the coordinator service layer via WebSocket. The coordinator service layer records the machine information and assigns a unique identifier. After receiving the assigned unique identifier, the AI ​​machine periodically sends heartbeat messages to update its status (busy or idle).

[0076] In addition, in one feasible implementation, the processing time of the user task request can be periodically detected, and the task status of the user task request can be updated according to the processing time, thereby feeding back the task status to the task initiator.

[0077] Upon receiving a user task request, an initial timestamp is generated for the task. At each key node in the task processing (such as when the task is assigned to a specific processing module, or when the processing module begins actual task processing), a corresponding timestamp is recorded. By calculating the difference between subsequent timestamps and the initial timestamp, the cumulative processing time from receipt to the current node can be obtained, enabling precise tracking of the time consumed by the task at different processing stages.

[0078] A fixed detection cycle can be set (the cycle length can be flexibly adjusted according to task type and business needs). At the beginning of each cycle, all user task requests currently in the processing state are scanned. For each task, its processing progress since the last detection time is queried, including the executed operation steps, the number of completed subtasks, etc., and the processing time of this stage is estimated accordingly. The processing time within the current detection cycle is added to the total processing time previously recorded for the task, thereby achieving continuous tracking of task processing time.

[0079] Multiple processing time thresholds are pre-set based on factors such as the average processing time of various tasks and business importance. When the processing time of a user task request is detected to reach a certain threshold, the task status is updated to the corresponding level.

[0080] In practical applications, task status can be dynamically adjusted based on real-time detected trends in task processing time and historical data on similar tasks. If the processing time of a task increases significantly faster than the average for similar tasks and lasts for a considerable period, the task status can be updated to "potentially delayed" in advance, even if a preset threshold is not reached, to provide early warning and inform the task initiator of potential risks. Conversely, for tasks with gradually decreasing processing time and smooth progress, the task status can be updated from "processing" to "progressing well," providing the task initiator with more accurate task progress information.

[0081] Based on the above embodiments, as a preferred embodiment, when matching and adapting the target AI machine or target AI model based on a set scheduling algorithm, the following steps may be included:

[0082] Step 1: Calculate the task weight of the user channel corresponding to the user task request based on the task type and the task parameters;

[0083] The second step is to perform hierarchical weighting on the user channels to obtain the task feature vector;

[0084] The third step is to calculate the similarity between the artificial intelligence features corresponding to each artificial intelligence model and each artificial intelligence machine and the task feature vector.

[0085] Step 4: Based on the similarity scores, select suitable target AI machine tools or target AI models.

[0086] First, user task requests are categorized, for example, into different types such as image recognition, natural language processing, and data analysis. Each task type is assigned a base weight based on factors such as its complexity, resource requirements, and priority. For example, image recognition tasks might be assigned a higher base weight due to their higher computational resource requirements, while simple text proofreading tasks would be assigned a lower base weight.

[0087] Secondly, analyze the specific parameters of the task, such as its size (data volume), urgency (priority), and complexity. Adjust the base weights based on these parameters. For example, increase the weight appropriately for tasks with large data volumes; increase the weight for urgent tasks to prioritize their processing.

[0088] In the second step, a channel weight is assigned to each user channel based on its characteristics (such as network bandwidth, stability, and user history). For example, a channel with higher network bandwidth may be more suitable for handling tasks with large amounts of data, and therefore should be assigned a higher weight. This weighted summation or product of the task type weight and the user channel weight can then be used to reflect the importance and resource requirements of the task on a specific user channel.

[0089] User channels are categorized into different levels based on performance metrics (such as response time, throughput, and error rate). For example, channels can be divided into three levels: high, medium, and low, each corresponding to a different performance level. A corresponding weighting coefficient is assigned to each level of user channel. Higher priority channels are assigned higher weighting coefficients, and lower priority channels are assigned lower weighting coefficients. Then, the task weight of each user channel is multiplied by its corresponding weighting coefficient to obtain the weighted task weight.

[0090] Key features, such as task type, task size, task priority, and channel performance level, are extracted from the weighted task weights to form a task feature vector, which is used for subsequent matching between artificial intelligence models and the machine.

[0091] The extracted features are normalized to ensure comparability between different features. For example, all feature values ​​are normalized to the [0,1] interval so that different features can be compared in subsequent similarity calculations. After normalizing the extracted model features, a model feature vector is generated.

[0092] When extracting features from artificial intelligence models, a performance evaluation is performed on each available AI model, extracting its features such as accuracy, response speed, and resource consumption. These features reflect the model's ability to handle different types of tasks. After normalizing the extracted model features, a model feature vector is generated.

[0093] Similarly, performance evaluations can be performed on each available AI machine to extract its features, such as hardware configuration (CPU, GPU, memory), network bandwidth, and load. These features reflect the resource status of the machine when running the AI ​​model. After normalizing the extracted machine features, a machine feature vector is generated.

[0094] The task feature vector is matched item by item with each model feature vector and the machine-side feature vector, and the similarity between them is calculated. Similarity can be calculated by using cosine similarity, Euclidean distance, or other similarity measures between vectors.

[0095] By comprehensively considering the similarity between the model and the machine, the overall similarity of each model-machine combination can be obtained. This can be achieved by weighting and summing the model similarity and the machine similarity to reflect the overall suitability of the model and the machine when processing tasks.

[0096] Based on the calculated comprehensive similarity, all model and machine-end combinations are ranked. The few model and machine-end combinations with the highest or relatively high similarity are selected as the target AI machine-end or target AI model. It is important to emphasize that the AI ​​machine-end or target AI model addressed in this application can include locally deployed models or third-party models invoked via API interfaces. In setting the scheduling algorithm to determine the suitable target AI machine-end or target AI model, the limitations of the third-party model, such as RPM (calls per minute), RPD (calls per day), TPM (tokens per minute), and concurrency limits, can also be considered.

[0097] At this point, a similarity threshold can be preset. Only when the combined similarity of the model and the machine is higher than the threshold will it be considered a valid candidate, thus avoiding the selection of unsuitable models and machines and ensuring the quality and efficiency of task processing.

[0098] Building upon this, channel gain can be used for further filtering. Specifically, an initial set of targets can be determined first. Based on this initial set, the equivalent channel gain of each AI model and each AI machine tool can be calculated. Target clusters can then be filtered based on the equivalent channel gain, and within these target clusters, suitable target AI machine tools or target AI models can be selected according to task type and task parameters.

[0099] When calculating channel gain for artificial intelligence models, the complexity of the reference model (such as the number of parameters, network layers, etc.) and its data processing capabilities (such as data throughput) are considered. These factors affect the model's performance in different channels (such as network transmission channels, computing channels, etc.).

[0100] For AI-powered machines, consider their hardware performance (such as computing power and storage capacity) and network connectivity performance (such as bandwidth and latency). For example, a machine with a high-bandwidth network connection may have higher gain in the data transmission channel.

[0101] Simulation can be used to construct a virtual channel environment and simulate the working state of the model and the machine in real-world application scenarios. For example, different network load conditions and data input / output conditions can be simulated to observe the performance changes of the model and the machine, thereby estimating the equivalent channel gain.

[0102] Alternatively, real-world testing can be used to test the model and the machine in a real network environment. For example, the model can be deployed on the machine, and the performance indicators of the model and the machine under different channel conditions, such as processing time and data transmission rate, can be measured through actual data transmission and processing, thereby calculating the equivalent channel gain.

[0103] The model and the machine are evaluated based on the calculated equivalent channel gain. The gain value can be used as the primary evaluation metric to rank the models and machine. For example, models and machine with higher equivalent channel gains can be ranked higher. Other auxiliary factors can also be considered for a comprehensive evaluation. These include the model's scalability (whether parameters can be easily adjusted or the model expanded) and the machine's reliability (such as failure rate and maintenance costs). These factors can be used as a basis for adjusting the ranking.

[0104] Based on the ranking results, a certain number of models and machines are selected to form the target cluster. This number can be determined according to the needs and resource availability of the actual application scenario. For example, if the application scenario has high reliability requirements, multiple models and machines with high equivalent channel gain can be selected to form the cluster to achieve redundancy backup.

[0105] Models and machines within the target cluster are filtered based on task type. For example, for image recognition tasks, models with advantages in image processing (such as convolutional neural network models) and machines with high-performance GPUs are prioritized.

[0106] For natural language processing tasks, consider choosing models that specialize in text processing (such as recurrent neural network models or Transformer models) and machines with large memory capacity (because natural language processing tasks typically require processing large amounts of text data).

[0107] When considering task parameters, factors such as data volume and processing precision requirements can be taken into account. For example, if the task involves a large amount of data, a model and processing machine with high data throughput capabilities should be selected. For tasks with high precision requirements, models with advantages in precision optimization (such as finely tuned deep learning models) and processing machines capable of providing high-precision computation (such as servers with high-precision floating-point operation capabilities) should be selected.

[0108] In addition, the real-time requirements of the task should be considered. For tasks with high real-time requirements, models and machine tools with fast processing speeds and low latency should be selected. For example, in a real-time speech recognition system, a model and machine tool capable of quickly processing speech signals and providing results should be chosen.

[0109] See Figure 2 , Figure 2 This is a schematic diagram of a task scheduling system provided in an embodiment of this application. The system includes:

[0110] The communication layer contains the communication protocol between the coordinator service layer and the processor layer;

[0111] The coordinator service layer is used to select all objects in the AI ​​machine group and AI model group that are adapted to the task type as candidate nodes, calculate the network transmission latency and inference latency of each candidate node according to the task parameters, calculate the node priority score of each candidate node based on the network transmission latency, the inference latency and their respective latency weights, and determine the target AI machine or target AI model based on the node priority score.

[0112] The processor layer is used to send the user task request to the AI ​​machine via an application layer communication protocol when the AI ​​machine is idle, or to transmit the user task request over the network to the application programming interface corresponding to the target AI model when the load of the target AI model is below a set threshold.

[0113] In conjunction with the above embodiments, see Figure 3 , Figure 3 This is a communication timing diagram provided in an embodiment of this application. Figure 3 AI machines include artificial intelligence machine-side components and artificial intelligence models. The coordinator corresponds to the coordinator service layer mentioned above. The ETCD configuration center is a configuration management system based on distributed key-value storage. Its exemplary application process may include the following stages:

[0114] Machine registration stage:

[0115] 1. The AI ​​machine sends a connection request to the coordinator via WebSocket.

[0116] 2. The coordinator confirms the connection.

[0117] 3. The AI ​​machine sends its registration information to the coordinator (which may include MAC address, model type, and model ID).

[0118] 4. The coordinator sends machine information to the ETCD configuration center, which then stores the machine information.

[0119] 5. The ETCD configuration center confirms the storage and sends the storage confirmation information to the coordinator.

[0120] 6. The coordinator sends a registration success confirmation to the ETCD configuration center.

[0121] Heartbeat detection phase:

[0122] 1. Every 30 seconds, the AI ​​machine sends heartbeat information to the coordinator (CPU, memory, status).

[0123] 2. The coordinator updates the machine status of the AI ​​machine based on the heartbeat information.

[0124] 3. The coordinator confirms the heartbeat.

[0125] Task distribution phase:

[0126] 1. The client sends a user task request (including task type and parameters).

[0127] 2. The coordinator enqueues the tasks corresponding to the user's task requests.

[0128] 3. The coordinator uses an intelligent scheduling algorithm to match and send the results to the corresponding AI machine segment or AI model.

[0129] 4. The coordinator confirms task acceptance.

[0130] 5. The coordinator sends a confirmation message to the client confirming that the task has been distributed.

[0131] Task execution phase:

[0132] 1. The AI ​​terminal executes the corresponding AI tasks.

[0133] 2. The coordinator updates the task progress on the AI ​​machine side.

[0134] 3. The coordinator sends progress notifications to the client.

[0135] 4. The AI ​​completes the task and sends the result data to the coordinator.

[0136] 5. The coordinator confirms the task completion event and reports it to the message queue.

[0137] 6. The coordinator sends a task completion notification to the client.

[0138] Troubleshooting phase:

[0139] If a machine malfunction occurs:

[0140] 1. If the connection between the AI ​​machine and the coordinator is lost.

[0141] 2. Update the coordinator's status to offline.

[0142] 3. The coordinator sends a fault event to the ETCD configuration center.

[0143] 4. The coordinator performs a task retry or transfer.

[0144] This application also provides an embodiment of a computer-readable storage medium and a computer program product. Both the computer-readable storage medium and the computer program product may store a computer program that, when executed by a processor, implements the steps of the method described in the above method embodiments.

[0145] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] The computer-readable storage medium provided in this embodiment includes the method mentioned above, and has the same effect.

[0147] This application also provides an electronic device, see [link to document]. Figure 4 The present application provides a structural diagram of an electronic device, such as... Figure 4 As shown, it may include a processor 1410 and a memory 1420.

[0148] The processor 1410 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 1410 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1410 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 1410 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 1410 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0149] The memory 1420 may include one or more computer-readable storage media, which may be non-transitory. The memory 1420 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 1420 is used to store at least the following computer program 1421, which, after being loaded and executed by the processor 1410, is capable of implementing the relevant steps in the methods executed by the electronic device side as disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 1420 may also include an operating system 1422 and data 1423, etc., and the storage method may be temporary storage or permanent storage. The operating system 1422 may include Windows, Linux, Android, etc.

[0150] In some embodiments, the electronic device may further include a display screen 1430, an input / output interface 1440, a communication interface 1450, a sensor 1460, a power supply 1470, and a communication bus 1480.

[0151] certainly, Figure 4 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than [other components]. Figure 4 More or fewer components as shown, or combinations of certain components.

[0152] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. As the system provided in the embodiments corresponds to the method provided in the embodiments, the description is relatively simple; relevant parts can be found in the method section.

[0153] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

[0154] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A task scheduling method, characterized by, The method comprises the following steps: acquiring a user task request; analyzing the task type and task parameters of the user task request; taking all objects in the artificial intelligence machine end group and artificial intelligence model group adapted to the task type as candidate nodes, calculating the network transmission delay and inference delay of each candidate node according to the task parameters, calculating the node priority score of each candidate node based on the network transmission delay, the inference delay and the respective delay weight, and determining the target artificial intelligence machine end or target artificial intelligence model based on the node priority score; when the artificial intelligence machine end is idle, sending the user task request to the artificial intelligence machine end through an application layer communication protocol, or when the load of the target artificial intelligence model is lower than a set threshold, network transmitting the user task request to the application program interface corresponding to the target artificial intelligence model, and updating the target artificial intelligence machine end or the target artificial intelligence model to a busy state.

2. The method of claim 1, wherein, Further comprising: receiving the machine model registration information sent by the artificial intelligence machine end through the application layer communication protocol; recording the machine information corresponding to the machine model registration information, and assigning a unique identifier to the machine information; the unique identifier is used to maintain a heartbeat connection with the artificial intelligence machine end.

3. The task scheduling method of claim 1, wherein, If the user task request fails to be sent, further comprising: recording the number of retries and performing task resending until the number of retries reaches the upper limit of retries, and recording the user task request that fails in task scheduling.

4. The task scheduling method of claim 1, wherein, Further comprising: periodically detecting the processing time length of the user task request; updating the task state of the user task request according to the processing time length; feeding back the task state to the task initiator.

5. The task scheduling method of claim 1, wherein, Calling a set scheduling algorithm to determine the adapted target artificial intelligence machine end or target artificial intelligence model comprises: calculating the task weight of the user channel corresponding to the user task request according to the task type and the task parameters; grading and weighting the user channel to obtain a task feature vector; calculating the similarity between the respective artificial intelligence features of each artificial intelligence model and each artificial intelligence machine end and the task feature vector; selecting the adapted target artificial intelligence machine end or target artificial intelligence model according to each similarity.

6. The task scheduling method of claim 5, wherein, When selecting the adapted target artificial intelligence machine end or target artificial intelligence model according to each similarity, further comprising: determining an initial object; calculating the equivalent channel gain of each artificial intelligence model and each artificial intelligence machine end based on the initial object; selecting a target cluster according to the equivalent channel gain; selecting the adapted target artificial intelligence machine end or target artificial intelligence model according to the task type and the task parameters in the target cluster.

7. A task scheduling system characterized by, The method comprises the following steps: a communication layer containing a communication protocol between a coordinator service layer and a processor layer; The coordinator service layer is configured to select all objects in an artificial intelligence machine end group and an artificial intelligence model group adapted to the task type as candidate nodes, calculate network transmission delay and inference delay of each candidate node according to the task parameters, calculate a node priority score of each candidate node based on the network transmission delay, the inference delay and a respective delay weight, and determine a target artificial intelligence machine end or a target artificial intelligence model based on the node priority score. The processor layer is configured to send the user task request to the artificial intelligence machine end through an application layer communication protocol when the artificial intelligence machine end is idle, or to network transmit the user task request to an application program interface corresponding to the target artificial intelligence model when a load of the target artificial intelligence model is lower than a set threshold, and update the target artificial intelligence machine end or the target artificial intelligence model to a busy state.

8. An electronic device, comprising: The computer program is stored in the computer readable storage medium and is executed to implement the steps of the method according to any one of claims 1 to 6. The computer program is stored in the computer readable storage medium and is executed to implement the steps of the method according to any one of claims 1 to 6. The computer program is stored in the computer readable storage medium and is executed to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, ​ 10. A computer program product, characterised in that, ​

Images