Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium. In a scene of performing multi-stage processing on input data, for a processing task of an Nth stage in the multi-stage processing on the input data, in response to the condition that an Nth node for processing the processing task of the Nth stage is in an overload state, whether a non-Nth node for processing a processing task of a non-Nth stage is in a non-overload state is detected. And in response to the non-Nth node being in the non-overload state, scheduling the processing task of the Nth stage to the non-Nth node for processing. Therefore, the nodes which are used for processing the processing tasks of one stage and are in the non-overload state can process the processing tasks of the other stage, system resources of the nodes which are used for processing the processing tasks of the other stages and are in the non-overload state can be fully utilized, and the overall system resource utilization rate of the data processing system can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In a reasoning service system based on a Large Language Model (LLM), the processing of input data can involve two reasoning phases, such as the Prefill phase and the Decode phase.

[0003] The Prefill phase is responsible for processing input data (or input prompts, such as Prompt), generating a key-value cache (KVCache), and the first output token.

[0004] The Decode phase is responsible for using the KVCache generated in the Prefill phase to gradually generate subsequent output identifiers (the second output identifier and subsequent output identifiers). Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and storage medium.

[0006] In a first aspect, this application discloses a data processing method, the method comprising: In a scenario where input data is processed in multiple stages, for the Nth stage of the multi-stage processing of the input data, it is detected whether the Nth node used to process the Nth stage processing task is in an overload state; N is a positive integer greater than or equal to 1. In response to the Nth node being in an overloaded state, it is detected whether the non-Nth node used to process processing tasks in non-Nth stages is in a non-overloaded state. In response to the non-Nth node being in a non-overloaded state, the processing task of the Nth stage is scheduled to be processed by the non-Nth node.

[0007] Secondly, this application discloses a data processing apparatus, the apparatus comprising: The first detection module is used to detect whether the Nth node used to process the Nth stage processing task is in an overload state in a scenario where the input data is processed in multiple stages; N is a positive integer greater than or equal to 1. The second detection module is used to detect whether the non-Nth node used for processing tasks in non-Nth stages is in a non-overload state in response to the Nth node being in an overload state. The first scheduling module is used to schedule the processing task of the Nth stage to the non-Nth node for processing in response to the non-Nth node being in a non-overload state.

[0008] Thirdly, this application discloses an electronic device comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the methods shown in any of the foregoing aspects.

[0009] Fourthly, this application discloses a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the methods shown in any of the foregoing aspects.

[0010] Fifthly, this application discloses a computer program product in which, when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the methods shown in any of the foregoing aspects.

[0011] This application has the following advantages: In this application, in a scenario involving multi-stage processing of input data, for the Nth stage processing task, it is detected whether the Nth node used to process the Nth stage processing task is in an overloaded state. N is a positive integer greater than or equal to 1. In response to the Nth node being overloaded, it is detected whether the non-Nth nodes used to process non-Nth stage processing tasks are in a non-overloaded state. In response to the non-Nth node being non-overloaded, the Nth stage processing task is scheduled to be processed by the non-Nth node.

[0012] If the Nth node is in an overloaded state, it means that the Nth node is heavily loaded. Even if the processing tasks of the Nth stage are scheduled to the Nth node, the Nth node will have difficulty completing the processing tasks of the Nth stage in a short time. This will result in low efficiency of the Nth node in processing the Nth stage tasks and further deteriorate the load situation of the Nth node.

[0013] Therefore, to avoid further deteriorating the load on the Nth node and to improve the efficiency of processing tasks in the Nth stage, the Nth stage processing tasks can be scheduled to nodes that handle other stages instead of the Nth node. If nodes that handle other stages need to process the Nth stage processing tasks, those nodes need to be in a non-overloaded state. To this end, in response to the Nth node being overloaded, it is possible to detect whether nodes that handle other stages are overloaded, for example, by detecting whether non-Nth nodes that handle non-Nth stage processing tasks are in a non-overloaded state. If a non-Nth node is not overloaded, it means that the load on the non-Nth node is light or moderate. The processing tasks of the Nth stage can be scheduled to the non-Nth node. The non-Nth node can complete the processing tasks of the Nth stage in a short time. The non-Nth node has high efficiency in processing the processing tasks of the Nth stage and will not cause the load situation of the non-Nth node to deteriorate further. Therefore, the processing tasks of the Nth stage can be scheduled to the non-Nth node.

[0014] This application enables nodes that are not overloaded and are used to process tasks in one stage to process tasks in another stage. Thus, if a node used to process a task in a certain stage is overloaded, a node that is not overloaded and is used to process tasks in other stages can be used to process tasks in that stage. This fully utilizes the system resources of nodes that are not overloaded and are used to process tasks in other stages, avoids resource waste, improves the overall system resource utilization of the data processing system, and also avoids interruption of the processing task in that stage. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a data processing system according to this application.

[0017] Figure 2 This is a schematic diagram of a node pool according to this application.

[0018] Figure 3 This is a schematic diagram of one type of node in this application.

[0019] Figure 4This is a flowchart of the steps of a data processing method according to this application.

[0020] Figure 5 This is a flowchart of the steps of a data processing method according to this application.

[0021] Figure 6 This is a flowchart of the steps of a data processing method according to this application.

[0022] Figure 7 This is a flowchart of the steps of a data processing method according to this application.

[0023] Figure 8 This is a flowchart of the steps of a data processing method according to this application.

[0024] Figure 9 This is a flowchart of the steps of a data processing method according to this application.

[0025] Figure 10 This is a flowchart of the steps of a data processing method according to this application.

[0026] Figure 11 This is a flowchart of the steps of a data processing method according to this application.

[0027] Figure 12 This is a schematic diagram of a data processing method according to this application.

[0028] Figure 13 This is a schematic diagram of a data processing method according to this application.

[0029] Figure 14 This is a structural block diagram of a data processing device according to this application.

[0030] Figure 15 This is a structural block diagram of a device according to this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Many specific details are set forth in the following description to facilitate a full understanding of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. This specification can be implemented in many other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific embodiments disclosed below. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort should fall within the scope of protection of this specification.

[0032] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0033] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0034] Currently, a node can handle processing tasks in both the Prefill and Decode stages.

[0035] Sometimes, both the Prefill and Decode phases require processing on the same node, for example, both needing to utilize the GPU. However, processing tasks are typically scheduled to the GPU in batches. A node can schedule a batch of Prefill phase tasks to the GPU, or a batch of Decode phase tasks to the GPU. However, if a batch of tasks is large and the node's system resources are limited (e.g., the node's GPU memory is limited), then each time processing, either only one batch of Prefill stage processing tasks can be scheduled to the GPU for processing, or only one batch of Decode stage processing tasks can be scheduled to the GPU for processing. It is usually not possible to process one batch of Prefill stage processing tasks and one batch of Decode stage processing tasks at the same time. Instead, one batch of tasks is processed first, and then another batch of tasks is processed.

[0036] Therefore, in this situation, a batch of Prefill stage processing tasks and a batch of Decode stage processing tasks that the same node needs to process will compete for the node's system resources (such as GPU resources). As a result, either a batch of Prefill stage processing tasks or a batch of Decode stage processing tasks cannot be processed, leading to low efficiency and high latency in processing Prefill stage processing tasks or Decode stage processing tasks. This interference makes it difficult for each stage to meet SLOs (Service Level Objectives).

[0037] SLOs are performance targets set for a specific service over a period of time, used to measure and monitor the service's performance, quality, and reliability. SLOs are defined based on Service Level Indicators (SLIs), which are quantitative metrics for a certain aspect of a service (such as latency, error rate, etc.), while SLOs are the target values ​​or ranges for these metrics (e.g., access latency <500ms, or availability reaching 99.9%).

[0038] For example, it may cause the TTFT (Time To First Token) metric of the Prefill phase processing task to fail to meet the requirements, or cause the TPOT (Time Per Output Token) metric of the Decode phase processing task to fail to meet the requirements.

[0039] To address these issues, another approach is to employ a Prefill-Decode phase separation strategy in the inference service system. For example, the Prefill and Decode phases are handled by different nodes, with each node processing only either the Prefill or Decode phase tasks. This allows for independent processing without interference, improving efficiency and supporting automated and flexible resource allocation across all phases to meet evolving needs.

[0040] However, statistical analysis has revealed that this separation strategy can also lead to low resource utilization in practice.

[0041] For example, in an inference service system with separate Prefill and Decode phases, nodes for handling Prefill tasks and nodes for handling Decode tasks are pre-configured. The number of nodes for Prefill tasks is fixed, the number of nodes for Decode tasks is fixed, and the ratio between the number of nodes for Prefill tasks and the number of nodes for Decode tasks is fixed.

[0042] For example, when a large number of Prefill tasks need to be processed, the nodes responsible for processing these tasks may not be able to handle them all simultaneously due to limited system resources. In this case, the large number of Prefill tasks must be processed in batches, for example, processing a portion of the Prefill tasks first, and then processing the remaining portion. Before the nodes processing Prefill tasks have completed their portion, the nodes responsible for processing Decode tasks remain in a waiting state, resulting in idle bubbles on the Decode task processing nodes. This leads to low system resource utilization on the Decode task processing nodes, such as low memory utilization.

[0043] Especially when the input data for a large number of Prefill stage processing tasks contains a large number of characters, the time spent by the nodes processing the large number of Prefill stage processing tasks is often very long. Consequently, the waiting time for the nodes processing the Decode stage processing tasks is also very long, resulting in a long idle bubble period for the nodes processing the Decode stage processing tasks. In other words, this leads to very low system resource utilization for the nodes processing the Decode stage processing tasks, such as very low memory utilization.

[0044] For example, after processing a large number of Prefill stage tasks, at least a portion of the Decode stage tasks still need to wait for the nodes used to process Decode stage tasks (these nodes are currently processing a portion of the Decode stage tasks). Before the nodes used to process Decode stage tasks finish processing this portion of the Decode stage tasks, the nodes used to process Prefill stage tasks remain in a waiting state. This results in idle bubbles appearing on the nodes used to process Prefill stage tasks, which in turn leads to low system resource utilization, such as low computational utilization.

[0045] Especially when the number of characters in the text of the output results corresponding to a large number of Decode stage processing tasks is large, the time spent by the nodes processing the Decode stage processing tasks is often very long. Correspondingly, the waiting time for the nodes processing the Prefill stage processing tasks is also very long, resulting in a long idle bubble period for the nodes processing the Prefill stage processing tasks. In other words, the system resource utilization of the nodes processing the Prefill stage processing tasks is very low, for example, the computational utilization is very low.

[0046] In summary, the overall system resource utilization rate of the reasoning service system is low.

[0047] In order to improve the overall system resource utilization of the reasoning service system, the inventors analyzed the reasons for the low overall system resource utilization of the reasoning service system and found that it was because the stages of the processing tasks that the nodes could perform were too fixed and rigid, which resulted in the failure to make full use of the system resources of idle nodes.

[0048] Therefore, in order to improve the overall system resource utilization, the inventors came up with the idea of ​​having nodes that handle processing tasks in the Prefill stage and nodes that handle processing tasks in the Decode stage work together.

[0049] For example, in a data processing system, a node used to handle processing tasks in one stage can handle processing tasks in another stage. For instance, if a node used to handle a certain stage of processing tasks is overloaded, a node used to handle processing tasks in other stages that is not overloaded can be used to handle the processing tasks in that stage. This makes full use of the system resources of the nodes used to handle processing tasks in other stages that are not overloaded, avoids resource waste, improves the overall system resource utilization of the data processing system, and also avoids interruption of the processing task in that stage.

[0050] For example, nodes that handle Prefill stage tasks can also handle Decode stage tasks when not overloaded, and nodes that handle Decode stage tasks can also handle Prefill stage tasks when not overloaded, so as to make full use of the system resources of nodes in a non-overloaded state and improve the overall system resource utilization of the data inference system.

[0051] For example, according to statistics, under actual testing, the solution proposed in this application can reduce the average output time of Tokens by 10%-70%, and the average end-to-end output time (e.g., the time from obtaining input data to outputting the final result) can be reduced by 10%-70%, which has great practical application value.

[0052] Before introducing the technical solution of this application, the technical terms that may be involved in the technical solution of this application will be explained.

[0053] Large Language Models (LLMs): Large Language Models are natural language processing systems based on deep learning techniques that can understand and generate human language. LLMs possess powerful text generation and understanding capabilities and are widely used in various language tasks. Separate inference (Prefill-Decode stage separation): Separate inference refers to deploying the computationally intensive Prefill stage processing tasks and the memory-intensive Decode stage processing tasks on different nodes (such as GPUs or CPUs) during the inference process of large language models, in order to reduce system resource contention and improve running efficiency. Prefill Phase: In language model inference, the prefill phase refers to the process of generating a key-value cache (KV cache) and the first output token through parallel computation of input prompts. The prefill phase primarily involves computationally intensive tasks. It is a computationally intensive phase, and its performance can be measured by the time taken from the start of input data processing to the generation of the first output token (TTFT). TTFT is a performance metric for the prefill phase, representing the time required from the start of input data processing to the generation of the first output token.

[0054] The Decode phase is the process of iteratively outputting subsequent tokens using the KV Cache generated in the Prefill phase during language model inference. The Decode phase is primarily a memory-intensive task, involving frequent accesses to the continuously expanding KV Cache. Because of these frequent accesses, the Decode phase is memory-intensive, and its performance can be evaluated using the TPOT (Total Time Required) for each subsequent output token (excluding the first output token). TPOT reflects the average time required to generate each subsequent output. TPOT is a performance metric for the Decode phase, representing the time required from utilizing the KV Cache to outputting each subsequent token.

[0055] End-to-End Delay: End-to-End Delay refers to the time required from the start of processing input data to the completion of the entire inference process, covering all computation and output time in the Prefill and Decode phases. See Figure 1 The diagram shows a data processing system proposed in this application, which includes an API (Application Programming Interface) server, a global scheduler, and a node pool.

[0056] The API server and the global scheduler can exchange data.

[0057] The global scheduler can interact with the nodes in the node pool.

[0058] The API server serves as the interface exposed by the data processing system. It receives input data (such as request data) from external clients and transmits it to the global scheduler. The global scheduler then schedules the input data to nodes in the node pool for processing. After receiving the processing results from the nodes, the global scheduler returns the processed results to the API server. The API server, in turn, returns the processed results to the external clients, enabling them to access the processed data.

[0059] Secondly, the API server can also be responsible for authenticating and authorizing external clients, ensuring that only valid input data can be transmitted to the global scheduler, and can also implement fine-grained access control to protect sensitive data and improve security.

[0060] The global scheduler receives input data transmitted from the API server and schedules the input data to nodes in the node pool for processing. This application performs multi-stage processing on the input data, which includes multiple processing tasks (such as Prefill and Decode stages). For any stage of processing, the global scheduler dynamically senses the load status of each node in the node pool in real time and schedules the processing task of that stage to a node in the node pool that is not overloaded based on a load balancing strategy. The same applies to the processing tasks of each other stage, thereby evenly distributing a large number of processing tasks in the multi-stage processing of a large amount of input data to nodes in the node pool, improving the overall system resource utilization, availability, and performance of the data processing system.

[0061] The global scheduler can contain multiple components, such as a scenario judgment component, a node load measurement component, a scheduling component, an elastic scaling component, and a node status monitoring component.

[0062] The scenario judgment component is used to determine the application scenario (such as chat scenario, text-to-image scenario, and question-and-answer scenario) involved in the input data after receiving the input data, and to give the weight or priority of the application scenario. This can provide the subsequent scheduling component with reference data such as "whether the input data focuses more on latency or throughput" and "whether queuing / preemption is allowed".

[0063] The node load measurement component can collect and quantify the system resource usage of nodes in the node pool in real time, such as GPU (Graphics Processing Unit) utilization, video memory utilization, CPU (Central Processing Unit) utilization, bandwidth utilization, queue length, and node temperature. It can also generate sortable load scores based on these data, and provide decision-making basis for the scheduling component and the elastic scaling component, respectively.

[0064] The scheduling component is used to determine the node to process the task based on the stage in the multi-stage processing and the system resource usage of the nodes in the node pool, and then schedules the task to the determined node for processing.

[0065] The elastic scaling component is used to automatically adjust the size of the node pool based on the current information of the nodes collected by the node load measurement component and the long-term trend information of the nodes (e.g., to expand the node pool with new nodes or shrink existing idle nodes).

[0066] This allows for elastic scaling of nodes in the node pool. When a large number of processing tasks need to be handled, new nodes can be added to the node pool. When the number of processing tasks decreases, some nodes in the node pool can be shut down or unloaded to reclaim some system resources, thus achieving flexible allocation of system resources.

[0067] The node status monitoring component monitors the health and availability (e.g., heartbeat, GPU failure, memory overflow, abnormal temperature, network isolation) of nodes in the node pool, and marks abnormal nodes as unavailable or triggers a restart in real time. It can also provide information on nodes that need to be replaced to the elastic scaling component.

[0068] The node pool consists of multiple nodes, which can be deployed on a single physical machine or distributed across a physical machine cluster.

[0069] A node can be a physical node, such as a GPU or CPU on a physical machine.

[0070] Alternatively, a node can also be a virtual node. For example, a node is a virtual node that is created on a physical machine. Virtual nodes include virtual compute instances, such as ECS (Elastic Compute Service). In one example, a virtual compute instance may include a virtual GPU or a virtual CPU.

[0071] If a node is a virtual node created on a physical machine, system resources can be allocated to the node on demand through virtualization technology. System resources can include computing resources, storage resources, and network resources on the physical machine it resides in, such as GPU, memory, disk, and network interfaces, so that each virtual node can use the allocated system resources to process tasks.

[0072] Virtual nodes can be created, started, stopped, or deleted as needed, and the system resources allocated to virtual nodes can also be dynamically adjusted according to actual requirements.

[0073] The physical machines mentioned above can be local physical machines or physical machines in the cloud, etc.

[0074] Multiple nodes in a node pool are connected via a network to work together. For example, they can collaboratively process tasks. The process of processing input data can include multiple processing stages. Different nodes or different types of nodes in the node pool process different stages of the "multi-stage processing of input data". In this way, it can provide stronger processing capabilities and higher availability than a single node.

[0075] In this application, nodes in the node pool can communicate with each other. The communication between nodes can be achieved through IPC (Inter-Process Communication), NCCL (NVIDIA Collective Communications Library), RDMA (RDMA Remote Direct Memory Access), or GPU Direct RAMA (GPU Direct Remote Direct Memory Access).

[0076] Further, see Figure 2 The diagram shows a node pool proposed in this application.

[0077] In this application, multiple nodes in the node pool can be divided into multiple categories according to multiple processing stages. The number of node categories can be the same as the number of stages in the multi-stage processing of input data. Each category of nodes is used to process the processing tasks of different processing stages in the multi-stage processing.

[0078] For example, if the multi-stage processing of input data includes M stages (M is a positive integer greater than or equal to 2), then the nodes in the node pool can be divided into M types of nodes according to the multiple processing stages. For example, there are nodes for processing the processing task of the first stage, nodes for processing the processing task of the second stage, and so on, and nodes for processing the processing task of the Mth stage. Each type has at least one node.

[0079] The number of nodes of each type in the node pool can be the same, or the number of nodes of each type in the node pool can be adjusted according to actual needs. For example, according to actual needs, at least one node of a certain type in the node pool can be changed to a node of another type so that the at least one node can then handle the processing tasks of the processing stage that the other type of node needs to handle.

[0080] In this application, for a node used to process a task in stage X, if the global scheduler obtains a task in stage X, it can by default schedule the task in stage X to a node in the node pool that is used to process tasks in stage X. However, the node used to process the task in stage X is not necessarily limited to processing tasks in stage X. Here, X is greater than or equal to and less than or equal to M.

[0081] For example, if a node used to process the Y-stage processing task is overloaded, and a Y-stage processing task is received, the Y-stage processing task may not be scheduled to a node in the node pool used to process the Y-stage processing task. Instead, the Y-stage processing task may be scheduled to a node in the node pool used to process the X-stage processing task (which is not overloaded).

[0082] In this case, stage X differs from stage Y. X is greater than or equal to and less than or equal to M. Y is greater than or equal to and less than or equal to M.

[0083] For example, if the node used to process the X-stage processing task is overloaded, the X-stage processing task may not be scheduled to the node in the node pool that is used to process the X-stage processing task. Instead, the X-stage processing task may be scheduled to the node in the node pool that is used to process the Y-stage processing task (and is not overloaded).

[0084] In other words, this application supports various types of nodes to handle processing tasks at any stage. For example, various types of nodes can handle processing tasks in the Prefill stage and processing tasks in the Decode stage.

[0085] One approach is load-aware scheduling, which dynamically schedules processing tasks at each stage of a multi-stage processing task to nodes with lower loads. This optimizes resource utilization, reduces waiting time and processing latency, and improves the overall throughput of the data processing system. It also dynamically balances the load across nodes, preventing some nodes from being overloaded while others are idle. By monitoring the load of each node in real time, the load-aware scheduler can make more reasonable task allocation decisions.

[0086] For example, in the data processing system described above, the API server can communicate with the global scheduler, and the global scheduler can communicate with each node in the node pool. In this case, after the API server schedules the input data to the global scheduler, for a processing task in one stage of the multi-stage processing of the input data, the global scheduler can schedule that stage of the processing task to a node in the node pool that is not overloaded (e.g., a node with a relatively low load) based on the load status of each node in the node pool. In this way, the system resources of nodes that are not overloaded can be fully utilized, improving the overall processing efficiency of the data processing system for processing tasks, thereby improving the performance and availability of the data processing system.

[0087] Each node in the node pool is independent of the others, and the system resources allocated to each node are also independent. This ensures that each node can use its own system resources independently without interfering with others, and guarantees the stability and security of each node's processing tasks.

[0088] Further, see Figure 3 The diagram shows a schematic of a node proposed in this application.

[0089] The node includes a hybrid scheduler.

[0090] In this application, a node used to process a task in a multi-stage processing process typically uses a scheduler for that stage deployed on the node. To ensure that a node processing a task in one stage can also handle tasks in other stages without overload, schedulers for each stage of the multi-stage processing can be deployed separately in a hybrid scheduler. For example, a scheduler for stage 1, stage 2, ..., and stage M can be deployed separately in the hybrid scheduler, where M is the number of stages involved in the multi-stage processing of the input data.

[0091] A hybrid scheduler in a node can manage the schedulers in that node that handle each stage separately.

[0092] The node's hybrid scheduler independently manages the schedulers used for processing each stage, enabling the schedulers used for processing each stage to execute the processing tasks for each stage.

[0093] Thus, for any stage of the processing task, after the global scheduler schedules the processing task of that stage to a node in the node pool, the hybrid scheduler in that node can process the processing task of that stage through the scheduler of that stage in that node.

[0094] Each stage's scheduler has its own Waiting queue, Running queue, Swapped queue, Pending queue, and Request Manager, etc. Each stage's scheduler can manage its own Waiting queue, Running queue, Swapped queue, Pending queue, and Request Manager, etc.

[0095] The Waiting queue is a waiting queue. After receiving a new processing task, before allocating system resources (such as GPU, memory, and KV cache space) to the new processing task, the new processing task can be temporarily stored in the Waiting queue.

[0096] The Running queue is a running queue used to store processing tasks that are currently being processed (system resources have been allocated to these tasks).

[0097] A swapped queue is used to store processing tasks that are temporarily swapped out to CPU memory or disk due to insufficient system resources (such as KV cache overflow). For example, when GPU memory is insufficient, low-priority or long-running processing tasks are swapped out to the CPU / disk to free up space for other processing tasks.

[0098] The Pending queue is used to store processing tasks that are temporarily suspended due to unmet dependencies. Some processing tasks may need to wait for other processing tasks to complete (such as depending on the result of the previous round in a multi-turn dialogue). For example, during the Decode phase, if the key-value cache of a processing task is swapped out (stored in the Swapped queue), it must wait for it to be swapped back in before processing of processing tasks in the Decode phase can continue.

[0099] The Request Manager is a core component for coordinating the lifecycle of processing tasks. It interacts closely with the scheduler, tracking the entire process from task reception, scheduling, execution to completion. It records the current state of the task (e.g., Waiting / Running / Swapped / Pending) and its context (KV cache location, number of generated tokens, etc.). It can collaborate with the scheduler to allocate / reclaim KV cache and GPU computing resources, and trigger swap-in / swap-out operations.

[0100] To illustrate with an example, after receiving a new processing task, the scheduler stores the new processing task in the Waiting queue (waiting for KV cache space, etc.). After the scheduler allocates system resources for the new processing task, it moves the new processing task from the Waiting queue to the Running queue (for example, to process processing tasks in the Prefill or Decode stages). If there is insufficient memory allocated for the new processing task, the scheduler swaps the new processing task out to the Swapped queue. In addition, if the new processing task depends on other results, the scheduler stores the new processing task in the Pending queue (such as waiting for KV cache swapping, etc.).

[0101] In addition, the hybrid scheduler also deploys a Shared Block Manager, which can be shared by the hybrid scheduler and the scheduler used to handle each stage.

[0102] The Shared Block Manager is used to manage the physical memory blocks of the KV cache, such as allocating / releasing physical memory blocks of the KV cache on the GPU or CPU. It maintains the mapping relationship between logical memory blocks and physical memory blocks and supports sharing mechanisms (such as prompt prefix sharing), which can significantly improve the utilization of video memory and is a key component to support high concurrency and low latency inference.

[0103] Secondly, in addition to the hybrid scheduler, the node also has a Model Executor and a States Monitor.

[0104] The Model Executor is a tool responsible for processing tasks (such as model inference). It is located between the scheduler and the worker, connecting to the scheduler and calling the worker to complete GPU computation.

[0105] States Monitor is used to track the number of key-value blocks used by each processing task, the system resources used, and so on.

[0106] For any type of node in the node pool, the node can cache the data used by the node when processing a stage of the processing task and the processing result obtained by the node when processing other stages of the processing task. This allows other types of nodes to directly retrieve the data used by the node when processing a stage of the processing task and the processing result obtained by the node when processing other stages of the processing task. This avoids other types of nodes repeatedly calculating the data used by the node when processing other stages of the processing task that are located after the current stage of the processing task, thus saving computing resources for other types of nodes.

[0107] In practical applications, to ensure the reliability and correctness of task scheduling, the global scheduler can maintain a scheduling queue.

[0108] For example, processing tasks belonging to the same stage (which may belong to different input data) can be temporarily stored in the scheduling queue. Then, the processing tasks belonging to the same stage in the scheduling queue can be scheduled in batches to nodes in the node pool, so that the nodes can process a batch of processing tasks belonging to the same stage in batches. For example, parallel processing of a batch of processing tasks belonging to the same stage can improve the overall processing efficiency and improve the utilization of the system resources of the nodes.

[0109] Secondly, it can also centrally manage the processing tasks in the scheduling queue, for example, by tracking the execution status of the processing tasks, including whether they are not executed, are being executed, have been completed, or have failed.

[0110] In one example, a data processing system may include a data inference system, which may provide inference services based on large models (such as generative models in natural language processing).

[0111] The reasoning process of a large model-based reasoning service can include a Prefill stage and a Decode stage.

[0112] In the inference process of large-model-based inference services, the Prefill and Decode phases are different steps in generating text. In practice, the Prefill and Decode phases are closely linked. The Prefill phase provides the initial text foundation, while the Decode phase is responsible for progressively expanding upon this foundation to generate the complete text output. These two phases together determine the quality and coherence of the final generated text.

[0113] The Prefill stage is the initial stage of text generation. Its main purpose is to provide an initial context for the large model so that it can better understand and generate subsequent text.

[0114] In the Prefill stage, large models typically generate an initial text fragment containing the beginning of the generated sequence. This stage involves using strategies or algorithms to select the most suitable starting words or phrases.

[0115] For example, when dealing with a conditional generation task, a large model might first generate a portion of text as a basis based on the input conditions (e.g., summary generation, question answering, dialogue, etc.).

[0116] The Decode stage is the main stage for generating text. In this stage, the large model gradually generates new words or characters until a complete output sequence is generated.

[0117] During the Decode phase, the large model generates new text word by word or character by character based on the existing text. In each iteration, the large model predicts the next most likely word and adds it to the current text sequence. This process continues until a termination condition is met (e.g., reaching the maximum text length, generating a text end marker, etc.).

[0118] During the decoding stage, strategies can be used to optimize the quality of the generated text, such as Top-k sampling and Top-p (also known as Nucleus Sampling) sampling, to help large models avoid generating overly mundane or meaningless content.

[0119] The nodes in the node pool can be divided into two categories: one category consists of nodes used to perform inference tasks in the Prefill phase, and the other category consists of nodes used to perform inference tasks in the Decode phase.

[0120] Furthermore, based on Figure 1-3 The illustrated embodiment can be found in [reference]. Figure 4 This application illustrates a data processing method that can be applied to... Figure 1 In the global scheduler shown, the method may include: In step S101, in a scenario where input data undergoes multi-stage processing, for the Nth stage processing task in the multi-stage processing of input data, it is detected whether the Nth node used to process the Nth stage processing task is in an overloaded state. N is a positive integer greater than or equal to 1.

[0121] N is less than or equal to the number of stages in a multi-stage processing. For example, assuming a multi-stage process is a 2-stage process, then the 2-stage processing of the input data includes 2 stages, and N is greater than or equal to 1 and less than or equal to 2.

[0122] Input data includes input requests, such as questions asked by users.

[0123] In this application, the processing of input data includes multi-stage processing.

[0124] For example, in one example, the first stage of processing includes processing the input data to obtain the result of the first stage. For any stage from the second stage onwards, that stage further processes the input data and / or the results obtained from the stages preceding it to obtain the result of that stage. If that stage is the last stage, then the result obtained in that stage can be the final result of multi-stage processing of the input data.

[0125] For example, in one scenario, suppose the multi-stage processing of input data includes a two-stage processing scenario, comprising a first stage and a second stage. The first stage can be the Prefill stage in the inference service, and the second stage can be the Decode stage in the inference service. Thus, when N is 1, the processing task of the Nth stage includes the inference task of the Prefill stage. Alternatively, when N is 2, the processing task of the Nth stage includes the inference task of the Decode stage.

[0126] For any stage in a multi-stage processing, nodes are pre-configured in the node pool to handle the processing tasks for that stage. By default, the processing tasks for that stage can be scheduled to the nodes designated for that stage.

[0127] However, sometimes the nodes used to handle tasks in this stage are overloaded. An overloaded node means it is heavily loaded and struggles to process newly scheduled tasks quickly, leading to low efficiency and further deterioration of its performance. Therefore, when a node is overloaded, its tasks can be rescheduled to nodes that handle other stages (where these nodes are not overloaded).

[0128] The same applies to each of the other stages in a multi-stage process.

[0129] Alternatively, sometimes the node used to handle the processing tasks in this stage is in a non-overloaded state. This means that the node's load is light or normal (generally). The node can complete the processing tasks newly scheduled to it in a short time, making the node's processing efficiency high and preventing the node's state from deteriorating further. Since the processing tasks in this stage are scheduled to the node used to handle the processing tasks in this stage by default, when the node used to handle the processing tasks in this stage is in a non-overloaded state, the processing tasks in this stage can be scheduled to the node used to handle the processing tasks in this stage.

[0130] For example, in the multi-stage processing of input data, the Nth stage processing task can be scheduled to the Nth node in the node pool for processing the Nth stage processing task by default. If the Nth node in the node pool is to be used to process the Nth stage processing task, then the Nth node in the node pool for processing the Nth stage processing task needs to be in a non-overloaded state. Therefore, for the Nth stage processing task in the multi-stage processing of input data, it is possible to detect whether the Nth node in the node pool for processing the Nth stage processing task is in an overloaded state.

[0131] If the Nth node is not overloaded, the processing tasks of the Nth stage will be scheduled to the Nth node for processing.

[0132] Alternatively, if the Nth node is overloaded, the processing tasks of the Nth stage will not be scheduled to the Nth node for processing.

[0133] The node used to process the task in the Nth stage can be considered as the Nth node. There can be one or more Nth nodes used to process the task in the Nth stage.

[0134] In response to the Nth node being in an overloaded state, in step S102, it is detected whether the non-Nth node used to process processing tasks in non-Nth stages is in a non-overloaded state.

[0135] If the Nth node is in an overload state, it means that the Nth node is heavily loaded. Even if the processing tasks of the Nth stage are scheduled to the Nth node, the Nth node will have difficulty completing the processing tasks of the Nth stage in a short time. This will result in low efficiency of the Nth node in processing the Nth stage tasks and further worsen the load situation of the Nth node.

[0136] Therefore, to avoid further deteriorating the load on the Nth node and to improve the efficiency of processing tasks in the Nth stage, the Nth stage processing tasks can be scheduled to nodes that handle other stages instead of the Nth node. If nodes that handle other stages need to process the Nth stage processing tasks, those nodes need to be in a non-overloaded state. To this end, in response to the Nth node being overloaded, it is possible to detect whether nodes that handle other stages are overloaded, for example, by detecting whether non-Nth nodes that handle non-Nth stage processing tasks are in a non-overloaded state.

[0137] Non-Nth stage processing tasks include: processing tasks in multiple stages of multi-stage processing of input data, excluding the Nth stage processing tasks.

[0138] Non-Nth nodes include: nodes in the node pool that are used to handle processing tasks for stages other than the Nth stage.

[0139] For example, if the processing task in stage N is the inference task in the Prefill stage, then the processing tasks in stages other than N can be the inference tasks in the Decode stage.

[0140] Accordingly, the Nth node used to handle processing tasks in the Nth stage includes: nodes in the node pool used to handle inference tasks in the Prefill stage. The non-Nth nodes used to handle processing tasks in stages other than N include: nodes in the node pool used to handle processing tasks in the Decode stage.

[0141] For example, if the processing task in stage N is the processing task in the Decode stage, then the processing tasks in stages other than N can be the processing tasks in the Prefill stage.

[0142] Accordingly, the Nth node used to process the Nth stage processing task includes: the node in the node pool used to process the Decode stage processing task. The non-Nth node used to process non-Nth stage processing tasks includes: the node in the node pool used to process the Prefill stage processing task.

[0143] In response to the fact that non-Nth node is in a non-overloaded state, in step S103, the processing task of the Nth stage is scheduled to be processed by non-Nth node.

[0144] If a non-Nth node is not overloaded, it indicates that the load on the non-Nth node is light or moderate. The processing task of the Nth stage can be scheduled to the non-Nth node. The non-Nth node can complete the processing task of the Nth stage in a short time. The non-Nth node has high efficiency in processing the processing task of the Nth stage and will not cause the load situation of the non-Nth node to deteriorate further. Therefore, the processing task of the Nth stage can be scheduled to the non-Nth node.

[0145] For example, the global scheduler can schedule the processing task of stage N to the hybrid scheduler in the non-Nth node, and the hybrid scheduler in the non-Nth node can schedule the processing task of stage N to the scheduler in the non-Nth stage managed by the hybrid scheduler in the non-Nth node, so that the scheduler in the non-Nth stage can process the processing task of stage N.

[0146] In one embodiment of this application, there are multiple non-Nth nodes in a non-overloaded state. Thus, when scheduling the processing task of the Nth stage to a non-Nth node, the load index of the multiple non-Nth nodes in a non-overloaded state can be obtained. Among the multiple non-Nth nodes in a non-overloaded state, the non-Nth node with the lowest load index is determined, and the processing task of the Nth stage is scheduled to the non-Nth node with the lowest load index. For example, among the multiple non-Nth nodes in a non-overloaded state, the non-Nth node with the lowest load index is determined, and the processing task of the Nth stage is scheduled to the non-Nth node with the lowest load index.

[0147] In this application, in a scenario involving multi-stage processing of input data, for the Nth stage processing task, it is detected whether the Nth node used to process the Nth stage processing task is in an overloaded state. N is a positive integer greater than or equal to 1. In response to the Nth node being overloaded, it is detected whether the non-Nth nodes used to process non-Nth stage processing tasks are in a non-overloaded state. In response to the non-Nth node being non-overloaded, the Nth stage processing task is scheduled to be processed by the non-Nth node.

[0148] If the Nth node is in an overloaded state, it means that the Nth node is heavily loaded. Even if the processing tasks of the Nth stage are scheduled to the Nth node, the Nth node will have difficulty completing the processing tasks of the Nth stage in a short time. This will result in low efficiency of the Nth node in processing the Nth stage tasks and further deteriorate the load situation of the Nth node.

[0149] Therefore, to avoid further deteriorating the load on the Nth node and to improve the efficiency of processing tasks in the Nth stage, the Nth stage processing tasks can be scheduled to nodes that handle other stages instead of the Nth node. If nodes that handle other stages need to process the Nth stage processing tasks, those nodes need to be in a non-overloaded state. To this end, in response to the Nth node being overloaded, it is possible to detect whether nodes that handle other stages are overloaded, for example, by detecting whether non-Nth nodes that handle non-Nth stage processing tasks are in a non-overloaded state. If a non-Nth node is not overloaded, it means that the load on the non-Nth node is light or moderate. The processing tasks of the Nth stage can be scheduled to the non-Nth node. The non-Nth node can complete the processing tasks of the Nth stage in a short time. The non-Nth node has high efficiency in processing the processing tasks of the Nth stage and will not cause the load situation of the non-Nth node to deteriorate further. Therefore, the processing tasks of the Nth stage can be scheduled to the non-Nth node.

[0150] This application enables nodes that are not overloaded and are used to process tasks in one stage to process tasks in another stage. Thus, if a node used to process a task in a certain stage is overloaded, a node that is not overloaded and is used to process tasks in other stages can be used to process tasks in that stage. This fully utilizes the system resources of nodes that are not overloaded and are used to process tasks in other stages, avoids resource waste, improves the overall system resource utilization of the data processing system, and also avoids interruption of the processing task in that stage.

[0151] For any given node, its load metric can be obtained. If the node's load metric is less than a preset metric, it can be determined that the node is not overloaded. Alternatively, if the node's load metric is greater than or equal to the preset metric, it can be determined that the node is overloaded.

[0152] The preset indicators are set in advance by technicians based on the actual situation. This application does not limit the specific value of the preset indicators. For example, they may include 0.7, 0.75, 0.8, 0.85 or 0.9, etc. This application does not limit them.

[0153] When obtaining the load metrics of a node, various parameters of the node can be obtained, such as the number of input data in the Running queue, the number of input data in the Waiting queue, the number of input data in the Swapped queue, the number of input data in the Pending queue, the node's token budget, the node's KV-cache utilization, the node's GPU computing power utilization, and the node's GPU memory bandwidth utilization, etc.

[0154] Alternatively, considering the sudden characteristics of nodes in processing tasks (such as short periods of high load followed by idle time), instantaneous sampling can cause the acquired data to fluctuate wildly, resulting in inaccurate load indicators for the node. Therefore, a sliding window approach can be used to smooth out instantaneous fluctuations.

[0155] For example, the following methods can be used: 1. Obtain the average number of input data in the Running queue of the node over a period of time, and use this average as the total number of input data in the Running queue of the node; 2. Obtain the average number of input data in the Waiting queue of the node over a period of time, and use this average as the total number of input data in the Waiting queue of the node; 3. Obtain the average number of input data in the Swapped queue of the node over a period of time, and use this average as the total number of input data in the Swapped queue of the node; 4. Obtain the average number of input data in the Pending queue of the node over a period of time, and use this average as the total number of input data in the Pending queue of the node; 5. Obtain the average token budget of the node over a period of time, and use this average as the total token budget of the node; 6. Obtain the average KV-cache utilization of the node over a period of time, and use this average as the KV-cache utilization of the node; 7. Obtain the average GPU computing power utilization of the node over a period of time, and use this average as the total GPU computing power utilization of the node; 8. Obtain the average GPU memory bandwidth utilization of the node over a period of time, and use this average as the total GPU memory bandwidth utilization of the node. The duration of a period of time may include 2 seconds, 3 seconds, 4 seconds, or 5 seconds, and the specific duration can be determined according to the actual situation. This application does not limit this.

[0156] In order to effectively measure the load status of each node, after obtaining the parameters of each node, the parameters of each node can be normalized.

[0157] Secondly, the weight values ​​of each parameter can be set in advance according to the actual situation. This application does not limit the specific values ​​of the weight values ​​of each parameter.

[0158] The load metric for a node is obtained by weighting and summing the following parameters: the number of input data in the Running queue, the number of input data in the Waiting queue, the number of input data in the Swapped queue, the number of input data in the Pending queue, the node's token budget, the node's KV-cache utilization, the node's GPU computing power utilization, and the node's GPU memory bandwidth utilization.

[0159] In one embodiment, the processing task of the Nth stage is not the processing task that is ranked as the first stage in the multi-stage processing of input data. For example, in the multi-stage processing of input data, there is a processing task of the (N-1)th stage before the processing task of the Nth stage, and the processing task of the (N-1)th stage has been completed.

[0160] For example, the processing task in stage N is the reasoning task in stage Decode, and the processing task in stage N-1 is the reasoning task in stage Prefill.

[0161] In one scenario, processing the Nth stage task requires the processing result obtained after processing the (N-1)th stage task. This processing result is located in the node that processed the (N-1)th stage task in the historical process. If the node that processes the Nth stage task later (e.g., a non-Nth node) is different from the node that processed the (N-1)th stage task in the historical process, then the node that processes the Nth stage task later needs to obtain the processing result obtained after processing the (N-1)th stage task from the node that processed the (N-1)th stage task in the historical process. This undoubtedly increases the transmission time and consumes transmission resources (such as bandwidth resources).

[0162] Therefore, in this application, the node that processes the Nth stage processing task can be the same as the node that processes the N-1th stage processing task in the historical process.

[0163] Thus, in another embodiment of this application, see... Figure 5 Before executing step S101, which detects whether the Nth node used to process the Nth stage processing task is in an overloaded state, the method further includes: In step S001, for the processing task of the (N-1)th stage in the multi-stage processing of input data, it is determined whether the node that processed the processing task of the (N-1)th stage in the historical process is the Nth node used to process the processing task of the Nth stage.

[0164] In one example, in response to the fact that the node that processed the processing task of stage N-1 in the historical process is the Nth node used to process the processing task of stage N, it is preferred to continue using the Nth node to process the processing task of stage N. Thus, step S101 can be executed: detect whether the Nth node used to process the processing task of stage N is in an overloaded state.

[0165] Alternatively, in another example, in response to the fact that the node that processed the processing task of stage N-1 in the historical process is not the Nth node used to process the processing task of stage N, then the Nth node is not preferred to be used to process the processing task of stage N in the future. Instead, the node that processed the processing task of stage N-1 in the historical process is preferred to continue to be used to process the processing task of stage N in the future. Thus, in step S002, it is possible to detect whether the node that processed the processing task of stage N-1 in the historical process is in a non-overloaded state.

[0166] In this other example, the node that processes the task in the (N-1)th stage in the historical process can be the (N-1)th node used to process the task in the (N-1)th stage, or it can be a non-(N-1)th node used to process the task in a stage other than the (N-1)th stage. The non-(N-1)th node used to process the task in a stage other than the (N-1)th stage is not the Nth node used to process the task in the Nth stage.

[0167] If the node processing the N-1 stage task in the historical process is not overloaded, it indicates that the current load on the node processing the N-1 stage task is light or moderate. The N-1 stage task will then be scheduled to be processed by the node processing the N-1 stage task in the historical process. This ensures that the node processing the N-1 stage task can complete the N-1 stage task in a short time, resulting in high efficiency in processing the N-1 stage task without further deteriorating the load on the node processing the N-1 stage task. Secondly, when a node that processes the N-1 stage processing task in the historical process processes the N stage processing task, it can directly retrieve the processing result obtained after the N-1 stage processing task is completed from the node that processes the N-1 stage processing task in the historical process, without retrieving the processing result obtained after the N-1 stage processing task is completed from other nodes, thereby saving transmission resources. Therefore, in response to the node that processes the N-1 stage processing task in the historical process being in a non-overloaded state, in step S003, the N stage processing task can be scheduled to be processed by the node that processes the N-1 stage processing task in the historical process.

[0168] For example, the global scheduler can schedule the processing task of stage N to the hybrid scheduler in the node that processed the processing task of stage N-1 in the historical process. The hybrid scheduler in the node that processed the processing task of stage N-1 in the historical process can schedule the processing task of stage N to the scheduler of stage N-1 managed by the hybrid scheduler in the node that processed the processing task of stage N-1 in the historical process, so that the scheduler of stage N-1 can process the processing task of stage N.

[0169] Alternatively, if a node processing the N-1 stage in the historical process is overloaded, it indicates that the current load on the node processing the N-1 stage is too heavy. Even if the N-1 stage processing task is rescheduled to the node processing the N-1 stage in the historical process, the node processing the N-1 stage will have difficulty completing the N-1 stage processing task in a short time. This will result in low efficiency for the node processing the N-1 stage in the historical process and further worsen the load situation of the node processing the N-1 stage in the historical process.

[0170] Therefore, in order to avoid further deteriorating the load on the nodes that handle the N-1 stage processing tasks in the historical process, and in order to improve the efficiency of handling the N stage processing tasks, the N stage processing tasks can be handled by default instead of scheduling them to the nodes that handle the N-1 stage processing tasks in the historical process. Thus, step S101 can be executed again: detect whether the Nth node used to handle the N stage processing tasks is overloaded.

[0171] Furthermore, in response to scheduling the processing task of stage N to the non-N node for processing, and the non-N node is not the node that processed the processing task of stage N-1 in the historical process, a first forwarding notification can be sent to the node that processed the processing task of stage N-1 in the historical process. The first forwarding notification is used to notify the non-N node that the node that processed the processing task of stage N-1 in the historical process forwards the processing result obtained by the node that processed the processing task of stage N-1 in the historical process after completing the processing of the processing task of stage N-1, so that the non-N node can use the processing result in the process of processing the processing task of stage N.

[0172] Furthermore, in another embodiment of this application, see... Figure 6 After step S101, in response to the fact that the Nth node is in a non-overloaded state, it indicates that the load of the Nth node is light or moderate. The processing task of the Nth stage is scheduled to the Nth node. The Nth node can complete the processing task of the Nth stage in a short time. The Nth node has high efficiency in processing the processing task of the Nth stage and will not cause the load of the Nth node to deteriorate further. Since the Nth node is the node used to process the processing task of the Nth stage by default, step S201 is executed: the processing task of the Nth stage can be scheduled to the Nth node for processing.

[0173] For example, the global scheduler can schedule the processing tasks of stage N to the hybrid scheduler in stage N, and the hybrid scheduler in stage N can schedule the processing tasks of stage N to the scheduler of stage N managed by the hybrid scheduler in stage N, so that the scheduler of stage N can process the processing tasks of stage N.

[0174] Furthermore, in response to scheduling the processing task of stage N to node N for processing, and node N is not the node that processed the processing task of stage N-1 in the historical process, a second forwarding notification can be sent to the node that processed the processing task of stage N-1 in the historical process. The second forwarding notification is used to notify that the node that processed the processing task of stage N-1 in the historical process forwards to node N the processing result obtained by the node that processed the processing task of stage N-1 in the historical process after completing the processing of the processing task of stage N-1, so that node N can use the processing result in the process of processing the processing task of stage N.

[0175] In another embodiment of this application, see Figure 7 After step S102, in response to the non-Nth node being in an overloaded state, it indicates that the non-Nth node is under heavy load. Even if the processing task of the Nth stage is scheduled to the non-Nth node, it will be difficult for the non-Nth node to complete the processing task of the Nth stage in a short time. This will result in low efficiency of the non-Nth node in processing the processing task of the Nth stage and further deteriorate the load situation of the non-Nth node.

[0176] Therefore, to avoid further deterioration of the load on the Nth node and to improve the efficiency of processing the Nth stage tasks, the Nth stage tasks can be scheduled to non-Nth nodes instead of being processed by the Nth node. However, at this time, the Nth node used to process the Nth stage is overloaded, and the non-Nth nodes used to process non-Nth stages are also overloaded. That is, there are no non-overloaded nodes in the node pool. Therefore, in order to process the Nth stage tasks and improve the processing efficiency of the Nth stage tasks, step S301 can be executed: add a new node, for example, add a new node to the node pool. The new node includes a new Nth node for processing the Nth stage tasks, etc. The new node is set to a non-overloaded state, for example, an idle state, and the Nth stage tasks can be scheduled to the new node for processing.

[0177] For example, the global scheduler can schedule the processing tasks of stage N to the hybrid scheduler in the new node, and the hybrid scheduler in the new node can schedule the processing tasks of stage N to the scheduler of stage N managed by the hybrid scheduler in the new node, so that the scheduler of stage N can process the processing tasks of stage N.

[0178] Secondly, communication connections can be added between new nodes and existing nodes in the node pool so that data can be exchanged between the new nodes and existing nodes in the node pool.

[0179] Secondly, the new nodes can also participate in processing tasks in other stages or other input data processing tasks to improve the overall processing capacity of the data processing system.

[0180] Furthermore, in response to scheduling the processing task of stage N to a new node (the new node is not the node that processed the processing task of stage N-1 in the historical process), a third forwarding notification can be sent to the node that processed the processing task of stage N-1 in the historical process. The third forwarding notification is used to notify the node that processed the processing task of stage N-1 in the historical process to forward the processing result obtained by the node that processed the processing task of stage N-1 in the historical process after completing the processing of the processing task of stage N-1 to the new node, so that the new node can use the processing result in the process of processing the processing task of stage N.

[0181] In this application, after the processing task in the Nth stage of the multi-stage processing of input data is completed, if there is a processing task in the N+1th stage after the Nth stage, then the processing task in the N+1th stage needs to be processed.

[0182] For example, the processing task in stage N is the reasoning task in stage Prefill, and the processing task in stage N+1 is the reasoning task in stage Decode.

[0183] In one scenario, processing the (N+1)th stage requires the processing result obtained after processing the Nth stage. This result is located in the node that processed the Nth stage (e.g., a non-Nth node). If the node processing the (N+1)th stage is different from the node that processed the Nth stage, then the node processing the (N+1)th stage will need to retrieve the processing result from the node that processed the Nth stage. This undoubtedly increases transmission time and consumes transmission resources (e.g., bandwidth).

[0184] Therefore, in this application, the node that processes the task in the (N+1)th stage can be the same as the node that processes the task in the Nth stage.

[0185] Thus, in another embodiment of this application, see... Figure 8In response to scheduling the processing task of the Nth stage to the non-Nth node for processing, in step S401, for the processing task of the N+1th stage in the multi-stage processing of input data, it is preferred to use the non-Nth node that processes the processing task of the Nth stage to process the processing task of the N+1th stage. For this purpose, it is possible to detect whether the non-Nth node is in a non-overload state.

[0186] The process of processing the N+1 stage task follows the process of processing the N stage task. For example, the N+1 stage task is processed after the N stage task is completed.

[0187] In step S102, it is detected that the non-Nth node is in a non-overload state. However, in step S401, it may be detected that the non-Nth node is in a non-overload state or it may be detected that the non-Nth node is in an overload state.

[0188] If the non-Nth node is in a non-overloaded state, it means that the current load of the non-Nth node is light or moderate. The processing task of the N+1 stage is scheduled to be processed by the non-Nth node. The non-Nth node can complete the processing task of the N+1 stage in a short time, which will make the processing efficiency of the non-Nth node in processing the N+1 stage high and will not cause the load of the non-Nth node to deteriorate further. Secondly, in the process of processing the processing task of the N+1 stage, the non-Nth node can directly retrieve the processing result obtained after the processing of the Nth stage processing task, without retrieving the processing result obtained after the processing of the Nth stage processing task from other nodes, thereby saving transmission resources. Therefore, in step S402, the processing task of the N+1 stage can be scheduled to be processed by the non-Nth node.

[0189] Alternatively, if the non-Nth node is in an overloaded state, it means that the current load of the non-Nth node is heavy. Even if the processing task of the N+1 stage is scheduled to the non-Nth node, it will be difficult for the non-Nth node to complete the processing task of the N+1 stage in a short time. This will result in low efficiency of the non-Nth node in processing the processing task of the N+1 stage and further deteriorate the load situation of the non-Nth node.

[0190] Therefore, in order to avoid further deterioration of the load situation of the non-Nth node and to improve the efficiency of processing the N+1th stage task, the Nth stage task can be not scheduled to the non-Nth node for processing. Instead, the N+1th node used for processing the N+1th stage task can be used by default. Thus, in step S403, it can be detected whether the N+1th node used for processing the N+1th stage task is in an overloaded state.

[0191] After executing step S403, the handling method for the processing task in stage N+1 can be referred to Figure 4-7 The handling method for the Nth stage processing task in the illustrated embodiment will not be described in detail here.

[0192] Alternatively, in yet another embodiment of this application, see [link to relevant documentation]. Figure 9 In response to scheduling the processing task of the Nth stage to the node that processed the processing task of the N-1th stage in the historical process, in step S501, for the processing task of the N+1th stage in the multi-stage processing of input data, it is preferred to use the node that processed the processing task of the N-1th stage in the historical process to process the processing task of the N+1th stage. To this end, it is possible to detect whether the node that processed the processing task of the N-1th stage in the historical process is in a non-overloaded state.

[0193] The process of processing the N+1 stage task follows the process of processing the N stage task. For example, the N+1 stage task is processed after the N stage task is completed.

[0194] In step S002, it is detected that the node processing the task of stage N-1 in the historical process is in a non-overloaded state. However, in step S501, it may be detected that the node processing the task of stage N-1 in the historical process is in a non-overloaded state or it may be detected that the node processing the task of stage N-1 in the historical process is in an overloaded state.

[0195] If the node processing the task for stage N-1 in the historical process is not overloaded, it means that the current load on the node processing the task for stage N-1 in the historical process is light or moderate (the node processing the task for stage N-1 in the historical process is the same node as the node processing the task for stage N). The task for stage N+1 is then scheduled to be processed by the node that processed the task for stage N-1 in the historical process. Since the node that processed the task for stage N-1 in the historical process can complete the task for stage N+1 in a short time, this will result in high efficiency for the node that processed the task for stage N-1 in the historical process in processing the task for stage N+1. Furthermore, this will not further worsen the load on the nodes that processed the N-1 stage processing task in the historical process. Secondly, when processing the N+1 stage processing task, the nodes that processed the N-1 stage processing task in the historical process can directly retrieve the processing result obtained after processing the N stage processing task from the nodes that processed the N-1 stage processing task in the historical process, without retrieving the processing result obtained after processing the N stage processing task from other nodes, thereby saving transmission resources. Therefore, in step S502, the N+1 stage processing task can be scheduled to the nodes that processed the N-1 stage processing task in the historical process.

[0196] Alternatively, if the node processing the N-1 stage task in the historical process is overloaded, it means that the node processing the N-1 stage task in the historical process is currently heavily loaded. Even if the N+1 stage task is scheduled to the node processing the N-1 stage task in the historical process, the node processing the N-1 stage task in the historical process will have difficulty completing the N+1 stage task in a short time. This will result in low efficiency for the node processing the N+1 stage task in the historical process and further worsen the load situation of the node processing the N-1 stage task in the historical process.

[0197] Therefore, in order to avoid further deterioration of the load on the nodes that handle the N-1 stage processing tasks in the historical process, and in order to improve the efficiency of handling the N+1 stage processing tasks, the N stage processing tasks can be handled by default instead of scheduling the N stage processing tasks to the nodes that handle the N-1 stage processing tasks in the historical process. Thus, in step S503, it can be detected whether the N+1 node that handles the N+1 stage processing tasks is in an overloaded state.

[0198] After executing step S503, the handling method for the processing task in stage N+1 can be referred to Figure 4-7 The handling method for the Nth stage processing task in the illustrated embodiment will not be described in detail here.

[0199] Alternatively, in yet another embodiment of this application, see [link to relevant documentation]. Figure 10 In response to scheduling the processing task of the Nth stage to the Nth node for processing, in step S601, for the processing task of the N+1th stage in the multi-stage processing of input data, it is preferred to use the Nth node that processes the processing task of the Nth stage to process the processing task of the N+1th stage. For this purpose, it is possible to detect whether the Nth node is in a non-overloaded state.

[0200] The process of processing the N+1 stage task follows the process of processing the N stage task. For example, the N+1 stage task is processed after the N stage task is completed.

[0201] In step S101, it is detected that the Nth node is in an overloaded state. However, in step S601, it is possible that the Nth node is in a non-overloaded state or that the Nth node is in an overloaded state.

[0202] If the Nth node is in a non-overloaded state, it means that the current load of the Nth node is light or moderate. The processing task of the N+1th stage is scheduled to be processed by the Nth node. The Nth node can complete the processing task of the N+1th stage in a short time, which will make the processing efficiency of the N+1th stage high and will not cause the load of the Nth node to deteriorate further. Secondly, in the process of processing the processing task of the N+1th stage, the Nth node can directly retrieve the processing result obtained after the processing of the Nth stage processing task, without retrieving the processing result obtained after the processing of the Nth stage processing task from other nodes, thereby saving transmission resources. Therefore, in step S602, the processing task of the N+1th stage can be scheduled to be processed by the Nth node.

[0203] Alternatively, if the Nth node is overloaded, it means that the Nth node is currently under heavy load. Even if the processing task of the N+1th stage is scheduled to the Nth node, the Nth node will have difficulty completing the processing task of the N+1th stage in a short time. This will result in low efficiency of the Nth node in processing the N+1th stage processing task and further deteriorate the load situation of the Nth node.

[0204] Therefore, in order to avoid further deterioration of the load on the Nth node and to improve the efficiency of processing the N+1th stage tasks, the Nth stage tasks can be handled by default instead of scheduling them to the Nth node. Thus, in step S603, it can be detected whether the N+1th node used to handle the N+1th stage tasks is overloaded.

[0205] After executing step S603, the handling method for the processing task in stage N+1 can be referred to Figure 4-7 The handling method for the Nth stage processing task in the illustrated embodiment will not be described in detail here.

[0206] Alternatively, in yet another embodiment of this application, see [link to relevant documentation]. Figure 11 In response to scheduling the processing task of the Nth stage to a new node for processing, in step S701, for the processing task of the N+1th stage in the multi-stage processing of input data, it is preferred to use the new node that processes the processing task of the Nth stage to process the processing task of the N+1th stage. For this purpose, it is possible to detect whether the new node is in a non-overloaded state.

[0207] The process of processing the N+1 stage task follows the process of processing the N stage task. For example, the N+1 stage task is processed after the N stage task is completed.

[0208] After adding a new node in step S301, the new node is in a non-overloaded state. However, in step S701, it may be detected that the new node is in a non-overloaded state or it may be detected that the new node is in an overloaded state.

[0209] If the new node is in a non-overloaded state, it means that the current load of the new node is light or moderate. The processing task of stage N+1 is scheduled to be processed by the new node. The new node can complete the processing task of stage N+1 in a short time, which will make the processing efficiency of the new node in processing stage N+1 high and will not cause the load of the new node to deteriorate further. Secondly, in the process of processing stage N+1, the new node can directly retrieve the processing result obtained after the processing of stage N is completed, without retrieving the processing result obtained after the processing of stage N is completed from other nodes, thereby saving transmission resources. Therefore, in step S702, the processing task of stage N+1 can be scheduled to be processed by the new node.

[0210] Alternatively, if the new node is overloaded, it means that the new node is currently under heavy load. Even if the processing task of stage N+1 is scheduled to the new node, the new node will have difficulty completing the processing task of stage N+1 in a short time. This will result in low efficiency of the new node in processing stage N+1 and further deteriorate the load situation of the new node.

[0211] Therefore, in order to avoid further deteriorating the load on the new node and to improve the efficiency of processing the N+1 stage task, the N stage task can be handled by default instead of scheduling it to the new node. Thus, in step S703, it can be detected whether the N+1 node used to handle the N+1 stage task is overloaded.

[0212] After executing step S703, the handling method for the processing task in stage N+1 can be referred to Figure 4-7 The handling method for the Nth stage processing task in the illustrated embodiment will not be described in detail here.

[0213] See Figure 12 The following example illustrates the solution of this application, but it is not intended to limit the scope of protection of this application. In this application, the multi-stage processing tasks for input data include: the processing tasks of the Prefill stage and the processing tasks of the Decode stage.

[0214] After the global scheduler receives the processing tasks for the Prefill stage of the input data, it will schedule the processing tasks for the Prefill stage.

[0215] If the Prefill phase processing task is scheduled to node A in the node pool that is not overloaded, the hybrid scheduler in node A will use the Prefill phase scheduler it manages to process the Prefill phase processing task until the KV Cache and the first output token are generated.

[0216] Then, the global scheduler schedules the "processing tasks for the Decode stage of the input data".

[0217] If the processing tasks of the Decode stage are also scheduled to node A in the node pool that is not in an overloaded state, the hybrid scheduler in node A will use a caching mechanism to determine whether the processing tasks of the Prefill stage of the input data were also completed on node A. If it is determined that the processing tasks of the Prefill stage of the input data were also completed on node A, the hybrid scheduler will issue an input data transfer instruction to the scheduler of the Prefill stage it manages. This will cause the scheduler of the Prefill stage managed by the hybrid scheduler to transfer the input data in the Running queue of the Prefill stage scheduler to the Running queue of the Decode stage scheduler according to the transfer instruction. This will allow the scheduler of the Decode stage to continue processing the input data in the Running queue of the Decode stage scheduler to complete the processing tasks of the Decode stage.

[0218] Furthermore, after the scheduler in the Decode stage has finished processing the input data, it can remove the input data from the scheduler in the Decode stage to discard the input data.

[0219] See Figure 13 The following example illustrates the solution of this application, but it is not intended to limit the scope of protection of this application. In this application, the multi-stage processing tasks for input data include: the processing tasks of the Prefill stage and the processing tasks of the Decode stage.

[0220] After the global scheduler receives the processing tasks for the Prefill stage of the input data, it will schedule the processing tasks for the Prefill stage.

[0221] If the Prefill phase processing task is scheduled to node A in the node pool that is not overloaded, the hybrid scheduler in node A will use the Prefill phase scheduler it manages to process the Prefill phase processing task until the KV Cache and the first output token are generated.

[0222] Then, the global scheduler schedules the "processing tasks for the Decode stage of the input data".

[0223] If the processing tasks of the Decode stage are also scheduled to node B in the node pool that is not overloaded, then the hybrid scheduler in node B will temporarily store the processing tasks of the Decode stage in the Pending queue of the Decode stage scheduler through the Decode stage scheduler it manages.

[0224] Secondly, the global scheduler will send a KV Cache transfer request to node A, so that node A will send its generated KV Cache and the first output token to node B. This way, the scheduler in node B, which is managed by the hybrid scheduler, can directly use the KV Cache and the first output token when processing tasks in the Decode stage, without having to generate the KV Cache and the first output token again, thus saving system resources.

[0225] After node A sends its generated KV Cache and first output token to node B, the input data can be deleted from the Running queue in the Prefill scheduler managed by node A's hybrid scheduler.

[0226] Secondly, the hybrid scheduler in node B transfers the input data from the Pending queue of the Decode stage scheduler to the Running queue of the Decode stage scheduler through the Decode stage scheduler it manages, and then processes the input data.

[0227] Furthermore, after the scheduler in the Decode stage has finished processing the input data, it can remove the input data from the scheduler in the Decode stage to discard the input data.

[0228] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by this application.

[0229] Corresponding to the embodiments of the data processing method, this application also provides embodiments of a data processing apparatus.

[0230] Reference Figure 14 The diagram shows a structural block diagram of a data processing apparatus according to this application, the apparatus comprising: The first detection module 11 is used to detect whether the Nth node used to process the Nth stage processing task is in an overload state in a scenario where the input data is processed in multiple stages; N is a positive integer greater than or equal to 1. The second detection module 12 is used to detect whether the non-Nth node used for processing tasks in non-Nth stages is in a non-overload state in response to the Nth node being in an overload state. The first scheduling module 13 is used to schedule the processing task of the Nth stage to the non-Nth node for processing in response to the non-Nth node being in a non-overload state.

[0231] In an alternative implementation, the apparatus further includes: The determination module is used to determine, before detecting whether the Nth node used to process the Nth stage processing task is overloaded, whether the node that processed the N-1th stage processing task in the multi-stage processing of the input data is the Nth node used to process the Nth stage processing task in the historical process for the N-1th stage processing task. The first detection module is further configured to, in response to the fact that the node processing the processing task of the N-1 stage in the historical process is the Nth node, further detect whether the Nth node processing the processing task of the Nth stage is in an overload state. or, The third detection module is used to detect whether the node that processed the processing task of the N-1 stage in the historical process is in a non-overload state in response to the fact that the node that processed the processing task of the N-1 stage in the historical process is not the Nth node. The second scheduling module is used to schedule the Nth stage processing task to the node that processed the N-1 stage processing task in the historical process when the node is in a non-overloaded state. or, The first detection module is further configured to detect whether the Nth node used to process the Nth stage processing task is overloaded in response to the node being overloaded during the historical process.

[0232] In an alternative implementation, the apparatus further includes: The first sending module is configured to, in response to scheduling the processing task of the Nth stage to the non-Nth node for processing, and the non-Nth node is not the node that processed the processing task of the N-1th stage in the historical process, send a first forwarding notification to the node that processed the processing task of the N-1th stage in the historical process. The first forwarding notification is used to notify the node that processed the processing task of the N-1th stage in the historical process to forward the processing result obtained by the node that processed the processing task of the N-1th stage in the historical process after completing the processing of the processing task of the N-1th stage to the non-Nth node.

[0233] In an alternative implementation, the apparatus further includes: The third scheduling module is used to schedule the processing task of the Nth stage to the Nth node for processing in response to the Nth node being in a non-overloaded state.

[0234] In an alternative implementation, the apparatus further includes: The second sending module is configured to, in response to scheduling the processing task of the Nth stage to the Nth node for processing, and the Nth node is not the node that processed the processing task of the N-1th stage in the historical process, send a second forwarding notification to the node that processed the processing task of the N-1th stage in the historical process. The second forwarding notification is used to notify the node that processed the processing task of the N-1th stage in the historical process to forward to the Nth node the processing result obtained by the node that processed the processing task of the N-1th stage in the historical process after completing the processing of the processing task of the N-1th stage.

[0235] In an alternative implementation, the apparatus further includes: The configuration module is used to add a new node in response to the non-Nth node being in an overload state; The fourth scheduling module is used to schedule the processing tasks of the Nth stage to the new node for processing.

[0236] In an alternative implementation, the apparatus further includes: The third sending module is configured to, in response to scheduling the processing task of the Nth stage to the new node for processing, send a third forwarding notification to the node that processed the processing task of the N-1th stage in the historical process. The third forwarding notification is used to notify the node that processed the processing task of the N-1th stage in the historical process to forward to the new node the processing result obtained by the node that processed the processing task of the N-1th stage in the historical process after completing the processing of the processing task of the N-1th stage.

[0237] In an alternative implementation, the apparatus further includes: The fourth detection module is used to respond to scheduling the processing task of the Nth stage to the non-Nth node for processing, and to detect whether the non-Nth node is in a non-overload state for the processing task of the N+1th stage in the multi-stage processing of the input data. The fifth scheduling module is used to schedule the processing tasks of the N+1 stage to the non-N node in response to the non-N node being in a non-overloaded state.

[0238] In an alternative implementation, the apparatus further includes: The fifth detection module is used to respond to scheduling the processing task of the Nth stage to the node processing the processing task of the N-1th stage in the historical process, and to detect whether the node processing the processing task of the N-1th stage in the historical process is in a non-overload state for the processing task of the N+1th stage in the multi-stage processing of the input data. The sixth scheduling module is used to schedule the processing task of the N+1th stage to the node processing task of the N-1th stage in the historical process when the node processing of the N-1th stage processing task is in a non-overloaded state.

[0239] In an alternative implementation, the apparatus further includes: The sixth detection module is used to respond to scheduling the processing task of the Nth stage to the Nth node for processing, and to detect whether the Nth node is in a non-overload state for the processing task of the N+1th stage in the multi-stage processing of the input data. The seventh scheduling module is used to schedule the processing task of the N+1 stage to the Nth node for processing in response to the Nth node being in a non-overloaded state.

[0240] In an alternative implementation, the apparatus further includes: The seventh detection module is used to detect whether the new node is in a non-overload state in response to scheduling the processing task of the Nth stage to the new node for processing in the multi-stage processing of the input data. The eighth scheduling module is used to schedule the processing task of the N+1th stage to the new node for processing in response to the new node being in a non-overloaded state.

[0241] In this application, in a scenario involving multi-stage processing of input data, for the Nth stage processing task, it is detected whether the Nth node used to process the Nth stage processing task is in an overloaded state. N is a positive integer greater than or equal to 1. In response to the Nth node being overloaded, it is detected whether the non-Nth nodes used to process non-Nth stage processing tasks are in a non-overloaded state. In response to the non-Nth node being non-overloaded, the Nth stage processing task is scheduled to be processed by the non-Nth node.

[0242] If the Nth node is in an overloaded state, it means that the Nth node is heavily loaded. Even if the processing tasks of the Nth stage are scheduled to the Nth node, the Nth node will have difficulty completing the processing tasks of the Nth stage in a short time. This will result in low efficiency of the Nth node in processing the Nth stage tasks and further deteriorate the load situation of the Nth node.

[0243] Therefore, to avoid further deteriorating the load on the Nth node and to improve the efficiency of processing tasks in the Nth stage, the Nth stage processing tasks can be scheduled to nodes that handle other stages instead of the Nth node. If nodes that handle other stages need to process the Nth stage processing tasks, those nodes need to be in a non-overloaded state. To this end, in response to the Nth node being overloaded, it is possible to detect whether nodes that handle other stages are overloaded, for example, by detecting whether non-Nth nodes that handle non-Nth stage processing tasks are in a non-overloaded state. If a non-Nth node is not overloaded, it means that the load on the non-Nth node is light or moderate. The processing tasks of the Nth stage can be scheduled to the non-Nth node. The non-Nth node can complete the processing tasks of the Nth stage in a short time. The non-Nth node has high efficiency in processing the processing tasks of the Nth stage and will not cause the load situation of the non-Nth node to deteriorate further. Therefore, the processing tasks of the Nth stage can be scheduled to the non-Nth node.

[0244] This application enables nodes that are not overloaded and are used to process tasks in one stage to process tasks in another stage. Thus, if a node used to process a task in a certain stage is overloaded, a node that is not overloaded and is used to process tasks in other stages can be used to process tasks in that stage. This fully utilizes the system resources of nodes that are not overloaded and are used to process tasks in other stages, avoids resource waste, improves the overall system resource utilization of the data processing system, and also avoids interruption of the processing task in that stage.

[0245] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0246] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0247] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0248] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more methods as described in the above embodiments. In this application, the electronic device includes a server, a gateway, sub-devices, etc., and the sub-devices are devices such as Internet of Things (IoT) devices.

[0249] Embodiments of this disclosure can be implemented as an apparatus with any suitable hardware, firmware, software, or any combination thereof, configured as desired. This apparatus may include electronic devices such as servers (clusters) and terminal devices such as IoT devices.

[0250] Figure 15 An exemplary apparatus 1300 is schematically shown that can be used to implement the various embodiments of this application.

[0251] In one embodiment, Figure 15 An exemplary device 1300 is shown, which includes one or more processors 1302, a control module (chipset) 1304 coupled to at least one of the processors 1302, a memory 1306 coupled to the control module 1304, a non-volatile memory (NVM) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0252] Processor 1302 may include one or more single-core or multi-core processors, and processor 1302 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 1300 can function as a server device such as a gateway in the embodiments of this application.

[0253] In some embodiments, apparatus 1300 may include one or more computer-readable media (e.g., memory 1306 or NVM / storage device 1308) having instructions 1314 and one or more processors 1302 that are combined with the one or more computer-readable media and configured to execute instructions 1314 to implement modules and thus perform the actions in this disclosure.

[0254] In one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1302 and / or any suitable device or component communicating with the control module 1304.

[0255] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0256] Memory 1306 may be used, for example, to load and store data and / or instructions 1314 for device 1300. In one embodiment, memory 1306 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 1306 may include double data rate quad synchronous dynamic random access memory (DDR4 SDRAM).

[0257] In one embodiment, the control module 1304 may include one or more input / output controllers to provide an interface to the NVM / storage device 1308 and (one or more) input / output devices 1310.

[0258] For example, NVM / storage device 1308 may be used to store data and / or instructions 1314. NVM / storage device 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drive (HDD), one or more optical disc (CD) drives, and / or one or more digital universal optical disc (DVD) drives).

[0259] NVM / storage device 1308 may include storage resources that are physically part of a device on which device 1300 is mounted, or that can be accessed by the device without being part of the device. For example, NVM / storage device 1308 may be accessed via a network via one or more input / output devices 1310.

[0260] One or more input / output devices 1310 may provide an interface for device 1300 to communicate with any other suitable device. Input / output devices 1310 may include communication components, pinyin components, sensor components, etc. Network interface 1312 may provide an interface for device 1300 to communicate via one or more networks. Device 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0261] In one embodiment, at least one of the processors 1302 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 1304. In one embodiment, at least one of the processors 1302 may be logically packaged with one or more controllers of the control module 1304 to form a system-in-package (SiP). In one embodiment, at least one of the processors 1302 may be integrated with the logic of one or more controllers of the control module 1304 on the same die. In one embodiment, at least one of the processors 1302 may be integrated with the logic of one or more controllers of the control module 1304 on the same die to form a system-on-a-chip (SoC).

[0262] In various embodiments, device 1300 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, device 1300 may have more or fewer components and / or different architectures. For example, in some embodiments, device 1300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0263] This application provides an electronic device, including: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the electronic device to perform one or more methods as described in this application.

[0264] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0265] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable information processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable information processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0266] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable information processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0267] These computer program instructions can also be loaded onto a computer or other programmable information processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0268] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0269] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0270] The data processing method and apparatus provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, The method includes: In a scenario where input data is processed in multiple stages, for the Nth stage of the multi-stage processing of the input data, it is detected whether the Nth node used to process the Nth stage processing task is in an overload state; N is a positive integer greater than or equal to 1. In response to the Nth node being in an overloaded state, it is detected whether the non-Nth node used to process processing tasks in non-Nth stages is in a non-overloaded state. In response to the non-Nth node being in a non-overloaded state, the processing task of the Nth stage is scheduled to be processed by the non-Nth node.

2. The method according to claim 1, characterized in that, The method further includes: Before detecting whether the Nth node used to process the Nth stage processing task is overloaded, for the N-1th stage processing task in the multi-stage processing of the input data, it is determined whether the node that processed the N-1th stage processing task in the historical process is the Nth node used to process the Nth stage processing task. In response to the fact that the node that handles the processing task of stage N-1 in the historical process is the Nth node, it is then detected whether the Nth node used to handle the processing task of stage N is in an overload state. or, In response to the fact that the node processing the task of stage N-1 in the historical process is not the Nth node, it is detected whether the node processing the task of stage N-1 in the historical process is in a non-overload state. In response to the node that was processing the N-1 stage processing task in the historical process being in a non-overloaded state, the N stage processing task is scheduled to be processed by the node that was processing the N-1 stage processing task in the historical process. or, In response to the node that is processing the N-1 stage processing task in the historical process being overloaded, the Nth node used to process the Nth stage processing task is then checked to see if it is overloaded.

3. The method according to claim 1 or 2, characterized in that, The method further includes: In response to scheduling the processing task of the Nth stage to the non-Nth node for processing, and the non-Nth node is not the node that processed the processing task of the N-1th stage in the historical process, a first forwarding notification is sent to the node that processed the processing task of the N-1th stage in the historical process. The first forwarding notification is used to notify the non-Nth node that the node that processed the processing task of the N-1th stage in the historical process forwards the processing result obtained by the node that processed the processing task of the N-1th stage in the historical process after completing the processing of the processing task of the N-1th stage.

4. The method according to claim 2, characterized in that, The method further includes: In response to scheduling the processing task of the Nth stage to the node that processed the processing task of the N-1th stage in the historical process, for the processing task of the N+1th stage in the multi-stage processing of the input data, it is detected whether the node that processed the processing task of the N-1th stage in the historical process is in a non-overload state. In response to the node processing the N-1 stage processing task being in a non-overloaded state during the historical process, the N+1 stage processing task is scheduled to be processed by the node processing the N-1 stage processing task during the historical process.

5. The method according to claim 1, characterized in that, The method further includes: In response to the fact that the Nth node is in a non-overloaded state, the processing task of the Nth stage is scheduled to be processed by the Nth node.

6. The method according to claim 5, characterized in that, The method further includes: In response to scheduling the processing task of the Nth stage to the Nth node for processing, and the Nth node is not a node that processed the processing task of the N-1th stage in the historical process, a second forwarding notification is sent to the node that processed the processing task of the N-1th stage in the historical process. The second forwarding notification is used to notify the node that processed the processing task of the N-1th stage in the historical process to forward to the Nth node the processing result obtained by the node that processed the processing task of the N-1th stage in the historical process after completing the processing of the processing task of the N-1th stage.

7. The method according to claim 5, characterized in that, The method further includes: In response to scheduling the processing task of the Nth stage to the Nth node for processing, for the processing task of the N+1th stage in the multi-stage processing of the input data, it is detected whether the Nth node is in a non-overload state. In response to the Nth node being in a non-overloaded state, the processing task of the N+1th stage is scheduled to be processed by the Nth node.

8. The method according to claim 1, characterized in that, The method further includes: In response to the non-Nth node being in an overloaded state, a new node is added; The processing task of the Nth stage is scheduled to be processed on the new node.

9. The method according to claim 8, characterized in that, The method further includes: In response to scheduling the processing task of the Nth stage to the new node for processing, a third forwarding notification is sent to the node that processed the processing task of the N-1th stage in the historical process. The third forwarding notification is used to notify the node that processed the processing task of the N-1th stage in the historical process to forward to the new node the processing result obtained by the node that processed the processing task of the N-1th stage in the historical process after completing the processing of the processing task of the N-1th stage.

10. The method according to any one of claims 1, 2, 4-9, characterized in that, The method further includes: In response to scheduling the processing task of the Nth stage to the non-Nth node for processing, for the processing task of the N+1th stage in the multi-stage processing of the input data, it is detected whether the non-Nth node is in a non-overload state. In response to the non-Nth node being in a non-overloaded state, the processing task of the N+1th stage is scheduled to be processed by the non-Nth node.

11. The method according to claim 8, characterized in that, The method further includes: In response to scheduling the processing task of the Nth stage to the new node for processing, for the processing task of the N+1th stage in the multi-stage processing of the input data, it is detected whether the new node is in a non-overload state. In response to the new node being in a non-overloaded state, the processing task of the N+1th stage is scheduled to be processed by the new node.

12. A data processing apparatus, characterized in that, The device includes: The first detection module is used to detect whether the Nth node used to process the Nth stage processing task is in an overload state in a scenario where the input data is processed in multiple stages; N is a positive integer greater than or equal to 1. The second detection module is used to detect whether the non-Nth node used for processing tasks in non-Nth stages is in a non-overload state in response to the Nth node being in an overload state. The first scheduling module is used to schedule the processing task of the Nth stage to the non-Nth node for processing in response to the non-Nth node being in a non-overload state.

13. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, A computer program is stored on a computer-readable storage medium, which, when executed by a processor, implements the method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1 to 11.

Images