Task processing method, question answering method, and task processing platform

Abstract

Provided in the embodiments of the present disclosure are a task processing method, a question answering method, and a task processing platform, the task processing method comprising: receiving task information sent by a client; dispatching the task information to a target parallel processing component, so as to trigger the target parallel processing component to call a task processing model for processing a target task on the basis of the task information; acquiring load information of the target parallel processing component; when it is determined on the basis of the load information that the target parallel processing component is overloaded, dispatching the task information to a target serial processing component, so as to trigger the target serial processing component to call the task processing model for processing the target task on the basis of the task information, overload task information being used for representing task information exceeding a load threshold value of the target parallel processing component.

Description

Task processing methods, question-answering methods, and task processing platforms

[0001] Cross-reference

[0002] This disclosure claims priority to the Chinese patent application filed with the China Patent Office on August 30, 2023, with application number 2023111145713 and invention name “Task processing method, question-and-answer method, and task processing platform,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the field of artificial intelligence technology, and in particular to a task processing method, a question-answering method, and a task processing platform. Background Art

[0004] With the development of machine learning technology, task processing models trained on large-scale samples can process target tasks based on input task information and obtain corresponding task processing results. Due to the large parameter scale of task processing models, they need to be deployed on task processing platforms with higher computing power and storage performance.

[0005] Currently, task processing models can be deployed on parallel processing components, such as graphics processing units (GPUs), neural processing units (NPUs), tensor processing units (TPUs), field programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs). Parallel processing components are scheduled to process target tasks based on task information. Due to their inherent parallel architecture (for example, graphics processing units include multiple sets of parallel caches, control modules, and a large number of arithmetic and logic units (ALUs)), parallel processing components adapt to the high computing power requirements of task processing models and have become the mainstream hardware for task processing model deployment.

[0006] However, since the number of parallel processing components on the task processing platform is often limited, when the parallel processing components are overloaded, processing congestion will occur, which will increase the delay of task processing, reduce the efficiency and availability of task processing, and reduce the user experience of the client. Therefore, a high-efficiency and high-availability task processing method is urgently needed.

[0007] Summary of the Invention

[0008] In view of this, the present disclosure provides a task processing method, a question-and-answer method, and a task processing platform to solve the technical problem of low task processing efficiency on the task processing platform.

[0009] According to a first aspect of the present disclosure, a task processing method is provided, which is applied to a scheduling component in a task processing platform. The task processing platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component, wherein both the parallel processing component and the serial processing component are deployed with a task processing model. The method includes:

[0010] Receive task information sent by the client;

[0011] Dispatching the task information to the target parallel processing component to trigger the target parallel processing component to call the task processing model to process the target task based on the task information;

[0012] Obtain load information of the target parallel processing component;

[0013] When it is determined that the target parallel processing component is overloaded based on the load information, the overload task information is scheduled to the target serial processing component to trigger the target serial processing component to call the task processing model to process the target task based on the task information, wherein the overload task information is used to represent task information that exceeds the load threshold of the target parallel processing component.

[0014] According to a second aspect of the present disclosure, a question-answering method is provided, which is applied to a scheduling component in a question-answering platform. The question-answering platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component, wherein both the parallel processing component and the serial processing component are equipped with a question-answering model. The method includes:

[0015] Receive the question text sent by the client;

[0016] Dispatching the question-answer text to the target parallel processing component to trigger the target parallel processing component to call the question-answer model to process the question-answer task based on the question-answer text and obtain the corresponding reply text;

[0017] Obtain load information of the target parallel processing component;

[0018] When it is determined that the target parallel processing component is overloaded based on the load information, the overloaded question and answer text is dispatched to the target serial processing component to trigger the target serial processing component to call the question and answer model to process the question and answer task based on the question and answer text to obtain the corresponding reply text, wherein the overloaded question and answer text is used to represent the question text that exceeds the load threshold of the target parallel processing component.

[0019] According to a third aspect of the present disclosure, another task processing method is provided, which is applied to a scheduling component in a task processing platform, wherein the task processing platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component, wherein both the parallel processing component and the serial processing component are deployed with an inference model; the method includes:

[0020] Receive an inference request sent by a client, where the inference request carries task information of the inference task;

[0021] Dispatching the task information to the target parallel processing component to trigger the target parallel processing component to call the inference model to process the inference task based on the task information;

[0022] receiving first initial load information reported by the target parallel processing component according to a preset period, adjusting the number of parallel processing component instances on the target parallel processing component based on the first initial load information and task information, and scheduling the task information to the target parallel processing component after resource adjustment;

[0023] Receive load information reported by the target parallel processing component according to a preset period;

[0024] In a case where it is determined based on the load information that the target parallel processing component is overloaded, overload task information is dispatched to the target serial parallel processing component to trigger the target serial processing component to call the inference model to process the inference task based on the overload task information, wherein the overload task information is used to represent task information that exceeds the load threshold of the target parallel processing component;

[0025] receiving second initial load information reported by the target parallel processing component according to a preset period, adjusting the number of serial processing component instances on the target serial processing component based on the second initial load information and the overload task information, and scheduling the overload task information to the target serial processing component after resource adjustment;

[0026] Receive inference results fed back by the parallel processing component and the serial processing component;

[0027] Feedback the inference results to the client.

[0028] According to a fourth aspect of the present disclosure, a task processing device is provided, which is applied to a scheduling component in a task processing platform. The task processing platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component. The parallel processing component and the serial processing component are both deployed with a task processing model. The device includes:

[0029] The first virtual service access point is configured to receive task information sent by the client;

[0030] A first application programming interface is configured to dispatch task information to a target parallel processing component to trigger the target parallel processing component to call a task processing model to process a target task based on the task information;

[0031] A first load information service module is configured to obtain load information of a target parallel processing component;

[0032] The first application programming interface is configured to schedule overloaded task information to a target serial processing component when it is determined based on load information that the target parallel processing component is overloaded, so as to trigger the target serial processing component to call a task processing model to process the target task based on the task information, wherein the overloaded task information is used to represent task information that exceeds a load threshold of the target parallel processing component.

[0033] According to a fifth aspect of the present disclosure, a question-answering device is provided, which is applied to a scheduling component in a question-answering platform. The question-answering platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component, wherein both the parallel processing component and the serial processing component are equipped with a question-answering model. The device includes:

[0034] The second virtual service access point is configured to receive a question text sent by the client;

[0035] The second application programming interface is configured to dispatch the question-answer text to the target parallel processing component, so as to trigger the target parallel processing component to call the question-answer model to process the question-answer task based on the question-answer text and obtain the corresponding reply text;

[0036] a second load information service module, configured to obtain load information of a target parallel processing component;

[0037] The second application programming interface is configured to dispatch the overloaded question and answer text to the target serial processing component when it is determined based on the load information that the target parallel processing component is overloaded, so as to trigger the target serial processing component to call the question and answer model to process the question and answer task based on the question and answer text, and obtain the corresponding reply text, wherein the overloaded question and answer text is used to represent the question text that exceeds the load threshold of the target parallel processing component.

[0038] According to a sixth aspect of the present disclosure, another task processing apparatus is provided, which is applied to a scheduling component in a task processing platform. The task processing platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component, wherein both the parallel processing component and the serial processing component are deployed with an inference model. The method includes:

[0039] A first receiving component is configured to receive an inference request sent by a client, wherein the inference request carries task information of an inference task;

[0040] A first scheduling component is configured to schedule task information to a target parallel processing component to trigger the target parallel processing component to call an inference model to process an inference task based on the task information;

[0041] a second receiving component configured to receive first initial load information reported by the target parallel processing component according to a preset period, adjust the number of parallel processing component instances on the target parallel processing component based on the first initial load information and task information, and schedule the task information to the target parallel processing component after resource adjustment;

[0042] a third receiving component configured to receive load information reported by the target parallel processing component according to a preset period;

[0043] a second scheduling component configured to, when determining based on the load information that the target parallel processing component is overloaded, schedule overloaded task information to the target serial parallel processing component, so as to trigger the target serial processing component to call the inference model to process the inference task based on the overloaded task information, wherein the overloaded task information is used to represent task information that exceeds a load threshold of the target parallel processing component;

[0044] a fourth receiving component configured to receive second initial load information reported by the target parallel processing component according to a preset period, adjust the number of serial processing component instances on the target serial processing component based on the second initial load information and the overload task information, and schedule the overload task information to the target serial processing component after resource adjustment;

[0045] a fifth receiving component, configured to receive inference results fed back by the parallel processing component and the serial processing component;

[0046] The feedback component is configured to feed back the inference results to the client.

[0047] According to a seventh aspect of the present disclosure, there is provided a task processing platform, comprising a scheduling component, at least one parallel processing component, and at least one serial processing component, wherein both the parallel processing component and the serial processing component are deployed with a task processing model;

[0048] a scheduling component configured to receive task information sent by a client, schedule the task information to a target parallel processing component, obtain load information of the target parallel processing component, and schedule the overloaded task information to a target serial processing component when determining that the target parallel processing component is overloaded based on the load information;

[0049] The target parallel processing component is configured to receive task information scheduled by the scheduling component, call the task processing model, and process the target task based on the task information;

[0050] The target serial processing component is configured to receive the overload task information scheduled by the scheduling component, call the task processing model, and process the target task based on the overload task information.

[0051] According to an eighth aspect of the present disclosure, a computer program product is provided, which includes a computer program that implements the steps of the method described in any one of the first to third aspects when executed by a processor.

[0052] According to a ninth aspect of the present disclosure, a computer program product is provided, which includes a non-volatile computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the steps of the method described in any one of the first to third aspects above.

[0053] According to a tenth aspect of the present disclosure, a computer program product is provided, which, when executed by a processor, implements the steps of the method described in any one of the first to third aspects above.

[0054] According to an eleventh aspect of the present disclosure, there is provided a computing device, including:

[0055] memory and processor;

[0056] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of the first to third aspects above are implemented.

[0057] According to the twelfth aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the method described in any one of the first to third aspects above are implemented.

[0058] In the present disclosure, a task processing method is provided, which is applied to a scheduling component in a task processing platform, wherein the task processing platform includes a scheduling component, at least one parallel processing component and at least one serial processing component, and both the parallel processing component and the serial processing component are deployed with a task processing model; the method includes: receiving task information sent by a client; scheduling the task information to a target parallel processing component to trigger the target parallel processing component to call the task processing model to process the target task based on the task information; obtaining the load information of the target parallel processing component; and in the case where it is determined that the target parallel processing component is overloaded based on the load information, scheduling the overloaded task information to the target serial processing component to trigger the target serial processing component to call the task processing model to process the target task based on the overloaded task information, wherein the overloaded task information is used to represent task information that exceeds the load threshold of the target parallel processing component. After receiving the task information sent by the client, the task information is scheduled to the target parallel processing component first, and the task processing model is called to perform efficient task processing based on the task information. Based on the load information of the target parallel processing component, the load situation of the target parallel processing component is monitored. In the event of overload, the overloaded task information is promptly scheduled to the target serial processing component, and the task processing model is called to perform stable task processing based on the overloaded task information, avoiding congestion problems, reducing the delay of task processing, improving the efficiency and availability of task processing, and thereby improving the user experience of the client. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0060] FIG1 is a flowchart of a task processing method provided by an embodiment of the present disclosure;

[0061] FIG2 is a flow chart of a question-answering method provided by an embodiment of the present disclosure;

[0062] FIG3 is an architecture diagram of a task processing platform in a task processing method provided by an embodiment of the present disclosure;

[0063] FIG4 is a flowchart of a processing process of a task processing method applied to a reasoning scenario provided by an embodiment of the present disclosure;

[0064] FIG5 is a schematic structural diagram of a task processing device provided by an embodiment of the present disclosure;

[0065] FIG6 is a schematic structural diagram of a question-answering device provided by an embodiment of the present disclosure;

[0066] FIG7 is a schematic diagram of the structure of a task processing platform provided by an embodiment of the present disclosure;

[0067] FIG8 is a structural block diagram of a computing device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0068] The following description sets forth many specific details to facilitate a full understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific implementations disclosed below.

[0069] The terms used in one or more embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present disclosure. The singular forms "a", "the", and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0070] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0071] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0072] In one or more embodiments of the present disclosure, a large model refers to a deep learning model with large-scale model parameters, which typically contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a cornerstone model / foundation model. The large model is pre-trained using large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as a large-scale language model (LLM) and a multi-modal pre-training model.

[0073] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0074] First, the terms involved in one or more embodiments of the present disclosure are explained.

[0075] Elastic scaling: In a computer system, the system resources (such as computing resources, storage resources, network bandwidth, etc.) are automatically adjusted according to the system load to adapt to the current load demand, thereby improving the system's availability and performance.

[0076] Overload awareness: refers to the system's ability to identify and handle load overload situations to ensure system availability and performance.

[0077] Fallback: A strategy used by deep learning models during task processing. When a task processing error or failure occurs, the system automatically switches to a fallback deep learning model to ensure the accuracy and reliability of the task processing results.

[0078] Physical Service: A physical service corresponds to a real minimum deployment and release component. Physical services can be distributed in different clusters, and clusters can be distributed in different regions. A physical service uses the same hardware model.

[0079] A virtual service (Visual Service) is a component composed of one or more physical services, providing a unified access point to the outside world. The same model can be deployed on one or more physical services, using heterogeneous hardware models distributed in different clusters in different regions.

[0080] Queries Per Second (QPS): A measure of the amount of traffic processed by a specific traffic node within a specified time.

[0081] Currently, as the mainstream hardware for deploying task processing models, online real-time task processing requires the deployment of a large number of parallel processing components. When performing elastic scaling of parallel processing components, real-time task processing will be unable to provide sufficient parallel processing components as expected due to the limited number of parallel processing components. This will cause congestion in real-time task processing, increase the delay of task processing, and reduce the efficiency and availability of task processing.

[0082] To address this issue, a more abundant number of serial processing components, such as central processing units (CPUs), can be used to replace parallel processing components to deploy task processing models. This avoids congestion and, overall, reduces task processing latency and improves task processing availability. However, due to the serial architecture of the serial processing components themselves (for example, the CPU includes a cache, a control module, and a small number of arithmetic logic components), the latency of individual task processing is high, reducing task processing efficiency. How to rationally utilize the high efficiency of a limited number of parallel processing components and the low efficiency of an abundant number of serial processing components to improve the efficiency and availability of task processing is an urgent problem that needs to be solved.

[0083] To address this problem, the present disclosure provides a task processing method. The present disclosure also involves a question-and-answer method, a task processing device, a question-and-answer device, a task processing platform, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

[0084] Referring to FIG1 , FIG1 shows a flowchart of a task processing method provided by an embodiment of the present disclosure. The method is applied to a scheduling component in a task processing platform. The task processing platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component. Both the parallel processing component and the serial processing component are deployed with a task processing model. The method includes the following specific steps:

[0085] Step 102: Receive task information sent by the client.

[0086] The task processing platform is a remote service platform (server) with task processing functions. The task processing platform can provide users with task processing services through virtual service clients in the form of applications, web pages, applets, etc. A network connection is established between the task processing platform and the client. The virtual service of the task processing platform provides a unified service access point to the outside world, such as an application programming interface (API), a socket interface or other forms of network connection. The task processing platform includes at least one parallel processing component and at least one serial processing component physical service. Generally, a physical service uses the same model of parallel processing components or serial processing components. At least one parallel processing component is generally of different models, and the same is true for serial processing components, and they are distributed in different areas, presenting heterogeneous models. The task processing platform also includes a scheduling component, which schedules the task information sent by the client to the parallel processing component or the serial processing component.

[0087] The scheduling component is a functional component that performs task information scheduling. The implementation of scheduling is based on load monitoring of the parallel processing components and serial processing components on the task processing platform. The scheduling component has the functions of receiving task information, task information scheduling, load monitoring, and elastic scaling. Correspondingly, the scheduling component includes a unified virtual service access point (Unified Access Point) for receiving task information, an application programming interface server (API Server) for task information scheduling, a load information service module (Load Metrics Server) for load monitoring, and a high-performance computing control module (High Performance Architecture Controller, or HPA Controller) for elastic scaling. The application programming interface defines a virtual service traffic distribution strategy: a priority scheduling strategy, which will receive the client's task information, which is the network data traffic, and perform corresponding scheduling. Specifically, the overload judgment is achieved through the dynamic load awareness module (Dynamic Load Aware).

[0088] Parallel processing components are priority scheduling objects pre-set in the priority scheduling strategy and are physical services, including but not limited to: graphics processing components, neural processing components, tensor processing components, FPGA hardware, and ASIC hardware. Serial processing components are backup scheduling objects pre-set in the priority scheduling strategy and are physical services, including but not limited to central processing components. A single parallel processing component and central processing component can be regarded as a task processing node. In addition to parallel processing components and serial processing components, other software and hardware resources must be configured, such as the model running environment (operating system, driver), scheduling interface, memory, database, etc.

[0089] The task processing model is a pre-trained deep learning model with task processing capabilities. The task processing model is deployed in parallel processing components and serial processing components. The model types and model parameters of the task processing models deployed on different parallel processing components and serial processing components can be the same or different. At the data level, the task processing model performs feature encoding on the scheduled task information to obtain the feature vector, and then performs a series of feature transformations to obtain the corresponding output vector, which is then decoded to obtain the corresponding task processing result. Task processing models are generally divided into three types according to their structure: encoding, decoding, and encoding-decoding. The large-scale language model (LLM) is used as an example in the disclosed embodiment for illustration.

[0090] Task information is the input information for the target task to be processed. Target tasks can be of different types, and accordingly, task information has different information modalities. For example, if the target task is a speech stream translation task, the task information is the user-input speech stream data; if the target task is a style transfer task, the task information is the user-input initial image and reference image. Task information is typically sent and received via task processing requests, which carry the target task's task information.

[0091] Receiving task information sent by a client, specifically by receiving the task information sent by the client through a virtual service access point, where the virtual service access point is a unified service access point preset for different clients on the task processing platform. After receiving the task information sent by the client, the task information can be stored in a queue, divided into batches according to a preset batch size for subsequent scheduling, or further scheduled in the form of real-time traffic. In the disclosed embodiments, the real-time traffic form is used for illustration.

[0092] For example, the task processing platform is a web service platform for large-scale language models. This web service platform includes a scheduling component, 100 graphics processing components, and 100 central processing components. The scheduling component includes a unified virtual service access point, an application programming interface (API) server, a load information service module, and a high-performance computing control module. The API server includes a dynamic load sensing module. The unified virtual service access point receives task processing requests from 3,000 clients. Each task processing request carries task information for the 3,000 target tasks, constituting real-time traffic.

[0093] Receive task information sent by the client, laying the foundation for subsequent scheduling processing.

[0094] Step 104: dispatching the task information to the target parallel processing component to trigger the target parallel processing component to call the task processing model to process the target task based on the task information.

[0095] The target parallel processing component is a priority scheduling object corresponding to the task information in at least one parallel processing component and is a physical service. The at least one parallel processing component is distributed in different areas, and the target parallel processing component can be determined from the at least one parallel processing component based on the network address of the client. The at least one parallel processing component is of different models and corresponds to different task processing performance. The target parallel processing component can also be determined from the at least one parallel processing component based on the task type of the target task corresponding to the task information. The at least one parallel processing component is of different models and corresponds to different task processing performance. The target parallel processing component can also be determined from the at least one parallel processing component based on the amount of information in the task information.

[0096] On the target parallel processing component, task information is input into the task processing model, and the target task is executed based on the task information to obtain the task processing result. Optionally, the target parallel processing component first feeds the task processing result back to the scheduling end, and then feeds it back to the corresponding client through the unified virtual service access point on the scheduling end.

[0097] The task information is dispatched to the target parallel processing component by, for example, dispatching the task information to the target parallel processing component through an application programming interface, and more specifically, dispatching the task information to the target parallel processing component through a priority scheduling strategy on the application programming interface.

[0098] For example, the real-time traffic containing task information of 3,000 target tasks is connected to the application programming interface. Through the priority scheduling strategy on the application programming interface: priority scheduling object: physical service 1 (target graphics processing component); backup scheduling object: physical service 2 (target central processing component), the real-time traffic containing task information of 3,000 target tasks is scheduled to physical service 1. On physical service 1, the 3,000 task information is input into the large-scale language model of the target graphics processing component, and the corresponding target tasks are executed based on the task information.

[0099] The task information is dispatched to the target parallel processing component to trigger the target parallel processing component to call the task processing model to process the target task based on the task information. The task information is dispatched to the target parallel processing component first, and the task processing model is called to perform efficient task processing based on the task information, thereby improving the efficiency of task processing.

[0100] Step 106: Obtain load information of the target parallel processing component.

[0101] The load information of the target parallel processing component is information about the load situation on the target parallel processing component, which is real-time information obtained through monitoring, including but not limited to: central processing component parameters (CPU parameters), storage parameters, latency, operating frequency, query rate per second, number of concurrent requests, number of batches (Batch), real-time data size (Size), and queue data size (Queue Size).

[0102] Obtaining load information of a target parallel processing component, specifically by obtaining monitored load information of the target parallel processing component. The monitoring method may include sending a load request to the target parallel processing component and receiving load information fed back by the target parallel processing component, or receiving load information reported by the target parallel processing component, such as triggered reporting or timed reporting, without limitation.

[0103] After the load information of the target parallel processing component is obtained, it is recorded in the load information table of the load information service module, and the load information of the target parallel processing component in the load information table is updated.

[0104] For example, the load information reported by physical service 1 (target parallel processing component) is received: CPU utilization: 67%; storage utilization: 100%; delay: 34ms; operating frequency: 1.8GHz..., and the load information of physical service 1 is recorded in the load information table of the load information service module.

[0105] Obtaining the load information of the target parallel processing component lays the foundation for determining whether the target parallel processing component is overloaded.

[0106] Step 108: When it is determined based on the load information that the target parallel processing component is overloaded, the overloaded task information is dispatched to the target serial processing component to trigger the target serial processing component to call the task processing model to process the target task based on the overloaded task information.

[0107] The target serial processing component is a physical service and is a backup scheduling object corresponding to task information in at least one serial processing component. Generally, a target serial processing component corresponds to a target parallel processing component. This means that the serial processing component corresponding to each parallel processing component is pre-determined and set as the priority scheduling object and backup scheduling object, respectively, in the priority scheduling policy.

[0108] On the target serial processing component, the task information is input into the task processing model, and the target task is executed based on the task information to obtain the task processing result. Optionally, the target serial processing component first feeds the task processing result back to the scheduling end, and then feeds it back to the corresponding client through the unified virtual service access point on the scheduling end.

[0109] Overloaded task information is task information that exceeds the load threshold of the target parallel processing component. For example, the task information is images input by 1,000 users, where the load threshold of the target parallel processing component is 8,000 images. The 2,000 images that exceed the load threshold are overloaded task information.

[0110] Determining whether the target parallel processing component is overloaded based on the load information is as follows: determining whether the target parallel processing component is overloaded based on the load information and a load threshold. Further, obtaining the load information through a dynamic load sensing module and determining whether the target parallel processing component is overloaded based on the load information and the load threshold.

[0111] The overload task information is dispatched to the target serial processing component, specifically by dispatching the overload task information to the target serial processing component through an application programming interface. Specifically, the overload task information is dispatched to the target serial processing component through a priority scheduling strategy on the application programming interface.

[0112] Exemplarily, load information is obtained through the dynamic load perception module, and based on the load information and load thresholds: CPU usage threshold: 65%; storage usage threshold: 90%; delay: 25ms; operating frequency: 1.5GHz..., it is determined that the target graphics processing component of physical service 1 is overloaded, and through the priority scheduling strategy on the application programming interface, the real-time traffic containing the overload task information of 1,000 target tasks is scheduled to physical service 2. On physical service 2, the 1,000 overload task information is input into the large-scale language model of the target central processing component, and the corresponding target task is executed based on the task information.

[0113] In an embodiment of the present disclosure, a task processing method is provided, which is applied to a scheduling component in a task processing platform, wherein the task processing platform includes a scheduling component, at least one parallel processing component and at least one serial processing component, and both the parallel processing component and the serial processing component are deployed with a task processing model; the method includes: receiving task information sent by a client; scheduling the task information to a target parallel processing component to trigger the target parallel processing component to call the task processing model to process the target task based on the task information; obtaining load information of the target parallel processing component; and when it is determined that the target parallel processing component is overloaded based on the load information, scheduling the overloaded task information to the target serial processing component to trigger the target serial processing component to call the task processing model to process the target task based on the overloaded task information. After receiving the task information sent by the client, the task information is scheduled to the target parallel processing component first, and the task processing model is called to perform efficient task processing based on the task information. Based on the load information of the target parallel processing component, the load situation of the target parallel processing component is monitored. In the event of overload, the overloaded task information is promptly scheduled to the target serial processing component, and the task processing model is called to perform stable task processing based on the overloaded task information, avoiding congestion problems, reducing the delay of task processing, improving the efficiency and availability of task processing, and thereby improving the user experience of the client.

[0114] In an optional embodiment of the present disclosure, before step 106, the following specific steps are further included:

[0115] Acquire first initial load information of the target parallel processing component, wherein the first initial load information is load information after the task information is scheduled to the target parallel processing component;

[0116] Adjusting resources of a target parallel processing component based on the first initial load information and the task information;

[0117] Schedule the task information to the target parallel processing component after resource adjustment.

[0118] By combining parallel and serial processing components, the high efficiency of a limited number of parallel processing components and the low efficiency of an abundant number of serial processing components are leveraged, achieving highly efficient and highly available task processing. However, this approach requires pre-planning the required parallel processing components and configuring their resources before receiving task information sent by the client. When task information triggers a high load, insufficient resources in the pre-planned parallel processing components can lead to underutilization of the available parallel processing components. For example, if 70% of the graphics processing components are pre-planned for task processing, the task information triggers a high load, leaving 70% of the graphics processing components unused and 30% unused. When task information triggers a low load, excess resources in the pre-planned graphics processing components can lead to underutilization of the already configured graphics processing components. For example, if 70% of the graphics processing components are pre-planned for task processing, the task information triggers a low load, leaving only 10% of the 70% unused, leaving 70% unused.

[0119] The first initial load information is the load information on the target parallel processing component after the task information is dispatched to the target parallel processing component. It is real-time information obtained by monitoring before resource adjustment and serves as a basis for adjusting resources of the target parallel processing component.

[0120] The first initial load information of the target parallel processing component is obtained by: obtaining the first initial load information of the target parallel processing component obtained through monitoring. The monitoring method is as described in step 106 above.

[0121] Based on the first initial load information and the task information, resources of the target parallel processing component are adjusted. Specifically, based on the first initial load information, it is determined that the target parallel processing component is overloaded, and based on the task information, resources of the target parallel processing component are increased; or based on the first initial load information, it is determined that the target parallel processing component is underloaded, and based on the task information, resources of the target parallel processing component are reduced.

[0122] It should be noted that the steps of obtaining the first initial load information of the target parallel processing component and adjusting the resources of the target parallel processing component based on the first initial load information and the task information are both implemented by the high performance computing control module.

[0123] The task information is dispatched to the target parallel processing component after resource adjustment, specifically by dispatching the task information to the target serial processing component after resource adjustment through an application programming interface.

[0124] In the embodiment of the present disclosure, the above steps may be specifically implemented in a loop, that is, after the task information is dispatched to the target parallel processing component after resource adjustment, the first initial load information is reacquired. If resource adjustment is still required, the adjustment is continued until there are insufficient resources for adjustment on the target parallel processing component. It can be finally determined that the target parallel processing component is overloaded, that is, step 106 is executed, and step 108 is started.

[0125] For example, the first initial load information reported by physical service 1 (target graphics processing component) is received: CPU utilization: 82%; storage utilization: 100%; delay: 34ms; operating frequency: 1.8GHz..., through the high-performance computing control module, based on the first initial load information, it is determined that the target graphics processing component is overloaded, and based on the task information, the resources of the target graphics processing component are increased, and the real-time traffic containing the task information of 3000 target tasks is scheduled to the physical service 1 after resource adjustment.

[0126] Obtaining first initial load information for a target parallel processing component, where the first initial load information is the load information after dispatching task information to the target parallel processing component; adjusting resources for the target parallel processing component based on the first initial load information and the task information; and dispatching the task information to the target parallel processing component after resource adjustment. By adjusting the resources of the target parallel processing component through elastic scaling, the performance of the target parallel processing component is fully utilized.

[0127] In an optional embodiment of the present disclosure, adjusting resources of a target parallel processing component based on the first initial load information and task information includes the following specific steps:

[0128] Based on the first initial load information and the task information, the number of parallel processing component instances on the target parallel processing component is adjusted.

[0129] Currently, parallel processing components use a preset number of instantiable resources. During actual use, they set up no more than the preset number of parallel processing component instances (e.g., GPU instances) on demand, allocating independent high-bandwidth storage, cache, and arithmetic logic components to each of them. This reduces task processing latency and improves task processing concurrency through parallel processing. In the disclosed embodiments, resource adjustment is achieved by adjusting the number of set parallel processing component instances.

[0130] The parallel processing component instance is a virtual service instance on the parallel processing component. Each image processing instance has corresponding software and hardware resources and runs independently of each other.

[0131] Based on the first initial load information and the task information, the number of parallel processing component instances on the target parallel processing component is adjusted. Specifically, based on the first initial load information, it is determined that the target parallel processing component is overloaded, and based on the task information, the number of parallel processing component instances on the target parallel processing component is increased; or, based on the first initial load information, it is determined that the target parallel processing component is underloaded, and based on the task information, the number of parallel processing component instances on the target parallel processing component is reduced.

[0132] It should be noted that the step of adjusting the number of parallel processing component instances on the target parallel processing component based on the first initial load information and the task information is implemented by the high performance computing control module.

[0133] Exemplarily, through the high-performance computing control module, based on the first initial load information, it is determined that the target graphics processing component is overloaded, and based on the task information, the number of graphics processing component instances on the target graphics processing component is increased: 2 instances are increased to 6 instances, and the real-time traffic containing the task information of 3,000 target tasks is scheduled to the physical service 1 after resource adjustment.

[0134] Based on the first initial load information and task information, the number of parallel processing component instances on the target parallel processing component is adjusted. The number of resource instances of the parallel processing component is adjusted in an elastic scaling manner, thereby fully utilizing the performance of the target parallel processing component.

[0135] In an optional embodiment of the present disclosure, step 108 includes the following specific steps:

[0136] Identifying whether the load information reaches a first load threshold preset for the target parallel processing component;

[0137] If so, the overload task information is dispatched to the target serial processing component.

[0138] The first load threshold is a pre-set load threshold used to identify whether the target parallel processing component is overloaded, including but not limited to: a serial processing component parameter threshold (e.g., a CPU parameter threshold), a storage parameter threshold, a latency threshold, an operating frequency threshold, a query rate per second threshold, a concurrent number threshold, a batch number threshold, a real-time data size threshold, and a queue data size threshold.

[0139] If the load information is identified to have reached a first load threshold, the target parallel processing component is determined to be overloaded and backup processing is required, i.e., the overloaded task information is dispatched to the target serial processing component. If the load information is identified to have not reached the first load threshold, the target parallel processing component is determined to be under normal load and backup processing is not required, i.e., task processing continues on the target parallel processing component.

[0140] It should be noted that the step of identifying whether the load information reaches a first load threshold preset for the target parallel processing component is implemented by the dynamic load sensing module.

[0141] Exemplarily, through the dynamic load perception module, load information is obtained from the load information table of the load information service module, and it is identified whether the load information reaches the first load threshold preset for the target graphics processing component: CPU usage threshold: 65%; storage usage threshold: 90%; delay: 25ms; operating frequency: 1.5GHz..., it is determined that the target graphics processing component of physical service 1 is overloaded, and through the priority scheduling strategy on the application programming interface, the real-time traffic containing the overload task information of 1,000 target tasks is scheduled to physical service 2.

[0142] Identify whether the load information reaches a first load threshold preset for the target parallel processing component; if so, dispatch the overloaded task information to the target serial processing component. This enables accurate identification of whether the target parallel processing component is overloaded and, if overloaded, timely dispatch of the overloaded task information to the target serial processing component.

[0143] In an optional embodiment of the present disclosure, after step 108, the following specific steps are further included:

[0144] Acquire second initial load information of the target serial processing component, wherein the second initial load information is load information after the overload task information is dispatched to the target serial processing component;

[0145] Adjusting resources of the target serial processing component based on the second initial load information and the overload task information;

[0146] Schedule the overloaded task information to the target serial processing component after resource adjustment.

[0147] By combining parallel and serial processing components, the high efficiency of a limited number of parallel processing components and the low efficiency of an abundant number of serial processing components are leveraged, achieving highly efficient and highly available task processing. However, this approach requires pre-planning the required serial processing components and configuring the corresponding serial processing component resources before receiving overload task information from the client. When the overload task information triggers a high load, insufficient resources are available for the pre-planned serial processing components, and the available serial processing components are not fully utilized. For example, if 70% of the serial processing components are pre-planned for task processing, the overload task information triggers a high load, and 70% of the serial processing components are insufficient, while 30% of the pre-planned serial processing components are not fully utilized. When the overload task information triggers a low load, the pre-planned serial processing components are underutilized due to an excess of resources. For example, if 70% of the serial processing components are pre-planned for task processing, and the overload task information triggers a low load, only 10% of the 70% are utilized, leaving 70% of the serial processing components underutilized.

[0148] The second initial load information is the load information on the target serial processing component after the overload task information is dispatched to the target serial processing component. It is the real-time information obtained by monitoring before resource adjustment and serves as the basis for adjusting resources of the target serial processing component.

[0149] The second initial load information of the target serial processing component is obtained by: obtaining the second initial load information of the target serial processing component obtained through monitoring. The monitoring method can refer to the above step 106.

[0150] Based on the second initial load information and the overload task information, resources of the target serial processing component are adjusted. Specifically, based on the second initial load information, it is determined that the target serial processing component is overloaded, and based on the overload task information, the resources of the target serial processing component are increased; or, based on the second initial load information, it is determined that the target serial processing component is underloaded, and based on the overload task information, the resources of the target serial processing component are reduced.

[0151] It should be noted that the steps of obtaining the second initial load information of the target serial processing component and adjusting the resources of the target serial processing component based on the second initial load information and the overload task information are both implemented by the high performance computing control module.

[0152] The overload task information is dispatched to the target serial processing component after resource adjustment, specifically in the following manner: the overload task information is dispatched to the target serial processing component after resource adjustment through an application programming interface.

[0153] In an embodiment of the present disclosure, the above steps may be specifically implemented in a loop, that is, after the overload task information is scheduled to the target serial processing component after resource adjustment, the second initial load information is reacquired. If resource adjustment is still required, the adjustment is continued until there are insufficient resources for adjustment on the target serial processing component.

[0154] For example, the second initial load information reported by physical service 2 (target central processing component) is received: CPU utilization: 13%; storage utilization: 11%; delay: 2ms; operating frequency: 0.2GHz..., through the high-performance computing control module, based on the second initial load information, it is determined that the target central processing component is underloaded, and based on the overload task information, the resources of the target central processing component are reduced, and the real-time traffic containing the overload task information of 1,000 target tasks is scheduled to the physical service 2 after resource adjustment.

[0155] The system obtains second initial load information for the target serial processing component; adjusts resources for the target serial processing component based on the second initial load information and the overloaded task information; and dispatches the overloaded task information to the target serial processing component after resource adjustment. By adjusting the resources of the target serial processing component through elastic scaling, the performance of the target serial processing component is fully utilized.

[0156] In an optional embodiment of the present disclosure, adjusting resources of a target serial processing component based on the second initial load information and the overload task information includes the following specific steps:

[0157] Based on the second initial load information and the overload task information, the number of serial processing component instances on the target serial processing component is adjusted.

[0158] Currently, serial processing components use a preset number of instantiable resources to set up no more than a preset number of serial processing component instances (e.g., CPU instances) as needed during actual use, allocating independent high-bandwidth storage, cache, and arithmetic logic components to each of them. This reduces task processing latency and improves task processing concurrency through parallel processing. In the disclosed embodiments, resource adjustment is achieved by adjusting the number of serial processing component instances.

[0159] The serial processing component instance is a virtual service instance on the serial processing component. Each central processing instance has corresponding software and hardware resources and runs independently of each other.

[0160] Based on the second initial load information and the overload task information, the number of serial processing component instances on the target serial processing component is adjusted. Specifically, based on the second initial load information, it is determined that the target serial processing component is overloaded, and based on the overload task information, the number of serial processing component instances on the target serial processing component is increased; or, based on the second initial load information, it is determined that the target serial processing component is underloaded, and based on the overload task information, the number of serial processing component instances on the target serial processing component is reduced.

[0161] It should be noted that the step of adjusting the number of serial processing component instances on the target serial processing component based on the second initial load information and the overload task information is implemented by the high performance computing control module.

[0162] Exemplarily, through the high-performance computing control module, based on the second initial load information, it is determined that the target central processing component is underloaded, and based on the overload task information, the number of central processing component instances on the target central processing component is reduced: 4 instances are reduced to 1 instance, and the real-time traffic containing the overload task information of 1,000 target tasks is scheduled to the physical service 2 after resource adjustment.

[0163] Based on the second initial load information and the overload task information, the number of serial processing component instances on the target serial processing component is adjusted. The number of resource instances of the serial processing component is adjusted by elastic scaling, thereby fully utilizing the performance of the target serial processing component.

[0164] In an optional embodiment of the present disclosure, step 106 includes the following specific steps:

[0165] Receive load information reported by the target parallel processing component according to a preset period.

[0166] The preset period is a preset period for reporting the load information of the target parallel processing component. For example, it is preset that the target parallel processing component reports its own load information every 20 ms.

[0167] The load information reported by the target parallel processing component according to a preset period is received, specifically in the following manner: the load information reported by the target parallel processing component according to the preset period is received through the load information service module.

[0168] After receiving the load information reported by the target parallel processing component according to the preset period, the load information is recorded in the load information table of the load information service module, and the load information of the target parallel processing component in the load information table is updated.

[0169] For example, the load information reported by physical service 1 (target graphics processing component) according to a preset period of 20ms is received: CPU utilization: 67%; storage utilization: 100%; delay: 34ms; operating frequency: 1.8GHz..., the load information of physical service 1 is recorded in the load information table of the load information service module, and the load information of physical service 1 is periodically updated.

[0170] Receive the load information reported by the target parallel processing component according to the preset period, monitor the timeliness and accuracy of the load situation of the target parallel processing component, determine whether overload occurs more promptly and accurately, schedule task information to the target serial processing component more promptly and accurately, call the task processing model to perform stable task processing based on the task information, further reduce the delay of task processing, and further improve the efficiency and availability of task processing.

[0171] In an optional embodiment of the present disclosure, at least one parallel processing component and at least one serial processing component are divided into a plurality of clusters, each cluster including at least one parallel processing component and at least one serial processing component;

[0172] Accordingly, step 104 includes the following specific steps:

[0173] Determine a target cluster from multiple clusters;

[0174] Dispatching task information to the target parallel processing component in the target cluster;

[0175] Accordingly, step 108 includes the following specific steps:

[0176] Schedule the overloaded task information to the target serial processing component in the target cluster.

[0177] Parallel and serial processing components, as physical services—the smallest deployment units—can be divided into multiple clusters based on factors such as region, hardware model, model type, and model parameters, providing external virtual services. When performing backup processing, serial processing components within the same cluster are prioritized to ensure overall task processing stability. For example, implementing backup processing within clusters in the same region can improve packet loss rates, network stability, and hardware compatibility compared to backup processing between different clusters.

[0178] Determining the target cluster from multiple clusters includes but is not limited to: based on a preset routing table, a preset order or a strategy (for example, determined based on the network address of the client, or determined based on the task type of the target task corresponding to the task information, or determined based on the amount of information in the task information), which is not limited here.

[0179] The task information is dispatched to the target parallel processing component in the target cluster, specifically by dispatching the task information to the target parallel processing component in the target cluster through the application programming interface. Specifically, the task information is dispatched to the target parallel processing component in the target cluster through the priority scheduling policy on the application programming interface.

[0180] The overloaded task information is dispatched to the target serial processing component in the target cluster, specifically by dispatching the overloaded task information to the target serial processing component in the target cluster through an application programming interface. Specifically, the overloaded task information is dispatched to the target serial processing component in the target cluster through a priority scheduling policy on the application programming interface.

[0181] For example, based on a preset routing table, a target cluster is determined from the 20 clusters of the task processing platform. Using the priority scheduling policy on the target cluster's application programming interface (API), the following are implemented: priority scheduling target: Physical Service 1 (target graphics processing component); backup scheduling target: Physical Service 2 (target central processing component). Real-time traffic containing task information for 3,000 target tasks is dispatched to Physical Service 1. On Physical Service 1, the 3,000 task information is input into the target graphics processing component's large-scale language model, and the corresponding target tasks are executed based on the task information. Load information reported by Physical Service 1 (target graphics processing component) is received: CPU utilization: 67%; storage utilization: 100%; latency: 34ms; operating frequency: 1.8GHz... The load information for Physical Service 1 is recorded in the load information table of the load information service module of the target cluster. The dynamic load perception module of the target cluster obtains load information, and based on the load information and load thresholds: CPU usage threshold: 65%; storage usage threshold: 90%; latency: 25ms; operating frequency: 1.5GHz..., it is determined that the target graphics processing component of physical service 1 is overloaded. Through the priority scheduling strategy on the application programming interface of the target cluster, the real-time traffic containing the overload task information of 1,000 target tasks is scheduled to physical service 2. On physical service 2, the 1,000 overload task information is input into the large-scale language model of the target central processing component, and the corresponding target tasks are executed based on the overload task information.

[0182] Determine the target cluster from multiple clusters; dispatch tasks to the target parallel processing component in the target cluster; and dispatch overloaded tasks to the target serial processing component in the target cluster. Prioritize backup processing within the same cluster, improving task processing stability.

[0183] In an optional embodiment of the present disclosure, after scheduling the overload task information to the target serial processing component in the target cluster, the following specific steps are further included:

[0184] Obtain load information of the target serial processing component;

[0185] If it is determined based on the load information that the target serial processing component is overloaded, the step of determining a target cluster from among the plurality of clusters is returned to execution.

[0186] After backup processing, if the target serial processing component is still overloaded, the target cluster can be re-determined for task processing through a cross-cluster approach, thereby improving the availability of task processing.

[0187] The load information of the target serial processing component is obtained by: obtaining the load information of the target serial processing component obtained through monitoring. The monitoring method can refer to the above step 106.

[0188] Determining whether a target serial processing component is overloaded based on the load information, specifically by determining that the target parallel processing component is overloaded based on the load information and a load threshold. Furthermore, the load information is obtained by a dynamic load sensing module, and the target parallel processing component is overloaded based on the load information and the load threshold. Determining whether a target parallel processing component is overloaded based on the load information and the load threshold includes: identifying whether the load information reaches a second load threshold preset for the target serial processing component; and if so, determining that the target parallel processing component is overloaded.

[0189] The second load threshold is a pre-set load threshold used to identify whether the target central processing component is overloaded, including but not limited to: central processing component parameter threshold (CPU parameter threshold), storage parameter threshold, latency threshold, operating frequency threshold, query rate per second threshold, concurrency threshold, batch number threshold, real-time data size threshold, and queue data size threshold.

[0190] If the identified load information reaches the second load threshold, it is determined that the target central processing component is overloaded and needs to be re-determined. If the identified load information does not reach the second load threshold, it is determined that the target central processing component load is normal and does not need to be re-determined. That is, task processing continues on the current target cluster.

[0191] For example, the load information reported by physical service 2 (target central processing component) is received: CPU utilization: 93%; storage utilization: 100%; delay: 38ms; operating frequency: 1.8GHz..., and the high-performance computing control module determines that the target central processing component is overloaded based on the load information, and returns to execute the step of determining the target cluster from 20 clusters.

[0192] Obtain load information for the target serial processing component. If the target serial processing component is determined to be overloaded based on the load information, the process returns to the step of determining a target cluster from multiple clusters. If the current target cluster is short of resources, a new target cluster is determined, improving the availability of task processing.

[0193] In an optional embodiment of the present disclosure, the method further includes the following specific steps:

[0194] Receive task processing results fed back by the parallel processing component and the serial processing component;

[0195] Feedback task processing results to the client;

[0196] Receive task result feedback sent by the client based on task processing results;

[0197] Adjust the parameters of the task processing model based on task result feedback.

[0198] After the task processing models in the parallel and serial processing components complete task processing and obtain the task processing results, they must be fed back to the client through the scheduling component. After receiving the task processing results, the client can provide feedback to adjust the parameters of the task processing model and improve its performance.

[0199] The task processing result is the output of the target task. Depending on the target task type, the corresponding result modality varies. For example, if the target task is a speech stream translation task, the task processing result is the translated text data or speech stream data; if the target task is a style transfer task, the task processing result is the generated target image after style transfer.

[0200] Task result feedback is model performance feedback based on the task processing results input by the logged-in client user. It is used to adjust the parameters of the task processing model. For example, the task information sent by the client is "Please indicate the 20th letter of the 26 English alphabet?" After processing from steps 102 to 108, the task processing result fed back to the client is "Of the 26 English alphabets, the 20th letter is Z." The task result feedback is "There is a factual error" entered by the user on the client. This task result feedback is sent to the scheduling component of the task processing platform. The scheduling component then dispatches this task result feedback to the target parallel processing component and / or target serial processing component to adjust the parameters of the deployed task processing model.

[0201] Feedback the task processing results to the client, specifically by feeding back the task processing results to the client through the virtual service access point.

[0202] Receive task result feedback sent by the client based on the task processing result, specifically by: receiving task result feedback sent by the client based on the task processing result through the virtual service access point.

[0203] Adjust the parameters of the task processing model based on the task result feedback. The specific method is: through the application programming interface, schedule the task result feedback to the target parallel processing component and / or target serial processing component, and adjust the parameters of the task processing model based on the task result feedback.

[0204] Exemplarily, 3,000 task processing results fed back by the graphics processing component and the central processing component are received, and the 3,000 task processing results are fed back to the corresponding 3,000 clients through a unified virtual service access point. Task result feedback sent by 12 clients based on the task processing results is received: factual errors, inability to understand the context, knowledge hallucinations, format errors, etc. The 12 task result feedbacks are dispatched to the target graphics processing component and the target central processing component through the application programming interface, and the parameters of the large-scale language model are adjusted based on the 12 task result feedbacks.

[0205] Receive task processing results from the parallel and serial processing components; transmit these results to the client; receive task result feedback from the client based on the task processing results; and adjust the parameters of the task processing model based on the task result feedback. By interacting with the client and obtaining task result feedback based on the task processing results, the parameters of the task processing model deployed on the target parallel and serial processing components are adjusted, improving the accuracy of subsequent task processing.

[0206] Referring to FIG. 2 , FIG. 2 shows a flowchart of a task processing method provided by an embodiment of the present disclosure. The method is applied to a scheduling component in a question-and-answer platform. The question-and-answer platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component. Both the parallel processing component and the serial processing component are deployed with a question-and-answer model. The method includes the following specific steps:

[0207] Step 202: Receive the question text sent by the client.

[0208] Step 204: dispatch the question-answer text to the target parallel processing component to trigger the target parallel processing component to call the question-answer model to process the question-answer task based on the question-answer text and obtain the corresponding reply text.

[0209] Step 206: Obtain load information of the target parallel processing component.

[0210] Step 208: When it is determined that the target parallel processing component is overloaded based on the load information, the overloaded question and answer text is dispatched to the target serial processing component to trigger the target serial processing component to call the question and answer model to process the question and answer task based on the overloaded question and answer text to obtain the corresponding reply text.

[0211] The Q&A platform is a remote service platform (server) with Q&A functionality. The Q&A platform can provide users with Q&A services through virtual service clients in the form of applications, web pages, applets, etc. A network connection is established between the Q&A platform and the client. The virtual service of the Q&A platform provides a unified service access point to the outside world, such as an API, Socket interface, or other forms of network connection. The Q&A platform includes physical services of at least one parallel processing component and at least one serial processing component. Generally, a physical service uses the same model of parallel processing components or serial processing components. At least one parallel processing component is generally of different models, and the same is true for serial processing components, and they are distributed in different areas, presenting heterogeneous models. The Q&A platform also includes a scheduling component, which schedules the question text sent by the received client to the parallel processing component or the serial processing component.

[0212] The question text is the natural language input text of the question-answering task to be answered. The answer text is the natural language output text of the question-answering task.

[0213] The question-answering model is a pre-trained deep learning model with question-answering capabilities. The question-answering model is deployed in parallel processing components and serial processing components. The model types and model parameters of the question-answering models deployed on different parallel processing components and serial processing components can be the same or different. At the data level, the question-answering model performs feature encoding on the scheduled question text to obtain a feature vector, and then performs a series of feature transformations to obtain the corresponding output vector, which is then decoded to obtain the corresponding reply text. Question-answering models are generally divided into three types according to their structure: encoding, decoding, and encoding-decoding. The large-scale language model (LLM) is used as an example for illustration in the embodiments of this disclosure.

[0214] The embodiment of the present disclosure and the embodiment of FIG. 1 are based on the same inventive concept. The specific methods of steps 202 to 208 can refer to steps 102 to 108 and will not be repeated here.

[0215] Exemplarily, the question-answering platform is a web service platform for a large-scale language model. The web service platform includes a scheduling component, 30 graphics processing components, and 30 central processing components. The scheduling component includes a unified virtual service access point, an application programming interface (API) server, a load information service module, and a high-performance computing control module. The API server includes a dynamic load sensing module. The unified virtual service access point receives 900 question-answering requests from clients, each carrying the question texts of 900 Q&A tasks, constituting real-time traffic. The real-time traffic containing the question texts of the 900 Q&A tasks is connected to the API. Using a priority scheduling policy within the API, with priority scheduling targeting physical service 1 (target graphics processing component) and backup scheduling targeting physical service 2 (target central processing component), the real-time traffic containing the question texts of the 900 Q&A tasks is dispatched to physical service 1. On physical service 1, the 900 question texts are input into the large-scale language model of the target graphics processing component. The corresponding Q&A tasks are executed based on the question texts, resulting in corresponding response texts. The system receives load information reported by Physical Service 1 (the target graphics processing component): CPU usage: 77%; storage usage: 100%; latency: 37ms; operating frequency: 1.6GHz. This load information is then recorded in the load information table of the load information service module. The dynamic load sensing module obtains this load information and, based on the load information and load thresholds (CPU usage threshold: 70%; storage usage threshold: 92%; latency: 35ms; operating frequency: 1.3GHz), determines that the target graphics processing component of Physical Service 1 is overloaded. Using a priority scheduling policy in the application programming interface, the real-time traffic containing 180 question-and-answer tasks is dispatched to Physical Service 2. On Physical Service 2, the 180 question texts are input into the large-scale language model of the target central processing component. The corresponding question-and-answer tasks are executed based on the question texts, resulting in the corresponding responses.

[0216] In an embodiment of the present disclosure, a question-and-answer method is provided, which is applied to a scheduling component in a question-and-answer platform, wherein the question-and-answer platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component, wherein both the parallel processing component and the serial processing component are deployed with a question-and-answer model; the method includes: receiving a question text sent by a client; scheduling the question-and-answer text to a target parallel processing component to trigger the target parallel processing component to call the question-and-answer model to process a question-and-answer task based on the question-and-answer text, and obtain a corresponding reply text; obtaining load information of the target parallel processing component; and when it is determined based on the load information that the target parallel processing component is overloaded, scheduling the question-and-answer text to a target serial processing component to trigger the target serial processing component to call the question-and-answer model to process a question-and-answer task based on the question-and-answer text, and obtain a corresponding reply text. After receiving the question text sent by the client, the question text is scheduled to the target parallel processing component first, and the question-answering model is called to perform efficient question-answering based on the question text. Based on the load information of the target parallel processing component, the load situation of the target parallel processing component is monitored. In the event of overload, the overloaded question text is promptly scheduled to the target serial processing component, and the question-answering model is called to perform stable question-answering based on the overloaded question text, avoiding congestion problems, reducing the delay of question-answering, improving the efficiency and availability of question-answering, and thereby improving the user experience of the client.

[0217] FIG3 shows an architecture diagram of a task processing platform in a task processing method provided by an embodiment of the present disclosure, as shown in FIG3 :

[0218] The task processing platform includes multiple clusters, each of which includes a scheduling component, a target parallel processing component, and a target serial processing component. The scheduling component includes an application programming interface (API), a load information service module, and a high-performance computing control module. The API includes a dynamic load sensing module. Task information sent by clients is received through a unified virtual service access point, constituting 100% of the traffic and determining the target cluster. Task information is connected to the API of the target cluster based on the virtual service: traffic allocation policy: priority scheduling policy; priority scheduling target: physical service 1 (parallel processing component); backup scheduling target: physical service 2 (serial processing component). Load information reported by physical services 1 and 2 is received through the load information service module. Through the API, traffic (80%) is prioritized and dispatched to physical service 1, where a target parallel processing component is deployed. The target parallel processing component is triggered to process the task information for the 80% traffic. The high-performance computing control module obtains load information from the load information service module and adjusts resources for physical service 1. The dynamic load sensing module obtains load information from the load information service module. If the target parallel processing component is determined to be overloaded, the application programming interface (API) is used to allocate 20% of the reserved traffic to physical service 2, where the 128GB target serial processing component is deployed. The target serial processing component is triggered to process the overloaded task information of the 20% traffic. The high-performance computing control module obtains load information from the load information service module and adjusts the resources of physical service 2.

[0219] The following, in conjunction with Figure 4, further illustrates the task processing method provided by the present disclosure, using its application in an inference scenario as an example. Figure 4 shows a flowchart of the processing process of a task processing method applied to an inference scenario, provided by an embodiment of the present disclosure. The method is applied to a scheduling component in a task processing platform, which includes a scheduling component, at least one parallel processing component, and at least one serial processing component, each of which is equipped with an inference model. The method includes the following specific steps:

[0220] Step 402: Receive an inference request sent by a client, wherein the inference request carries task information of the inference task.

[0221] Step 404: dispatch the task information to the target parallel processing component to trigger the target parallel processing component to call the inference model to process the inference task based on the task information.

[0222] Step 406: Receive the first initial load information reported by the target parallel processing component according to the preset period, adjust the number of parallel processing component instances on the target parallel processing component based on the first initial load information and the task information, and schedule the task information to the target parallel processing component after resource adjustment.

[0223] Step 408: Receive load information reported by the target parallel processing component according to a preset period.

[0224] Step 410: Identify whether the load information reaches a first load threshold preset for the target parallel processing component.

[0225] Step 412: If yes, determine that the target parallel processing component is overloaded, and dispatch the overload task information to the target serial processing component to trigger the target serial processing component to call the inference model to process the inference task based on the overload task information.

[0226] Step 414: Receive the second initial load information reported by the target parallel processing component according to the preset period, adjust the number of serial processing component instances on the target serial processing component based on the second initial load information and the overload task information, and schedule the overload task information to the target serial processing component after resource adjustment.

[0227] Step 416: Receive the inference results fed back by the parallel processing component and the serial processing component.

[0228] Step 418: Feedback the inference result to the client.

[0229] In the embodiment of the present disclosure, backup processing of inference is implemented based on service overload awareness. On the one hand, while enabling elastic scaling, parallel processing components can be used preferentially for inference to improve the performance and efficiency of inference, thereby overcoming the problem of low performance and efficiency of inference when using only serial processing components. Moreover, when the number of parallel processing components is limited, resulting in the inability to adjust resources, the overloaded task information can be dispatched to the target serial processing component by sensing service overload in advance, thereby ensuring both the elastic scaling of real-time inference tasks using heterogeneous environments and the high efficiency and high availability of inference tasks.

[0230] Corresponding to the above-mentioned method embodiments, the present disclosure also provides a task processing device embodiment. FIG5 shows a schematic diagram of the structure of a task processing device provided by an embodiment of the present disclosure. As shown in FIG5 , the device is applied to a scheduling component in a task processing platform. The task processing platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component. Both the parallel processing component and the serial processing component are deployed with a task processing model. The device includes:

[0231] The first virtual service access point 502 is configured to receive task information sent by the client;

[0232] The first application programming interface 504 is configured to dispatch the task information to the target parallel processing component to trigger the target parallel processing component to call the task processing model to process the target task based on the task information;

[0233] A first load information service module 506 is configured to obtain load information of a target parallel processing component;

[0234] The first application programming interface 504 is configured to dispatch overloaded task information to the target serial processing component when it is determined based on the load information that the target parallel processing component is overloaded, so as to trigger the target serial processing component to call the task processing model to process the target task based on the overloaded task information.

[0235] Optionally, the first load information service module 506 is further configured to:

[0236] Acquire first initial load information of the target parallel processing component, wherein the first initial load information is load information after the task information is scheduled to the target parallel processing component;

[0237] Accordingly, the device further comprises:

[0238] a high performance computing control module configured to adjust resources of a target parallel processing component based on the first initial load information and the task information;

[0239] Accordingly, the first application programming interface 504 is further configured to:

[0240] Schedule the task information to the target parallel processing component after resource adjustment.

[0241] Optionally, the high performance computing control module is further configured to:

[0242] Based on the first initial load information and the task information, the number of parallel processing component instances on the target parallel processing component is adjusted.

[0243] Optionally, the first application programming interface 504 further includes:

[0244] a dynamic load sensing module, configured to identify whether the load information reaches a first load threshold preset for the target parallel processing component;

[0245] Accordingly, the first application programming interface 504 is further configured to:

[0246] If so, the overload task information is dispatched to the target serial processing component.

[0247] Optionally, the first load information service module 506 is further configured to:

[0248] Acquire second initial load information of the target serial processing component, wherein the second initial load information is load information after the overload task information is dispatched to the target serial processing component;

[0249] Accordingly, the high performance computing control module is configured to adjust resources of the target serial processing component based on the second initial load information and the overload task information;

[0250] Accordingly, the first application programming interface 504 is further configured to:

[0251] Schedule the overloaded task information to the target serial processing component after resource adjustment.

[0252] Optionally, the high performance computing control module is further configured to:

[0253] Based on the second initial load information and the overload task information, the number of serial processing component instances on the target serial processing component is adjusted.

[0254] Optionally, the first load information service module 506 is further configured to:

[0255] Receive load information reported by the target parallel processing component according to a preset period.

[0256] Optionally, the at least one parallel processing component and the at least one serial processing component are divided into a plurality of clusters, each cluster including at least one parallel processing component and at least one serial processing component;

[0257] Accordingly, the first application programming interface 504 is further configured to:

[0258] A target cluster is determined from multiple clusters; task information is dispatched to a target parallel processing component in the target cluster; and overloaded task information is dispatched to a target serial processing component in the target cluster.

[0259] Optionally, the first load information service module 506 is further configured to:

[0260] Obtain load information of the target serial processing component;

[0261] Accordingly, the first application programming interface 504 is further configured to:

[0262] If it is determined based on the load information that the target serial processing component is overloaded, the step of determining a target cluster from among the plurality of clusters is returned to execution.

[0263] Optionally, the first application programming interface 504 is further configured to:

[0264] Receive task processing results fed back by the parallel processing component and the serial processing component;

[0265] Accordingly, the first virtual service access point 502 is further configured as follows:

[0266] Feedback the task processing results to the client; receive task result feedback sent by the client based on the task processing results;

[0267] Accordingly, the device further comprises:

[0268] The parameter adjustment module is configured to adjust the parameters of the task processing model based on task result feedback.

[0269] In an embodiment of the present disclosure, a task processing device is provided, which is applied to a scheduling component in a task processing platform. The task processing platform includes a scheduling component, at least one parallel processing component and at least one serial processing component, and both the parallel processing component and the serial processing component are deployed with a task processing model; the device includes: a first virtual service access point, configured to receive task information sent by a client; a first application programming interface, configured to schedule task information to a target parallel processing component, so as to trigger the target parallel processing component to call the task processing model to process the target task based on the task information; a first load information service module, configured to obtain load information of the target parallel processing component; and the first application programming interface, configured to schedule overloaded task information to a target serial processing component when it is determined that the target parallel processing component is overloaded based on the load information, so as to trigger the target serial processing component to call the task processing model to process the target task based on the overloaded task information. After receiving the task information sent by the client, the task information is scheduled to the target parallel processing component first, and the task processing model is called to perform efficient task processing based on the task information. Based on the load information of the target parallel processing component, the load situation of the target parallel processing component is monitored. In the event of overload, the overloaded task information is promptly scheduled to the target serial processing component, and the task processing model is called to perform stable task processing based on the overloaded task information, avoiding congestion problems, reducing the delay of task processing, improving the efficiency and availability of task processing, and thereby improving the user experience of the client.

[0270] The above is a schematic scheme of a task processing device of this embodiment. It should be noted that the technical scheme of the task processing device and the technical scheme of the task processing method described above are of the same concept. For details not described in detail in the technical scheme of the task processing device, please refer to the description of the technical scheme of the task processing method described above.

[0271] Corresponding to the above-mentioned method embodiments, the present disclosure also provides a question-and-answer device embodiment. FIG6 shows a schematic diagram of the structure of a question-and-answer device provided by an embodiment of the present disclosure. As shown in FIG6, a scheduling component is applied to a question-and-answer platform. The question-and-answer platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component. Both the parallel processing component and the serial processing component are deployed with a question-and-answer model. The device includes:

[0272] The second virtual service access point 602 is configured to receive a question text sent by the client;

[0273] The second application programming interface 604 is configured to dispatch the question-answer text to the target parallel processing component, so as to trigger the target parallel processing component to call the question-answer model to process the question-answer task based on the question-answer text and obtain the corresponding reply text;

[0274] A second load information service module 606 is configured to obtain load information of a target parallel processing component;

[0275] The second application programming interface 604 is configured to dispatch the overloaded question and answer text to the target serial processing component when it is determined based on the load information that the target parallel processing component is overloaded, so as to trigger the target serial processing component to call the question and answer model to process the question and answer task based on the overloaded question and answer text and obtain the corresponding reply text.

[0276] In an embodiment of the present disclosure, a question-and-answer device is provided, which is applied to a scheduling component in a question-and-answer platform. The question-and-answer platform includes a scheduling component, at least one parallel processing component, and at least one serial processing component, and both the parallel processing component and the serial processing component are deployed with a question-and-answer model; the device includes: a second virtual service access point, configured to receive a question text sent by a client; a second application programming interface, configured to schedule the question-and-answer text to a target parallel processing component, so as to trigger the target parallel processing component to call the question-and-answer model to process a question-and-answer task based on the question-and-answer text, and obtain a corresponding reply text; a second load information service module, configured to obtain load information of the target parallel processing component; and the second application programming interface, configured to schedule the overloaded question-and-answer text to a target serial processing component when it is determined based on the load information that the target parallel processing component is overloaded, so as to trigger the target serial processing component to call the question-and-answer model to process the question-and-answer task based on the overloaded question-and-answer text, and obtain a corresponding reply text. After receiving the question text sent by the client, the question text is scheduled to the target parallel processing component first, and the question-answering model is called to perform efficient question-answering based on the question text. Based on the load information of the target parallel processing component, the load situation of the target parallel processing component is monitored. In the event of overload, the overloaded question text is promptly scheduled to the target serial processing component, and the question-answering model is called to perform stable question-answering based on the overloaded question text, avoiding congestion problems, reducing the delay of question-answering, improving the efficiency and availability of question-answering, and thereby improving the user experience of the client.

[0277] The above is a schematic diagram of a question-and-answer device according to this embodiment. It should be noted that the technical solution of the question-and-answer device and the technical solution of the question-and-answer method described above are based on the same concept. For details not described in detail in the technical solution of the question-and-answer device, please refer to the description of the technical solution of the question-and-answer method described above.

[0278] Corresponding to the above-mentioned method embodiments, the present disclosure also provides a task processing platform embodiment. FIG7 shows a schematic diagram of the structure of a task processing platform provided by an embodiment of the present disclosure. As shown in FIG7, the apparatus includes a scheduling component 702, at least one parallel processing component, and at least one serial processing component. Both the parallel processing component and the serial processing component are deployed with a task processing model.

[0279] The scheduling component 702 is configured to receive task information sent by the client, schedule the task information to the target parallel processing component 704, obtain load information of the target parallel processing component 704, and schedule the overloaded task information to the target serial processing component 706 if the target parallel processing component 704 is determined to be overloaded based on the load information.

[0280] The target parallel processing component 704 is used to receive the task information scheduled by the scheduling component 702, call the task processing model, and process the target task based on the task information;

[0281] The target serial processing component 706 is used to receive the overload task information scheduled by the scheduling component 702, call the task processing model, and process the target task based on the overload task information.

[0282] In an embodiment of the present disclosure, a task processing platform is provided, including a scheduling component, at least one parallel processing component and at least one serial processing component, wherein both the parallel processing component and the serial processing component are deployed with a task processing model; the scheduling component is used to receive task information sent by a client, schedule the task information to a target parallel processing component, obtain load information of the target parallel processing component, and when it is determined based on the load information that the target parallel processing component is overloaded, schedule the overloaded task information to the target serial processing component; the target parallel processing component is used to receive task information scheduled by the scheduling component, call the task processing model, and process the target task based on the task information; the target serial processing component is used to receive overloaded task information scheduled by the scheduling component, call the task processing model, and process the target task based on the overloaded task information. After receiving the task information sent by the client, the task information is scheduled to the target parallel processing component first, and the task processing model is called to perform efficient task processing based on the task information. Based on the load information of the target parallel processing component, the load situation of the target parallel processing component is monitored. In the event of overload, the overloaded task information is promptly scheduled to the target serial processing component, and the task processing model is called to perform stable task processing based on the overloaded task information, avoiding congestion problems, reducing the delay of task processing, improving the efficiency and availability of task processing, and thereby improving the user experience of the client.

[0283] The above is a schematic scheme of a task processing platform of this embodiment. It should be noted that the technical scheme of the task processing platform and the technical scheme of the task processing method described above are of the same concept. For details not described in detail in the technical scheme of the task processing platform, please refer to the description of the technical scheme of the task processing method described above.

[0284] Figure 8 shows a block diagram of a computing device provided by an embodiment of the present disclosure. Components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0285] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of such networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface controller (NIC)) whether wired or wireless, such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC).

[0286] In one embodiment of the present disclosure, the aforementioned components of the computing device 800 and other components not shown in FIG8 may also be connected to each other, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG8 is for illustrative purposes only and does not limit the scope of the present disclosure. Those skilled in the art may add or replace other components as needed.

[0287] Computing device 800 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 800 can also be a mobile or stationary server.

[0288] Among them, the processor 820 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned task processing method or question-answering method.

[0289] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device is based on the same concept as the technical solutions of the task processing method and question-and-answer method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the task processing method or question-and-answer method described above.

[0290] The embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned task processing method or question-answering method.

[0291] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solutions of the task processing method and question-and-answer method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the task processing method or question-and-answer method described above.

[0292] The present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the task processing methods or question-answering methods in the aforementioned method embodiments.

[0293] An embodiment of the present disclosure also provides a computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any one of the task processing methods or question-and-answer methods in the aforementioned method embodiments.

[0294] The embodiments of the present disclosure further provide a computer program, which, when executed by a processor, implements the steps of any one of the task processing methods or question-answering methods in the aforementioned method embodiments.

[0295] The above is a schematic diagram of a computer program according to this embodiment. It should be noted that the technical solution of this computer program is based on the same concept as the technical solutions of the task processing method and question-answering method described above. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solutions of the task processing method or question-answering method described above.

[0296] The foregoing description describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0297] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0298] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present disclosure are not limited by the order of the actions described, because according to the embodiments of the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present disclosure.

[0299] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0300] The preferred embodiments of the present disclosure disclosed above are only used to help illustrate the present disclosure. The optional embodiments do not describe all details in detail, nor do they limit the invention to only the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of the present disclosure. The present disclosure selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present disclosure, so that those skilled in the art can better understand and utilize the present disclosure. The present disclosure is limited only by the claims and their full scope and equivalents. Industrial Applicability

[0301] The solution provided by the embodiment of the present disclosure can be applied to the task processing process in the task processing platform, receiving the task information sent by the client; scheduling the task information to the target parallel processing component to trigger the target parallel processing component to call the task processing model to process the target task based on the task information; obtaining the load information of the target parallel processing component; in the case where the target parallel processing component is determined to be overloaded based on the load information, scheduling the overloaded task information to the target serial processing component to trigger the target serial processing component to call the task processing model to process the target task based on the task information. That is, after receiving the task information sent by the client, the task information is preferentially scheduled to the target parallel processing component, the task processing model is called to perform efficient task processing based on the task information, the load situation of the target parallel processing component is monitored based on the load information of the target parallel processing component, and in the case of overload, the overloaded task information is promptly scheduled to the target serial processing component, and the task processing model is called to perform stable task processing based on the overloaded task information, thereby avoiding congestion problems, reducing the delay of task processing, improving the efficiency and availability of task processing, and thus improving the user experience of the client.

Claims

1. A task processing method, applied to a scheduling component in a task processing platform, wherein the task processing platform comprises the scheduling component, at least one parallel processing component and at least one serial processing component, wherein both the parallel processing component and the serial processing component are deployed with a task processing model; the method comprises: Receive task information sent by the client; Dispatching the task information to a target parallel processing component to trigger the target parallel processing component to call the task processing model to process the target task based on the task information; Obtaining load information of the target parallel processing component; When it is determined that the target parallel processing component is overloaded based on the load information, the overload task information is scheduled to the target serial processing component to trigger the target serial processing component to call the task processing model to process the target task based on the overload task information, wherein the overload task information is used to represent task information that exceeds the load threshold of the target parallel processing component.

2. The method according to claim 1, before obtaining the load information of the target parallel processing component, further comprising: Acquire first initial load information of the target parallel processing component, wherein the first initial load information is load information after the task information is scheduled to the target parallel processing component; Based on the first initial load information and the task information, adjusting resources of the target parallel processing component; The task information is dispatched to the target parallel processing component after resource adjustment.

3. The method according to claim 2, wherein adjusting resources of the target parallel processing component based on the first initial load information and the task information comprises: Based on the first initial load information and the task information, the number of parallel processing component instances on the target parallel processing component is adjusted.

4. The method according to claim 1, wherein, in the case where it is determined based on the load information that the target parallel processing component is overloaded, scheduling the overloaded task information to the target serial processing component comprises: Identifying whether the load information reaches a first load threshold preset for the target parallel processing component; If so, the overload task information is dispatched to the target serial processing component.

5. The method according to claim 1, after scheduling the overload task information to the target serial processing component, further comprises: Acquire second initial load information of the target serial processing component, wherein the second initial load information is load information after the overload task information is scheduled to the target serial processing component; Based on the second initial load information and the overload task information, adjusting resources of the target serial processing component; The overload task information is dispatched to the target serial processing component after resource adjustment.

6. The method according to claim 5, wherein adjusting resources of the target serial processing component based on the second initial load information and the overload task information comprises: Based on the second initial load information and the overload task information, the number of serial processing component instances on the target serial processing component is adjusted.

7. The method according to claim 1, wherein obtaining the load information of the target parallel processing component comprises: The load information reported by the target parallel processing component according to a preset period is received.

8. The method according to any one of claims 1 to 7, wherein at least one of the parallel processing components and at least one of the serial processing components are divided into a plurality of clusters, each cluster comprising at least one of the parallel processing components and at least one of the serial processing components; The step of dispatching the task information to the target parallel processing component includes: determining a target cluster from among the plurality of clusters; Scheduling the task information to the target parallel processing component in the target cluster; The step of dispatching the overload task information to the target serial processing component includes: The overload task information is dispatched to the target serial processing component in the target cluster.

9. The method according to claim 8, after scheduling the overload task information to the target serial processing component in the target cluster, further comprises: Obtaining load information of the target serial processing component; In response to determining that the target serial processing component is overloaded based on the load information, returning to the step of determining the target cluster from the plurality of clusters.

10. The method according to claim 1, further comprising: Receive task processing results fed back by the parallel processing component and the serial processing component; Feedback the task processing result to the client; Receiving task result feedback sent by the client based on the task processing result; Parameters of the task processing model are adjusted based on the task result feedback.

11. A question-answering method, applied to a scheduling component in a question-answering platform, the question-answering platform comprising the scheduling component, at least one parallel processing component and at least one serial processing component, wherein both the parallel processing component and the serial processing component are deployed with a question-answering model; the method comprising: Receive the question text sent by the client; Dispatching the question text to a target parallel processing component to trigger the target parallel processing component to call a question-answering model to process a question-answering task based on the question text to obtain a corresponding answer text; Obtaining load information of the target parallel processing component; When it is determined that the target parallel processing component is overloaded based on the load information, the overloaded question and answer text is scheduled to the target serial processing component to trigger the target serial processing component to call the question and answer model to process the question and answer task based on the overloaded question and answer text to obtain the corresponding reply text, wherein the overloaded question and answer text is used to represent the question text that exceeds the load threshold of the target parallel processing component.

12. A task processing method, applied to a scheduling component in a task processing platform, wherein the task processing platform comprises the scheduling component, at least one parallel processing component and at least one serial processing component, wherein both the parallel processing component and the serial processing component are deployed with an inference model; the method comprises: Receiving an inference request sent by a client, wherein the inference request carries task information of an inference task; Dispatching the task information to a target parallel processing component to trigger the target parallel processing component to call the reasoning model to process the reasoning task based on the task information; receiving first initial load information reported by the target parallel processing component according to a preset period, adjusting the number of parallel processing component instances on the target parallel processing component based on the first initial load information and the task information, and scheduling the task information to the target parallel processing component after resource adjustment; Receiving load information reported by the target parallel processing component according to a preset period; In the case where it is determined that the target parallel processing component is overloaded based on the load information, overload task information is dispatched to the target serial parallel processing component to trigger the target serial processing component to call the inference model to process the inference task based on the overload task information, wherein the overload task information is used to represent task information that exceeds a load threshold of the target parallel processing component; receiving the second initial load information reported by the target parallel processing component according to the preset period, adjusting the number of the serial processing component instances on the target serial processing component based on the second initial load information and the overload task information, and scheduling the overload task information to the target serial processing component after resource adjustment; receiving the inference results fed back by the parallel processing component and the serial processing component; The inference result is fed back to the client.

13. A task processing platform, comprising a scheduling component, at least one parallel processing component and at least one serial processing component, wherein both the parallel processing component and the serial processing component are deployed with a task processing model; The scheduling component is used to receive task information sent by the client, schedule the task information to the target parallel processing component, obtain the load information of the target parallel processing component, and when it is determined based on the load information that the target parallel processing component is overloaded, schedule the overloaded task information to the target serial processing component; The target parallel processing component is used to receive the task information scheduled by the scheduling component, call the task processing model, and process the target task based on the task information; The target serial processing component is used to receive the overload task information scheduled by the scheduling component, call the task processing model, and process the target task based on the overload task information, wherein the overload task information is used to represent task information that exceeds the load threshold of the target parallel processing component.

14. A task processing device, applied to a scheduling component in a task processing platform, the task processing platform comprising a scheduling component, at least one parallel processing component and at least one serial processing component, the parallel processing component and the serial processing component are both deployed with a task processing model; the device comprises: The first virtual service access point is configured to receive task information sent by the client; A first application programming interface is configured to dispatch the task information to a target parallel processing component to trigger the target parallel processing component to call the task processing model to process a target task based on the task information; A first load information service module is configured to obtain load information of the target parallel processing component; The first application programming interface is configured to schedule overload task information to the target serial processing component when it is determined that the target parallel processing component is overloaded based on the load information, so as to trigger the target serial processing component to call the task processing model to process the target task based on the task information, wherein the overload task information is used to represent task information that exceeds the load threshold of the target parallel processing component.

15. A question-answering device, applied to a scheduling component in a question-answering platform, the question-answering platform comprising a scheduling component, at least one parallel processing component and at least one serial processing component, wherein both the parallel processing component and the serial processing component are deployed with a question-answering model; the device comprises: The second virtual service access point is configured to receive a question text sent by the client; A second application programming interface is configured to dispatch the question text to a target parallel processing component to trigger the target parallel processing component to call the question-answering model to process a question-answering task based on the question text to obtain a corresponding answer text; A second load information service module is configured to obtain load information of the target parallel processing component; The second application programming interface is configured to schedule the overloaded question and answer text to the target serial processing component when it is determined that the target parallel processing component is overloaded based on the load information, so as to trigger the target serial processing component to call the question and answer model to process the question and answer task based on the question and answer text to obtain the corresponding reply text, wherein the overloaded question and answer text is used to represent the question text that exceeds the load threshold of the target parallel processing component.

16. A computer program product, wherein: The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

17. A computer program product, wherein: The invention comprises a non-volatile computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 12 when executed by a processor.

18. A computer program, wherein When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

19. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 12 are implemented.

20. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images