Data processing system and method, and related device

Abstract

The present application provides a data processing system, comprising a model deployment unit and at least one model operation unit. The model deployment unit is used for: deploying, in a first model operation unit, a first pre-trained model requested to be deployed by a first user, and acquiring a model deployment request provided by a second user, wherein the model deployment request is used for requesting to deploy a second pre-trained model; and when the first pre-trained model is the same as the second pre-trained model, configuring the first model operation unit to process first service inference traffic on the basis of the first pre-trained model, wherein the first service inference traffic comprises inference traffic of the first user and inference traffic of the second user for a same pre-trained model. Therefore, compared with independently deploying the same pre-trained model for each user, computing resources and memory resources required for deploying the pre-trained model can be effectively reduced, thereby reducing hardware costs required for providing cloud services on the basis of AI models. In addition, the present application further provides a corresponding data processing method and a related device.

Description

Data processing system, method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on March 27, 2024, with application number 202410361750.5 and application name “Data processing systems, methods and related equipment”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 14, 2024, with application number 202410598422.7 and application name “Data processing systems, methods and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a data processing system, method, and related equipment. Background Art

[0003] With the development of artificial intelligence (AI) technology, AI models are widely used in various fields, and the number of parameters in AI models is gradually increasing. For example, the number of parameters in large models such as large language models (LLMs) can reach hundreds of billions.

[0004] In actual applications, users (such as operators) can deploy AI models on one or more sites to provide corresponding cloud services based on the AI ​​models, such as AWS Bedrock. In this case, the AI ​​models deployed by users can be pre-trained models and fine-tuned models. The fine-tuned model can be a model obtained by fine-tuning the pre-trained model, such as training the pre-trained model for a specific type of task.

[0005] However, the AI ​​models deployed by users usually include a large number of parameters, which means that when providing inference cloud services based on AI models, more resources are needed to support the operation of the AI ​​models, including memory resources and computing resources, resulting in higher hardware costs. Summary of the Invention

[0006] In view of this, embodiments of the present application provide a data processing system for reducing the cost of reasoning based on AI models. This application also provides a corresponding data processing method, a computing device cluster, a computer-readable storage medium, and a computer program product.

[0007] In the first aspect, an embodiment of the present application provides a data processing system, including a model deployment unit and at least one model running unit, each model running unit can be implemented, for example, by a computing device or a computing device cluster, and at least one model running unit includes a first model running unit; wherein the model deployment unit is used to: deploy the first pre-trained model requested by the first user in the first model running unit, such as the first user can provide corresponding cloud services by requesting the deployment of the pre-trained model, and obtain a model deployment request provided by the second user, and the model deployment request is used to request the deployment of the second pre-trained model in the data processing system; when the first pre-trained model is the same as the second pre-trained model, that is, two users request to deploy the same pre-trained model, configure the first model running unit to process the first business reasoning traffic based on the first pre-trained model, the first business reasoning traffic includes the reasoning traffic of the first user for the first pre-trained model and the reasoning traffic of the second user for the second pre-trained model, that is, the data processing system uses the same pre-trained model to process the reasoning traffic of different users.

[0008] In this way, when multiple users request to deploy the same pre-trained model, the data processing system processes the inference traffic of multiple users by deploying one pre-trained model. Compared with deploying the same pre-trained model for each user separately, this can effectively reduce the computing resources and memory resources required to deploy the pre-trained model, thereby reducing the hardware cost required to provide cloud services based on AI models.

[0009] In one possible implementation, the first fine-tuning model requested to be deployed by the first user is deployed in the first model running unit. The first fine-tuning model is trained based on the first pre-trained model. The model deployment request is also used to request the deployment of a second fine-tuning model in the data processing system. The model deployment unit is also used to: when the second fine-tuning model is the same as the first fine-tuning model, configure the first model running unit to process the second business inference traffic based on the first fine-tuning model. The second business inference traffic includes the first user's inference traffic for the first fine-tuning model and the second user's inference traffic for the second fine-tuning model. That is, when the fine-tuning models requested to be deployed by different users are also the same, the data processing system can use the same deployed fine-tuning model to process the inference traffic of different users for the fine-tuning model. In this way, compared to deploying the same fine-tuning model for each user separately, the computing resources and memory resources required to deploy the fine-tuning model can be effectively reduced, thereby reducing the hardware cost required to provide cloud services based on the fine-tuning model.

[0010] In one possible embodiment, the model deployment unit is also used to: when the load generated by the first pre-trained model processing the first business inference traffic is greater than the first load threshold corresponding to the first model running unit, deploy a second pre-trained model in the second model running unit in at least one model running unit, and configure the first model running unit to process the first user's inference traffic for the first pre-trained model based on the first pre-trained model, and configure the second model running unit to process the second user's inference traffic for the second pre-trained model based on the second pre-trained model. In this way, when the data processing system uses the same pre-trained model to process the inference traffic of different users and the load is too large, such as the load generated by the pre-trained model processing the inference traffic reaches the maximum load that the first model running unit can bear, by deploying another pre-trained model on the second model running unit and using the newly deployed pre-trained model to share the traffic pressure of the pre-trained model on the first model running unit, the efficiency of the data processing system in inference for the inference traffic of different users can be guaranteed, thereby ensuring the user experience.

[0011] In one possible implementation, the model deployment request is also used to request the deployment of a second fine-tuning model in the data processing system, and the second fine-tuning model is deployed in the second model running unit in at least one model running unit; then, the model deployment unit is also used to: when the load generated by the first pre-trained model processing the first business reasoning traffic is less than the second load threshold corresponding to the first model running unit, the second fine-tuning model in the second model running unit is migrated to the first model running unit. In this way, when the first model running unit can withstand the operation of more models, by deploying the pre-trained model and the fine-tuning model in the same model running unit (i.e., the first model running unit), this can fully utilize the computing power of the first model running unit and improve the overall performance of the data processing system, such as the second model running unit can enter a low power state, or can be used to execute other businesses, etc. No, when the fine-tuning model and the pre-training model jointly provide reasoning services, by deploying the fine-tuning model and the pre-training model in the same model running unit, the data interaction delay between the fine-tuning model and the pre-training model during the reasoning process can be reduced, thereby improving the overall reasoning efficiency.

[0012] In one possible implementation, the first model execution unit is configured to perform batch inference on the inference traffic of a first user for the first pre-trained model and the inference traffic of a second user for the second pre-trained model using the first pre-trained model. In this way, by batching the inference traffic of multiple users, the overall efficiency of inference on the traffic of multiple users can be improved.

[0013] In one possible embodiment, the first model operation unit is deployed at the first site, at least one model operation unit further includes a third model operation unit, the third model operation unit is deployed at the second site, and the third model operation unit is deployed with the first pre-trained model; the data processing system further includes a scheduling unit, the scheduling unit being used to: obtain first historical reasoning information of the first site processing the reasoning traffic of the second user based on the first pre-trained model, and second historical reasoning information of the second site processing the reasoning traffic of the second user based on the first pre-trained model; obtain a model reasoning request provided by the second user, the model reasoning request being used to request reasoning using the first pre-trained model; when the reasoning delay indicated by the first historical reasoning information is greater than the reasoning delay indicated by the second historical reasoning information, schedule the third model operation unit in the second site to process the model reasoning request. In this way, when there are multiple sites that have deployed the same pre-trained model, the data processing system can allocate the reasoning traffic to the site with a shorter reasoning delay for reasoning when processing the user's reasoning traffic for the pre-trained model, thereby improving the overall efficiency of the data processing system in processing the user's reasoning traffic.

[0014] In one possible implementation, the first model running unit is deployed at the first site, and the model deployment unit is further used to: obtain the first inference latency of the first site processing the model inference requests respectively provided by multiple users based on the first pre-trained model, where the multiple users include the first user and the second user; estimate the second inference latency of the second site processing the model inference requests respectively provided by multiple users using the first pre-trained model; when the second inference latency is less than the first inference latency, migrate the first pre-trained model from the first site to the second site. In this way, when there is a site with a shorter overall latency for inference using the pre-trained model, by migrating the pre-trained models on other sites to this site, the overall efficiency of the subsequent data processing system in processing the inference traffic of the multiple users can be improved, thereby improving the user experience.

[0015] In the second aspect, the present application provides a data processing method, characterized in that the data processing method is applied to a data processing system, the data system includes a model deployment unit and at least one model running unit, and the at least one model running unit includes a first model running unit; the method includes: the model deployment unit deploys the first pre-trained model requested to be deployed by the first user in the first model running unit; the model deployment unit obtains a model deployment request provided by the second user, and the model deployment request is used to request the deployment of the second pre-trained model in the data processing system; when the first pre-trained model is the same as the second pre-trained model, the model deployment unit configures the first model running unit to process the first business inference traffic based on the first pre-trained model, and the first business inference traffic includes the inference traffic of the first user for the first pre-trained model and the inference traffic of the second user for the second pre-trained model.

[0016] In one possible implementation, the first fine-tuning model requested to be deployed by the first user is deployed in the first model running unit, the first fine-tuning model is trained based on the first pre-trained model, and the model deployment request is also used to request deployment of a second fine-tuning model in the data processing system; the method also includes: when the second fine-tuning model is the same as the first fine-tuning model, the model deployment unit configures the first model running unit to process second business inference traffic based on the first fine-tuning model, and the second business inference traffic includes the first user's inference traffic for the first fine-tuning model and the second user's inference traffic for the second fine-tuning model.

[0017] In one possible embodiment, the method also includes: when the load generated by the first pre-trained model processing the first business inference traffic is greater than the first load threshold corresponding to the first model running unit, the model deployment unit deploys the second pre-trained model in the second model running unit of the at least one model running unit; the model deployment unit configures the first model running unit to process the first user's inference traffic for the first pre-trained model based on the first pre-trained model, and configures the second model running unit to process the second user's inference traffic for the second pre-trained model based on the second pre-trained model.

[0018] In one possible embodiment, the model deployment request is also used to request deployment of a second fine-tuning model in the data processing system, and the second fine-tuning model is deployed in a second model running unit among the at least one model running unit; the method also includes: when the load generated by the first pre-trained model processing the first business inference traffic is less than a second load threshold corresponding to the first model running unit, migrating the second fine-tuning model in the second model running unit to the first model running unit.

[0019] In a possible implementation, the method further includes: the first model running unit, using the first pre-trained model, performing batch inference on the first user's inference traffic for the first pre-trained model and the second user's inference traffic for the second pre-trained model.

[0020] In one possible embodiment, the first model running unit is deployed at the first site, the at least one model running unit further includes a third model running unit, the third model running unit is deployed at the second site, and the third model running unit is deployed with the first pre-trained model; the data processing system further includes a scheduling unit, and the method further includes: the scheduling unit obtains first historical reasoning information of the first site processing the reasoning traffic of the second user based on the first pre-trained model, and second historical reasoning information of the second site processing the reasoning traffic of the second user based on the first pre-trained model; the scheduling unit obtains a model inference request provided by the second user, and the model inference request is used to request reasoning using the first pre-trained model; when the reasoning delay indicated by the first historical reasoning information is greater than the reasoning delay indicated by the second historical reasoning information, the scheduling unit schedules the third model running unit in the second site to process the model inference request.

[0021] In one possible implementation, the first model running unit is deployed at a first site, and the method further includes: the model deployment unit obtains a first inference latency for the first site to process model inference requests respectively provided by multiple users based on the first pre-trained model, where the multiple users include the first user and the second user; the model deployment unit estimates a second inference latency for the second site to process the model inference requests respectively provided by the multiple users using the first pre-trained model; when the second inference latency is less than the first inference latency, the model deployment unit migrates the first pre-trained model from the first site to the second site.

[0022] The data processing method provided in the second aspect corresponds to the data processing system provided in the first aspect. Therefore, the technical effects of any implementation method in the second aspect can refer to the technical effects of the corresponding implementation method in the above-mentioned first aspect, and will not be repeated here.

[0023] In a third aspect, the present application provides a computing device cluster, wherein the computing device includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory; the at least one memory is used to store instructions, and the at least one processor executes the instructions stored in the at least one memory, so that the computing device cluster performs the data processing method in the above-mentioned second aspect or any possible implementation of the second aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The at least one computing device may also include a bus. The processor is connected to the memory via a bus. The memory may include a readable memory and a random access memory.

[0024] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on at least one computing device, the at least one computing device executes the data processing method in the above-mentioned second aspect or any possible implementation of the second aspect.

[0025] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on at least one computing device, enables the at least one computing device to execute the data processing method in the above-mentioned second aspect or any possible implementation of the second aspect.

[0026] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0028] FIG1 is a schematic diagram of the structure of an exemplary data processing system provided by the present application;

[0029] FIG2 is a schematic diagram of a fine-tuning model;

[0030] FIG3 is a flow chart of an exemplary data processing method provided by the present application;

[0031] FIG4 is a schematic diagram of a process for processing user inference traffic using a pre-trained model provided by the present application;

[0032] FIG5 is a flow chart of another exemplary data processing method provided by the present application;

[0033] FIG6 is a schematic diagram of the structure of a computing device provided by the present application;

[0034] FIG7 is a schematic diagram of the structure of a computing device cluster provided in this application. DETAILED DESCRIPTION

[0035] The following will describe the solutions in the embodiments provided in this application in conjunction with the drawings in this application.

[0036] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate and are merely used to describe the manner in which objects with the same attributes are described in the embodiments of this application.

[0037] Referring to Figure 1 , a schematic diagram of the structure of a data processing system is shown. As shown in Figure 1 , data processing system 10 includes a model deployment unit 101 and at least one model execution unit. For ease of understanding and description, Figure 1 illustrates an example system including three model execution units: model execution unit 102, model execution unit 103, and model execution unit 104.

[0038] The model deployment unit 101 is used to deploy an AI model for multiple users (such as operators, etc.) on the model operation unit 102 to the model operation unit 104. The AI ​​model can be a pre-trained model or a fine-tuned model.

[0039] Each model running unit can support the operation of one or more AI models, and when multiple AI models are running on the model running unit, the multiple AI models can be different models belonging to the same user, or can be models belonging to different users. The model running unit can be implemented by a physical device or a virtual device. When implemented by a physical device, the model running unit can be a physical machine (such as a bare metal server or processor, etc.); when implemented by a virtual device, the model running unit can be a virtual machine or a container. In actual application, the model running unit can include an inference framework, which includes a framework for deep learning / machine learning operation and business processing code corresponding to the AI ​​model. Among them, the business processing code can, for example, be a code that preprocesses the model input data into the input format of the model, or can be a code that post-processes the output of the AI ​​model into a response in Hypertext Transfer Protocol (HTTP) format, etc. In actual application, the model running unit can download and start running the inference framework from other storage systems or other storage devices during the startup process.

[0040] Furthermore, multiple model execution units in the data processing system 10 can be deployed at the same site or at different sites. For example, model execution unit 102 and model execution unit 103 are deployed at site 1, and model execution unit 104 is deployed at site 2. A site refers to a network node that provides services or network resources.

[0041] Among them, the fine-tuning model can be obtained by fine-tuning the pre-training model.

[0042] In the first implementation example, the fine-tuned model can be a new AI model generated by training a pre-trained model using training samples. In this case, the fine-tuned model and the pre-trained model have the same number of parameters. Furthermore, the reasoning process of the fine-tuned model and the pre-trained model can be independent of each other; that is, the pre-trained model does not need to participate in the reasoning process of the fine-tuned model.

[0043] In the second implementation example, the fine-tuning model may include multiple network layers, which can supplement the network layers in the pre-training model or replace some of the network layers in the pre-training model; at this time, the number of parameters in the fine-tuning model is much smaller than the number of parameters in the pre-training model.

[0044] For example, the fine-tuning model can be the low-rank adaptation of large language models (LORA) shown in Figure 2. During inference, the pre-trained model and the fine-tuning model will have the same model input; then, the pre-trained model and the fine-tuning model will perform inference based on the model input, and the inference results obtained by the two models will be added together to form the final inference result (also known as the output of the fine-tuning model). During this process, the pre-trained model needs to perform inference.

[0045] In actual application scenarios, the data processing system 10 can deploy AI models for multiple different users so that each user can provide corresponding cloud services based on the AI ​​model they requested to deploy. Since different users provide cloud services independently, the data processing system 10 usually deploys the AI ​​models requested by different users separately. However, when providing cloud services based on AI models, more memory resources and computing resources are required to support the operation of AI models. The memory resources can be, for example, high bandwidth memory (HBM), dual in-line memory module (DIMM) and other memories; the computing resources can be, for example, graphics processing unit (GPU), neural network processing unit (NPU), tensor processing unit (TPU) and other processors. At the same time, different users may request to deploy the same pre-trained model, such as different operators may deploy the same pre-trained model to provide cloud services with basic functions. At this time, the data processing system 10 deploys the same pre-trained model for different users separately, which will result in excessively high hardware costs for the data processing system 10 to support cloud services for multiple users, thereby limiting the number of users that the data processing system 10 can support.

[0046] Based on this, in the data processing system 10 provided in the present application, in the process of deploying AI models for multiple users, when there are multiple users requesting to deploy the same pre-trained model, the model deployment unit 101 will configure a deployed pre-trained model to multiple users so that the multiple users can provide corresponding cloud services based on the same shared pre-trained model.

[0047] Specifically, taking the deployment of pre-trained models by user 1 and user 2 as an example, the model deployment unit 101 deploys the pre-trained model A requested by user 1 in the model running unit 102. When user 2 requests to deploy pre-trained model B in the data processing system 10, the model deployment unit 101 obtains the model deployment request provided by user 2 and determines whether the pre-trained model B is the same as the deployed pre-trained model A. When the pre-trained model B requested to be deployed by user 2 is the same as the deployed pre-trained model A, the model deployment unit 101 configures pre-trained model A to handle user 1's inference traffic for pre-trained model A and user 2's inference traffic for pre-trained model B. That is, the model deployment unit 101 configures user 1 and user 2 to share the same pre-trained model to handle their respective inference traffic, without the need to deploy a separate pre-trained model B (the same as pre-trained model A) for user 2. In this way, when user 1 and user 2 can provide corresponding cloud services based on pre-trained model A, the computing resources and memory resources required to deploy the pre-trained model in the data processing system 10 can be reduced, that is, the hardware cost required to provide cloud services based on pre-trained models can be reduced.

[0048] In a further possible implementation, the model deployment unit 101 can also share the fine-tuning model in the data processing system 10 with multiple different users. Specifically, taking the deployment of fine-tuning models for user 1 and user 2 as an example, after the model deployment unit 101 obtains the fine-tuning model a trained according to the pre-trained model A, the fine-tuning model a can be deployed in the model running unit 102 (or in other model running units). Then, when user 2 requests to deploy the fine-tuning model b in the data processing system 10, the model deployment unit 101 can determine whether the fine-tuning model b requested to be deployed by user 2 is the same as the deployed fine-tuning model a. When the fine-tuning model b is the same as the deployed fine-tuning model a, the model deployment unit 101 configures the fine-tuning model a to handle the inference traffic of user 1 for the fine-tuning model a and the inference traffic of user 2 for the fine-tuning model b. In this way, when user 1 and user 2 can provide corresponding cloud services based on the fine-tuning model a, the computing resources and memory resources required for deploying the fine-tuning model in the data processing system 10 can be reduced, that is, the computing resources and memory resources required for separately deploying the fine-tuning model b can be saved, thereby reducing the hardware cost required to provide cloud services based on the fine-tuning model.

[0049] In addition, the data processing system 10 may further include a model storage unit 105 , a traffic distribution unit 106 , a billing unit 107 , a scheduling unit 108 , and the like.

[0050] The model storage unit 105 is used to store AI models supported by the data processing system 10. For example, it can store one or more pre-trained models, as well as fine-tuned models based on the pre-trained models. For example, the AI ​​models can be stored in object storage or other means. Thus, the model deployment unit 101 can deploy the AI ​​models in the model storage unit 105 to the corresponding model execution unit.

[0051] The traffic distribution unit 106 is used to distribute the inference traffic of multiple users for the pre-trained model or the fine-tuned model to the corresponding model running unit, so that the pre-trained model or the fine-tuned model on the model running unit can be used to process the traffic. Specifically, the pre-trained model or the fine-tuned model can be used to perform business reasoning, etc.

[0052] The billing unit 107 is used to charge users for the corresponding cloud services provided by the AI ​​model. For example, the billing can be based on the number of input tokens (representations) used as the model, for example, charging 1 yuan per 1,000 tokens; or the billing can be based on the statistics of the resources used to deploy the AI ​​model for the user. Furthermore, when the user performs model training in the data processing system 10, the billing unit 107 can also charge for it.

[0053] The scheduling unit 108 is used to schedule the inference request provided by the user to the appropriate site when different model running units in the data processing system 10 are located at different sites, so that the AI ​​model on the site can execute the corresponding inference process based on the inference request.

[0054] Optionally, when different model running units in the data processing system 10 are located at different sites, the model deployment unit 101 can also schedule the AI ​​model in the model running unit of the corresponding site to perform inference for the user, or migrate the AI ​​model deployed at one of the sites to other sites, etc.

[0055] In actual applications, the data processing system 10 may also include units with other functions to support the data processing system 10 with more functions, and this is not limited to this. For example, the data processing system 10 may also include a training unit for training the pre-trained model to obtain a fine-tuned model required for deployment by the user.

[0056] As an example, the data processing system 10 shown in FIG1 can be deployed in the cloud to provide users with cloud services for deploying AI models. As another example, the data processing system 10 shown in FIG1 can be deployed locally, etc., which is not limited to this.

[0057] In actual application, the data processing system 10 shown in FIG. 1 may be implemented through software or hardware.

[0058] The data processing system 10, as an example of a software functional unit, may include code running on a computing instance. The computing instance may include at least one of a host, a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the data processing system 10 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0059] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0060] As an example of a hardware functional unit, the data processing system 10 may include at least one computing device, such as a server. Alternatively, the data processing system 10 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), or any combination thereof.

[0061] The multiple devices included in the data processing system 10 can be distributed in the same region or in different regions. The multiple devices included in the data processing system 10 can be distributed in the same availability zone (AZ) or in different AZs, and each AZ includes a data center or multiple data centers with close geographical locations. Generally, a region can include multiple AZs. Similarly, the multiple devices included in the data processing system 10 can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually a VPC is set in a region, and cross-region communication between two VPCs in the same region and between VPCs in different regions requires setting up a communication gateway in each VPC, and interconnection between VPCs is achieved through the communication gateway.

[0062] Next, various non-limiting specific implementations of the data processing process are described in detail.

[0063] Refer to Figure 3, which is a flow chart of a data processing method in an embodiment of the present application. This method can be applied to the data processing system shown in Figure 1 above, or it can also be applied to other applicable application scenarios. The following description is based on the application to the data processing system 10 shown in Figure 1 as an example. In the data processing system 10 shown in Figure 1, the functions of each unit in the data processing system 10 are specifically described in the relevant description of the following embodiment.

[0064] The data processing method shown in FIG3 may specifically include:

[0065] S301 : The model storage unit 105 registers the AI ​​model to the model deployment unit 101 .

[0066] The AI ​​model can be a pre-trained model or a fine-tuned model based on a pre-trained model. For example, a pre-trained model can be a general conversation model (which performs poorly in specific domains), while a fine-tuned model can be trained using conversation samples from a specific domain to enable the fine-tuned model to conduct professional conversations with users in that specific domain, i.e., the conversation results output by the fine-tuned model better meet user expectations.

[0067] In actual application, users or operation and maintenance personnel can import the AI ​​model into the data processing system 10 in advance.

[0068] As some implementation examples, the model storage unit 105 can provide an application programming interface (API) to the outside, and the user can import the AI ​​model into the model storage unit 105 for storage by calling the API. Alternatively, the user can import the AI ​​model into the model storage unit 105 for storage through the control panel (Web UI console) in the Web front-end debugging tool provided by the data processing system 10, etc., and this is not limited to this.

[0069] Among them, when the AI ​​model stored in the model storage unit 105 includes a fine-tuning model, the user can import the trained fine-tuning model into the model storage unit 105. Alternatively, the user can use the training unit (not shown in Figure 1) in the data processing system 10 to train and obtain the fine-tuning model. Specifically, the user can select a pre-trained model from at least one pre-trained model stored in the model storage unit 105 by calling the API or Web UI console provided by the data processing system 10, and provide a training sample to the data processing system 10, so that the training unit can train the pre-trained model based on the training sample provided by the user, obtain the fine-tuning model desired by the user, and save the fine-tuning model in the model storage unit 105.

[0070] Alternatively, the model storage unit 105 may pre-store one or more AI models, for example, one or more AI model files may be pre-imported into the model storage unit 105 by operation and maintenance personnel.

[0071] For an AI model that has been saved, the model storage unit 105 can register the AI ​​model to the model deployment unit 101. For example, the model storage unit 105 can provide the identifier of one or more saved AI models to the model deployment unit 101, so that the model deployment unit 101 can know the AI ​​model that can be deployed in the data processing system 10.

[0072] After learning about deployable pre-trained or fine-tuned models, model deployment unit 101 can deploy AI models for multiple users on the model execution unit to meet their needs for cloud services based on AI models. For ease of understanding, this embodiment uses the example of model deployment unit 101 deploying AI models for user 1 and user 2, respectively.

[0073] S302: The model deployment unit 101 obtains a model deployment request 1 provided by user 1, where the model deployment request 1 is used to request deployment of a pre-trained model 1 in the data processing system 10.

[0074] For example, user 1 can send a model deployment request 1 to model deployment unit 101 through a client provided by data processing system 10. The client can be, for example, a web browser or an application running on a user terminal, and this is not limited to this. Model deployment request 1 is used to request deployment of a pre-trained model in data processing system 10. By carrying an identifier of pre-trained model 1 in model deployment request 1, the pre-trained model 1 requested by user 1 can be indicated from among multiple pre-trained models supported for deployment by model deployment unit 101.

[0075] For example, the model deployment unit 101 can present relevant information of multiple pre-trained models supported for deployment to user 1, including the functions and identification information of the pre-trained models, so that user 1 can select the pre-trained model 1 that user 1 expects to deploy from a variety of pre-trained models through the client, and the client generates a model deployment request 1 including the identification of the pre-trained model 1 based on the selection operation performed by user 1, and sends the model deployment request 1 to the model deployment unit 101.

[0076] In actual application, user 1 can also request the model deployment unit 101 to deploy the pre-trained model 1 in other ways, and there is no limitation on this.

[0077] S303: The model deployment unit 101 deploys the pre-trained model 1 on the model running unit 102.

[0078] The model deployment unit 101 can deploy the pre-trained model 1 to a suitable model running unit based on a variety of deployment strategies, which are exemplified below.

[0079] In the first implementation example, the model deployment unit 101 can adopt a polling strategy based on the load of each model running unit to deploy the pre-trained model 1 to the model running unit with sufficient load. The load of the model running unit can be measured, for example, by the load of the memory components (such as HBM, DIMM, etc.) and computing components (such as CPU, GPU, etc.) in the model running unit that support the operation of the AI ​​model.

[0080] Specifically, the model deployment unit 101 can obtain the load of each model running unit and estimate whether deploying the pre-trained model 1 on the model running unit 102 will cause the load on the model running unit 102 to exceed the threshold. If not, the model deployment unit 101 deploys the pre-trained model 1 to the model running unit 102. If not, the model deployment unit 101 estimates whether deploying the pre-trained model 1 on the next model running unit will cause the load on the next model running unit to exceed the threshold, and so on, until a model running unit whose load does not exceed the threshold is determined, and the pre-trained model 1 is deployed to the model running unit. In this embodiment, the deployment of the pre-trained model 1 on the model running unit 102 is used as an example for explanation.

[0081] The model deployment unit 101 can estimate the load added to the model running unit 102 after the pre-trained model is deployed on the model running unit 102 based on the inference traffic of the pre-trained model 1 per unit time. For example, the model deployment unit 101 can estimate the number of requests or tokens processed by the pre-trained model 1 per minute or per second, or the number of inferences performed by the pre-trained model 1 per hour, and thus estimate the load generated by the pre-trained model 1 per unit time.

[0082] In the second implementation example, the model deployment unit 101 can deploy the pre-trained model 1 to a suitable model running unit based on a load balancing strategy.

[0083] Specifically, the model deployment unit 101 can obtain the load of each model running unit and determine the model running unit 102 with the smallest load from multiple model running units (assuming that the load of model running unit 102 is the smallest). Before deploying the pre-trained model 1 to the model running unit 102, the model deployment unit 101 can estimate whether deploying the pre-trained model 1 to the model running unit 102 will cause the load on the model running unit to exceed the threshold. If so, the model deployment unit 101 can feedback to the user 1 that the deployment of the pre-trained model 1 failed; if not, the model deployment unit 101 deploys the pre-trained model 1 in the model running unit 102.

[0084] In the third implementation example, the model deployment unit 101 can estimate the latency of the pre-trained model 1 in processing inference traffic when running on each model running unit, and deploy the pre-trained model 1 on the model running unit 102 with the minimum latency (assuming that the model running unit 102 has the minimum latency in processing inference traffic).

[0085] Among them, the delay of processing inference traffic when the pre-trained model 1 is running on the model running unit 102 can be measured according to one or more dimensions of the time it takes for the model running unit 102 to obtain the inference traffic of user 1, the time it takes to feed back the inference results to user 1, or the time it takes for the pre-trained model 1 to complete inference on the model running unit 102, or it can be measured according to other factors, which is not limited to this.

[0086] S304: The model running unit 102 runs the pre-trained model 1 and uses the pre-trained model 1 to process the inference traffic of user 1.

[0087] Among them, the inference traffic of user 1 refers to the traffic generated when user 1 requests to use pre-trained model 1 for inference.

[0088] In actual application, before running the pre-trained model 1, the model running unit 102 can first verify whether an inference framework suitable for the pre-trained model 1 is deployed. If so, the model running unit 102 runs the pre-trained model 1 based on the inference framework to process the inference traffic of user 1. If not, the model running unit 102 can first obtain the inference framework, such as downloading and starting the inference framework from other storage systems or storage devices, and then run the pre-trained model 1 based on the inference framework.

[0089] While model operation unit 102 is processing user 1's inference traffic using pre-trained model 1, it can also perform batch optimization on user 1's inference traffic. That is, pre-trained model 1 can be used to batch process multiple portions of user 1's inference traffic, eliminating the need to wait for pre-trained model 1 to complete inference based on one portion of inference traffic before starting to execute inference based on the next portion of inference traffic. In this way, pre-trained model 1 can execute multiple inference processes in parallel, thereby improving the efficiency of pre-trained model 1 in processing user 1's inference traffic.

[0090] Similarly, when user 1 requests to deploy multiple pre-trained models in the data processing system 10, the model deployment unit 101 can deploy the multiple pre-trained models in one or more model running units in accordance with the above method, which will not be elaborated herein.

[0091] Furthermore, in addition to deploying the pre-trained model 1 for user 1, the model deployment unit 101 can also deploy one or more fine-tuned models generated based on the pre-trained model 1 for user 1. The model deployment unit 101 can also deploy the fine-tuned model for user 1 in the same manner as described above. For example, the model deployment unit 101 can deploy the fine-tuned model requested by user 1 on the model running unit 103 in the manner described above.

[0092] S305: The model deployment unit 101 obtains a model deployment request 2 provided by user 2, where the model deployment request 2 is used to request deployment of a pre-trained model 2 in the data processing system 10.

[0093] Typically, different users will independently request the deployment of pre-trained models in the data processing system 10. Furthermore, the pre-trained models requested by different users may be the same or different. Therefore, when user 2 requests the deployment of a pre-trained model in the data processing system 10, the model deployment unit 101 receives model deployment request 2 provided by user 2.

[0094] The specific implementation process of step S305 can be found in the relevant description of step S302 above, which will not be repeated here.

[0095] S306: When the pre-trained model 2 requested to be deployed by user 2 is the same as the deployed pre-trained model 1, the model deployment unit 101 configures the model operation unit 102 to process the business inference traffic 1 based on the pre-trained model 1, and the business inference traffic 1 includes the inference traffic of user 1 for pre-trained model 1 and the inference traffic of user 2 for pre-trained model 2.

[0096] Typically, pre-trained models can have large parameters, requiring more memory and computing resources to deploy. For example, if a user requests to deploy a pre-trained model with 200 quintillion (200B) parameters, the file size of the pre-trained model may reach 400GB. Deploying this pre-trained model may require 400GB of HBM memory and eight NPUs on the model execution unit.

[0097] To this end, in this embodiment, the model deployment unit 101 does not deploy pre-trained models for user 1 and user 2 separately. Instead, after receiving the model deployment request 2 provided by user 2, it first determines whether the pre-trained model 2 requested to be deployed by user 2 is the same pre-trained model as the deployed pre-trained model 1. If not, the model deployment unit 101 can deploy the pre-trained model 2 requested by user 2 on some model running units for user 2. If the pre-trained model 2 is the same pre-trained model as the pre-trained model 1, the model deployment unit 101 can reuse the deployed pre-trained model 1 to process the inference traffic of user 2, without having to redeploy an identical pre-trained model for user 2, thereby reducing the hardware cost required to deploy pre-trained models for multiple users.

[0098] In specific implementation, when the pre-trained model 2 requested to be deployed by user 2 is the same pre-trained model as the deployed pre-trained model 1, the model deployment unit 101 configures the model operation unit 102 to process the inference traffic of user 1 and user 2 based on the pre-trained model 1, that is, user 1 and user 2 share the same pre-trained model to implement the cloud services provided by each of them.

[0099] Furthermore, before configuring the pre-trained model 1 to user 2, the model deployment unit 101 can also estimate whether the load generated by using the pre-trained model 1 to process the business inference traffic (that is, the inference traffic of user 1 and the inference traffic of user 2) is greater than the load threshold 1 corresponding to the model operation unit 102. The load threshold 1 can be determined based on the remaining computing power of the model operation unit 102, or can be specified in advance by a technician. If not, the model deployment unit 101 configures the model operation unit 102 to process the inference traffic of user 1 and user 2 based on the pre-trained model 1. If so, it indicates that reusing the pre-trained model 1 to process the inference traffic of multiple users at the same time will cause the load of the model processing unit 102 to be too large (thereby reducing the effect of providing cloud services based on the AI ​​model for user 1 and user 2). At this time, the model deployment unit 101 can deploy the pre-trained model 2 separately for user 2 on the model operation unit 103 or the model operation unit 104. Alternatively, the model deployment unit 101 may migrate the pre-trained model 1 to a model running unit with a smaller load, and use the pre-trained model 1 on the migrated model running unit to simultaneously process the inference traffic of multiple users.

[0100] In the process of the model operation unit 102 using the pre-trained model 1 to process business inference traffic, the model operation unit 102 can batch process the inference traffic of user 1 for pre-trained model 1 and the inference traffic of user 2 for pre-trained model 2 (that is, pre-trained model 1) to improve the overall efficiency of the pre-trained model 1 in processing business inference traffic.

[0101] Similarly, when user 2 requests to deploy a fine-tuned model in data processing system 10, model deployment unit 101 obtains user 2's model deployment request for the fine-tuned model and determines whether the fine-tuned model requested by user 2 is the same as the fine-tuned model already deployed by user 1. If so, model deployment unit 101 can configure the deployed fine-tuned model for user 2 in the same manner as described above. At this point, the fine-tuned model can handle business inference traffic 2, which includes traffic requested by user 1 for inference using fine-tuned model 1 and traffic requested by user 2 for inference using fine-tuned model 2. If the fine-tuned model requested by user 2 is different from the already deployed fine-tuned model, model deployment unit 101 can deploy the fine-tuned model requested by user 2 on one of the model processing units for user 2.

[0102] In actual application, the load of the business inference traffic processed by the model operation unit 102 may vary greatly. For example, within a period of time (such as 24 hours), the inference traffic of user 1 or user 2 for the pre-trained model 1 may surge, resulting in the load generated by the pre-trained model processing the business inference traffic 1 being greater than the load threshold 1 corresponding to the model operation unit 102. At this time, the model operation unit 102 may cause the timeliness of the pre-trained model 1 in processing the business inference traffic of user 1 or user 2 to be reduced due to excessive load, such as part of the inference traffic of user 1 or user 2 failing to be processed by the pre-trained model 1 in a timely manner, thereby affecting the effect of user 1 or user 2 providing cloud services based on the pre-trained model 1.

[0103] To this end, in a further possible implementation, the model deployment unit 101 may newly deploy the pre-trained model 2 requested by user 2 in other model running units in the data processing system 10, such as deploying pre-trained model 2 on model running unit 103, when determining that the load generated by the pre-trained model 1 processing business inference traffic 1 is greater than the load threshold 1. Then, the model deployment unit 101 may configure the model running unit 102 to process user 1's inference traffic for pre-trained model 1 based on pre-trained model 1, and configure the model running unit 103 to process user 2's inference traffic for pre-trained model 2 based on pre-trained model 2. In this way, the loads generated by the pre-trained model 1 and the pre-trained model 2 processing the user's inference traffic are both within the load range that can be borne by the model running units in which they are located, thereby ensuring the reliability of the cloud services provided to user 1 and user 2 based on the pre-trained models.

[0104] Similarly, when the data processing system 10 processes the business inference traffic 2 of multiple users for the fine-tuning model based on the same fine-tuning model, the model deployment unit 101 can deploy a new fine-tuning model for user 2 on other model running units when the load generated by the fine-tuning model processing the business inference traffic 2 is too large, and use the same fine-tuning model on different model running units to process the inference traffic of different users for the fine-tuning model, so as to ensure the reliability of cloud services provided by multiple users based on the fine-tuning model.

[0105] In addition, when the load generated by the pre-trained model 1 processing the business inference traffic 1 is small, the model deployment unit 101 can also migrate and fine-tune the model between different model running units to save the overall power consumption of multiple model running units.

[0106] As an implementation example, assume that user 2 provides a model deployment request requesting the deployment of both pre-trained model 2 and fine-tuned model 2. Model deployment unit 101 configures user 1 and user 2 to share the same pre-trained model (i.e., pre-trained model 2) on model execution unit 102, and deploys fine-tuned model 2 for user 2 on model execution unit 103. Model deployment unit 101 can then detect whether the load generated by pre-trained model 1 processing business inference traffic 1 is less than a corresponding load threshold 2 for model execution unit 102. This load threshold 2 can, for example, be pre-configured by technical personnel based on the computing power of model execution unit 102. Furthermore, if the load generated by pre-trained model 1 processing business inference traffic 1 is less than load threshold 2, model deployment unit 101 can migrate fine-tuned model 2 running on model execution unit 103 to model execution unit 102. At this point, the total load generated by the execution of pre-trained model 1 and fine-tuned model 2 on model execution unit 102 remains within the load that model execution unit 102 can withstand. Accordingly, after the load of the model operation unit 103 is reduced, the model operation unit 103 can operate based on lower power consumption, such as entering a low-power operation mode. Similarly, the fine-tuned model or pre-trained model running on the model operation unit 104 can also be migrated to the model operation unit 102, and the model operation unit 104 can enter an even lower power consumption operation state. In this way, the overall power consumption of the multiple model operation units in the data processing system 10 can be reduced, thereby achieving energy saving of the data processing system 10.

[0107] It can be understood that in actual application, the data processing system 10 can deploy the pre-trained model and fine-tuning model requested by more than three (including three) users. At this time, the model deployment unit 101 can refer to the above method to configure more than three users to share the same pre-trained model or fine-tuning model, which will not be elaborated.

[0108] In the embodiment shown in FIG3 above, the process of the model deployment unit 101 deploying pre-trained models and fine-tuning models for multiple users is mainly introduced. The following, in conjunction with FIG4 , introduces the process of the data processing system 10 using the pre-trained model to process the user's inference traffic.

[0109] 4, which shows a flow chart of how the data processing system 10 processes user inference traffic using a pre-trained model. As shown in FIG4, the flow may specifically include:

[0110] S401: The traffic distribution unit 106 obtains a model inference request provided by user 2 for pre-trained model 1.

[0111] In this embodiment, after the model deployment unit 101 deploys the pre-trained model 1 for user 2 and user 2, the data processing system 10 can use the deployed pre-trained model 1 to process the inference traffic of user 2 and user 2 for the pre-trained model 1.

[0112] In a specific implementation, user 2 may send a model inference request to the traffic distribution unit 106 through the client. The model inference request may include the data used as input of the pre-trained model 1 when user 2 requests inference, and may also include the identifier of the pre-trained model 1. The model inference request received by the traffic distribution unit 106 may be the inference traffic of user 2.

[0113] S402 : The traffic distribution unit 106 distributes the model inference request provided by user 2 to the model operation unit 102 .

[0114] In actual application scenarios, the model deployment unit 101 deploys different pre-trained models or fine-tuned models for different users in different model operation units. Therefore, after receiving a model inference request, the traffic distribution unit 106 can distribute the model inference request to the model operation unit 102 where the pre-trained model 1 is located.

[0115] Exemplarily, the traffic distribution unit 106 can send a query request to the model deployment unit 101 based on the identifier of the pre-trained model 1 carried in the model inference request, requesting to query one or more model running units including the pre-trained model 1. Then, the model deployment unit 101 can return the identifier of the model running unit 102 to the traffic distribution unit 106. For example, the model deployment unit 101 can determine the model running unit 102 with a smaller load and running the pre-trained model 1 based on the load of each model running unit and the AI ​​model running thereon, and feed back the identifier of the model running unit 102 to the traffic distribution unit 106. Thus, the traffic distribution unit 106 sends the model inference request to the model running unit 102.

[0116] In actual application, there may be multiple model running units in the data processing system 10, all of which are deployed with the pre-trained model 1. For example, multiple model running units are located in different sites, and user 2 requests that the pre-trained model 1 be deployed on multiple sites. At this time, the model deployment unit 101 can randomly determine the site for processing the model inference request from multiple sites, and further feedback the identifier of the model running unit in the site for processing the model inference request to the traffic distribution unit 106. Alternatively, the traffic distribution unit 106 can determine the site with the smallest communication delay based on the communication delay between each site and user 2, and further feedback the identifier of the model running unit in the site for processing the model inference request to the traffic distribution unit 106, and there is no limitation on this.

[0117] S403: The model running unit 102 responds to the received model inference request and executes the corresponding inference process.

[0118] The model inference request may include input data of the model, so that the model running unit 102 can use the pre-trained model 1 to perform inference based on the input data to obtain corresponding inference results.

[0119] In one possible implementation, the model running unit 102 may run an inference framework and use the inference framework to call the pre-trained model 1. Then, the model running unit 102 may input the model input data carried in the model inference request into the pre-trained model 1, so that the pre-trained model 1 outputs the corresponding inference result.

[0120] In a further possible implementation, when the model inference request carries multiple copies of model input data (or there are multiple users providing different model inference requests, etc.), the model running unit 102 can batch optimize the multiple copies of model input data so that the pre-trained model 1 can perform batch inference on the multiple copies of model input data, thereby improving the overall inference efficiency of the pre-trained model 1 for the multiple copies of model input data.

[0121] In actual application, the data processing system 10 also supports a billing function for the user 2. Specifically, this embodiment may further include the following steps.

[0122] S404 : The model running unit 102 collects and calculates billing information, and sends the collected billing information to the billing unit 107 .

[0123] S405: The billing unit 107 generates a billing result according to the billing information.

[0124] The model running unit 102 may use pre-configured rules to collect billing information.

[0125] For example, the model running unit 102 may count the number of tokens used as input to the pre-trained model 1, or count the number of tokens output by the pre-trained model 1 after inference is completed, and use the number of input or output tokens as billing information. Generally, the greater the number of input or output tokens, the higher the fee for using the pre-trained model 1.

[0126] For another example, the model running unit 102 may count the number of resources occupied by the pre-trained model 1 during the inference process or the number of times the inference is performed, and use the number of resources occupied or the number of times the inference is performed as billing information. Generally, the more resources occupied, the higher the usage fee for the pre-trained model 1; the more times the pre-trained model 1 performs inference, the higher the usage fee for the pre-trained model 1.

[0127] In other embodiments, the model running unit 102 may also collect other relevant information of the pre-trained model 1 during the inference process and use it as billing information, which is not limited to this.

[0128] Then, the billing unit 107 can generate a corresponding billing result based on the billing information, so as to perform a charging operation for user 2's use of the pre-trained model 1. For example, assuming that the billing information specifically refers to the number of tokens input into the pre-trained model 1, the billing unit 107 can increase the charge by 1 yuan for every 1,000 tokens. Thus, the billing unit 107 can determine the final charge amount based on the number of tokens. For example, when the number of tokens input into the pre-trained model 1 is 38,150, the billing unit 107 can determine the final charge amount to be 38 yuan, etc.

[0129] In actual applications, the billing unit 107 may also charge for other user behaviors. For example, when user 1 or user 2 requests to deploy a pre-trained model or a fine-tuned model in the data processing system 10, the billing unit 107 may charge based on the amount of resources used for model deployment.

[0130] It is worth noting that in the embodiment shown in FIG4 above, the example of user 2 requesting to use pre-trained model 1 is used for illustration. When other users (such as user 1) request to use pre-trained model 1, the traffic distribution unit 106 and the model operation unit 102 can also be processed in a similar manner as described above. Furthermore, when user 1 or user 2 requests to use the fine-tuned model deployed in the data processing system 10, the traffic distribution unit 106 and the corresponding model operation unit can also use a similar manner as described above to implement the use of the fine-tuned model to process the user's inference traffic, which will not be described in detail.

[0131] In the embodiments shown in Figures 3 and 4 above, model execution unit 102 and model execution unit 103 in data processing system 10 can be deployed at the same site. In actual applications, the multiple model execution units included in data processing system 10 can be located at different sites. The following example uses the deployment of model execution unit 102 at site 1 and model execution unit 104 at site 2 as an example. Pre-trained model 1 can be deployed at both site 1 and site 2.

[0132] Referring to Figure 5, a flow chart of another data processing method is shown. As shown in Figure 5, the method may specifically include:

[0133] S501: The scheduling unit 108 collects relevant information of site 1 and site 2.

[0134] In actual application, the scheduling unit 108 can periodically (such as every 10 minutes, etc.) collect relevant information of each site. The collected relevant information of the site may include, for example, the function and identification of the AI ​​model included in the site, the network location of the site (such as IP address), the number of model running units in the site, the load of the site, the historical reasoning information of each site based on the AI ​​model deployed thereon to process the reasoning traffic of each user, etc., or it may be other information that can be used to participate in the decision-making of which site to schedule to process the user's reasoning traffic, and there is no limitation on this.

[0135] S502: The scheduling unit 108 receives a model inference request provided by user 2 through the client.

[0136] In this embodiment, when multiple model running units in the data processing system 10 are deployed at multiple sites, the scheduling unit 108 can schedule the model inference request in the entire domain, specifically, scheduling the model inference request to one of the sites for inference.

[0137] In one possible implementation, after receiving the model inference request, the scheduling unit 108 may perform domain name resolution on the model inference request to determine site 1 and site 2 corresponding to the same domain name. The scheduling unit 108 may then determine the site for processing the inference traffic of user 2 based on the collected information about site 1 and site 2, assuming it is site 1.

[0138] As a first implementation example, the scheduling unit 108 may determine, based on the loads of site 1 and site 2, that the load of site 1 is smaller. Then, the scheduling unit 108 may determine that site 1 processes the inference traffic of user 2.

[0139] As a second implementation example, the site-related information collected by the scheduling unit 108 may include historical inference information 1 of site 1 processing user 2's inference traffic based on pre-trained model 1, and historical inference information 2 of site 1 processing user 2's inference traffic based on pre-trained model 1. The scheduling unit 108 can then compare the inference latency indicated by historical inference information 1 with the inference latency indicated by historical inference information 2. When the inference latency indicated by historical inference information 1 is less than or equal to the inference latency indicated by historical inference information 2, the scheduling unit 108 schedules site 1 to process user 2's inference traffic. When the inference latency indicated by historical inference information 1 is greater than the inference latency indicated by historical inference information 2, the scheduling unit 108 schedules site 2 to process user 2's inference traffic. In this way, the scheduling unit 108 determines the site to process the model processing request based on the inference latency of each site processing user 2's inference traffic using pre-trained model 1 over a past period of time. This can improve the overall efficiency of the data processing system 10 in processing user model inference requests, thereby improving the effectiveness of the cloud services provided by users based on AI models, such as improving the responsiveness of the cloud services provided by users.

[0140] S503: The scheduling unit 108 feeds back the IP address of site 1 to the client of user 2.

[0141] S504: The client of user 2 sends a model inference request to the traffic distribution unit 106 in site 1 according to the received IP address.

[0142] The model inference request includes data as input to the pre-trained model 1.

[0143] S505 : The traffic distribution unit 106 distributes the model inference request to the model execution unit 102 .

[0144] That is, the traffic distribution unit 106 distributes the user's inference traffic to the model operation unit 102 .

[0145] S506: The model running unit 102 executes a corresponding reasoning process for the received reasoning traffic.

[0146] In a further possible implementation, the data processing system 10 can further improve the overall efficiency of the data processing system 10 in processing user model inference requests by adjusting the deployment of pre-trained models (and fine-tuned models).

[0147] In a specific implementation, taking the adjustment of the site deployment of the pre-trained model 1 in the data processing system 10 as an example, it is assumed that the pre-trained model 1 is only deployed at site 1 (other pre-trained models are deployed at site 2, but the pre-trained model 1 is not deployed). Then, the model deployment unit 101 can obtain the inference delay 1 of site 1 processing model inference requests respectively provided by multiple users based on the pre-trained model 1. The multiple users include user 1 and user 2 who share the pre-trained model 1. Then, the model deployment unit 101 can estimate the inference delay 2 of site 2 processing model inference requests respectively provided by multiple users using the pre-trained model 1. For example, the model deployment unit 101 can estimate the inference delay a of the pre-trained model 1 processing model processing requests at site 2 based on the performance of the hardware at site 2 that can support the operation of the pre-trained model 1, and estimate the average delay b between site 2 and obtaining the model inference requests of each user, and the average delay c of site 2 feeding back the inference results to each user. Thus, the model deployment unit 101 can use the sum of the inference delay a, the average delay b, and the average delay c as the estimated inference delay 2. Then, the model deployment unit 101 can compare the inference delay 1 and the inference delay 2, and when the inference delay 2 is less than the inference delay 1, the model deployment unit 101 migrates the pre-trained model 1 from site 1 to site 2; or, when the inference delay 2 is less than the inference delay 1 and the difference between the inference delay 2 and the inference delay 1 is greater than the delay threshold, the model deployment unit 101 migrates the pre-trained model 1 from site 1 to site 2, specifically, migrating the pre-trained model 1 from the model running unit 102 in site 1 to the model running unit 104 in site 2.

[0148] In this way, after adjusting the site where the pre-trained model 1 is deployed, the inference delay generated when the data processing system 10 processes the model inference requests of multiple users who share the pre-trained model 1 can be reduced, thereby improving the overall efficiency of the data processing system 10 in processing the user's model inference requests, thereby further improving the effect of users providing cloud services based on AI models.

[0149] It should be noted that the above-mentioned adjustment of the site deployment of the pre-trained model 1 in the data processing system 10 based on the delay of each site using the pre-trained model 1 to process the user inference traffic is only an implementation example. In other embodiments, the model deployment unit 101 can also adjust the deployment of the pre-trained model (or fine-tuning model) in the site based on other factors. For example, the model deployment unit 101 can adjust the site deployment of the AI ​​model in the data processing system 10 based on one or more factors including the number of model inference requests generated by the region served by each site, the resources of each site, the AI ​​models currently deployed at each site, and the number of resources in the site required to be occupied during the AI ​​model inference process. Alternatively, the site deployment of the AI ​​model in the data processing system 10 can be adjusted based on other types of factors, and there is no limitation on this.

[0150] In the embodiments shown in Figures 3 to 5 above, the multiple units in the data processing system 10 involved in the data processing process, such as the model deployment unit 101, the traffic distribution unit 106, the billing unit 107, the scheduling unit 108, etc., can be software configured on a computing device or a computing device cluster, and by running the software on the computing device or the computing device cluster, the computing device or the computing device cluster can realize the functions of the multiple units in the above-mentioned data processing system 10. Among them, each model running unit and model storage unit in the data processing system 10 can be implemented by the computing device or the hardware in the computing device. Below, based on the perspective of hardware device implementation, a detailed introduction is given to the multiple units in the data processing system 10 involved in the data processing process.

[0151] Figure 6 shows a structural diagram of a computing device, on which the model deployment unit 101 to the scheduling unit 108 in the above-mentioned data processing system 10 can be deployed. The computing device can be a computing device in a cloud environment (such as a server), or a computing device in an edge environment, etc., which can be specifically used to implement the methods executed by the embodiments shown in Figures 3 to 5 above.

[0152] As shown in Figure 6, the computing device 600 includes a processor 610, a memory 620, a communication interface 630 and a bus 640. The processor 610, the memory 620 and the communication interface 630 communicate with each other through the bus 640. The bus 640 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in Figure 6, but it does not mean that there is only one bus or one type of bus. The communication interface 630 is used to communicate with the outside, such as receiving a model deployment request provided by a user.

[0153] The processor 610 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits. The processor 610 may also be an integrated circuit chip with signal processing capabilities. During implementation, the functions of the multiple units included in the data processing system 10 may be performed by hardware integrated logic circuits or software instructions in the processor 610. The processor 610 may also be a general-purpose processor, a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The methods disclosed in the embodiments of this application may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 620 , and the processor 610 reads the information in the memory 620 and completes part or all of the functions of the multiple units included in the data processing system 10 in combination with its hardware.

[0154] The memory 620 may include a volatile memory, such as a random access memory (RAM), or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a HDD, or an SSD.

[0155] The memory 620 stores executable codes, and the processor 610 executes the executable codes to perform the methods performed by the multiple units in the aforementioned data processing system 10 .

[0156] Specifically, in the case of implementing the embodiments shown in Figures 3 to 5, and the model deployment unit 101, traffic distribution unit 106, billing unit 107, scheduling unit 108 and other units described in the embodiments shown in Figures 3 to 5 are implemented by software, the software or program code required to execute the functions of the multiple units in Figures 3 to 5 is stored in the memory 620, and the interaction between the multiple units included in the data processing system 10 and other devices is implemented through the communication interface 630. The processor is used to execute the instructions in the memory 620 to implement the method executed in the above-mentioned data processing system 10.

[0157] FIG7 is a schematic diagram illustrating the structure of a computing device cluster. The computing device cluster 70 shown in FIG7 includes multiple computing devices, and the multiple units in the data processing system 10 described above can be distributedly deployed on the multiple computing devices in the computing device cluster 70. As shown in FIG7 , the computing device cluster 70 includes multiple computing devices 700, each of which includes a memory 720, a processor 710, a communication interface 730, and a bus 740. The memory 720, the processor 710, and the communication interface 730 are communicatively connected to each other via the bus 740.

[0158] Processor 710 can be implemented as a CPU, GPU, ASIC, or one or more integrated circuits. Processor 710 can also be an integrated circuit chip with signal processing capabilities. During implementation, some functions of the multiple units included in data processing system 10 can be performed by hardware integrated logic circuits or software instructions in processor 710. Processor 710 can also be a DSP, FPGA, general-purpose processor, other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and can implement or execute some of the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in storage media well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory 720. In each computing device 700, processor 710 reads information from memory 720 and, in conjunction with its hardware, can perform some of the functions of the multiple units included in data processing system 10.

[0159] The memory 720 may include ROM, RAM, static storage devices, dynamic storage devices, hard disks (e.g., SSD, HDD), etc. The memory 720 may store program codes, for example, part or all of the program codes for implementing the model deployment unit 101, part or all of the program codes for implementing each model operation unit, part or all of the program codes for implementing the model storage unit 105, part or all of the program codes for implementing the traffic distribution unit 106, part or all of the program codes for implementing the billing unit 107, part or all of the program codes for implementing the scheduling unit 108, etc. For each computing device 700, when the program code stored in the memory 720 is executed by the processor 710, the processor 710 executes part of the methods executed by multiple units in the data processing system 10 based on the communication interface 730, such as a part of the computing devices 700 may be used to execute the method executed by the above-mentioned model deployment unit 101, and another part of the computing devices 700 may be used to execute the methods executed by the above-mentioned each model operation unit. The memory 720 may also store data, for example, intermediate data or result data generated by the processor 710 during execution, for example, the above-mentioned inference traffic, billing information, etc.

[0160] The communication interface 730 in each computing device 700 is used for external communication, such as interacting with other computing devices 700 .

[0161] The bus 740 may be a peripheral component interconnect standard bus or an extended industry standard architecture bus, etc. For ease of illustration, the bus 740 in each computing device 700 in FIG7 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0162] The plurality of computing devices 700 establish communication paths through a communication network to implement the functions of the plurality of units in the data processing system 10. Any computing device may be a computing device in a cloud environment (eg, a server) or a computing device in an edge environment.

[0163] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on one or more computing devices, the one or more computing devices execute the method executed in the data processing system 10 of the above embodiment.

[0164] In addition, embodiments of the present application further provide a computer program product. When the computer program product is executed by one or more computing devices, the one or more computing devices perform any of the aforementioned data processing methods. The computer program product may be a software installation package. When any of the aforementioned data processing methods is required, the computer program product may be downloaded and executed on a computer.

[0165] It should also be noted that the embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the embodiment drawings provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0166] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0167] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0168] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A data processing system, characterized in that: The data system includes a model deployment unit and at least one model running unit, wherein the at least one model running unit includes a first model running unit; The model deployment unit is used to: Deploy the first pre-trained model requested by the first user to be deployed in the first model running unit; Obtaining a model deployment request provided by a second user, where the model deployment request is used to request deployment of a second pre-trained model in the data processing system; When the first pre-trained model is the same as the second pre-trained model, the first model running unit is configured to process the first business inference traffic based on the first pre-trained model, and the first business inference traffic includes the inference traffic of the first user for the first pre-trained model and the inference traffic of the second user for the second pre-trained model.

2. The data processing system according to claim 1, wherein: The first fine-tuning model requested to be deployed by the first user is deployed in the first model running unit, the first fine-tuning model is trained based on the first pre-trained model, and the model deployment request is further used to request deployment of a second fine-tuning model in the data processing system; The model deployment unit is further configured to: When the second fine-tuning model is the same as the first fine-tuning model, the first model running unit is configured to process the second business inference traffic based on the first fine-tuning model, and the second business inference traffic includes the inference traffic of the first user for the first fine-tuning model and the inference traffic of the second user for the second fine-tuning model.

3. The data processing system according to claim 1, wherein: The model deployment unit is further configured to: When the load generated by the first pre-trained model processing the first business inference traffic is greater than a first load threshold corresponding to the first model running unit, the second model running unit in the at least one model running unit deploys the second pre-trained model; Configure the first model running unit to process the first user's inference traffic for the first pre-trained model based on the first pre-trained model, and configure the second model running unit to process the second user's inference traffic for the second pre-trained model based on the second pre-trained model.

4. The data processing system according to claim 1 or 2, characterized in that The model deployment request is further used to request deployment of a second fine-tuning model in the data processing system, the second fine-tuning model being deployed in a second model running unit among the at least one model running unit; The model deployment unit is further configured to: When the load generated by the first pre-trained model processing the first business inference traffic is less than the second load threshold corresponding to the first model running unit, the second fine-tuning model in the second model running unit is migrated to the first model running unit.

5. The data processing system according to any one of claims 1 to 4, characterized in that: The first model running unit is configured to: The first pre-trained model is used to perform batch inference on the inference traffic of the first user for the first pre-trained model and the inference traffic of the second user for the second pre-trained model.

6. The data processing system according to any one of claims 1 to 5, characterized in that: The first model operation unit is deployed at the first site, the at least one model operation unit further includes a third model operation unit, the third model operation unit is deployed at the second site, and the third model operation unit is deployed with the first pre-trained model; The data processing system further includes a scheduling unit, wherein the scheduling unit is configured to: Obtain first historical reasoning information of the first site processing the reasoning traffic of the second user based on the first pre-trained model, and second historical reasoning information of the second site processing the reasoning traffic of the second user based on the first pre-trained model; Obtaining a model inference request provided by the second user, where the model inference request is used to request inference using the first pre-trained model; When the inference delay indicated by the first historical reasoning information is greater than the inference delay indicated by the second historical reasoning information, the third model running unit in the second site is scheduled to process the model inference request.

7. The data processing system according to any one of claims 1 to 5, characterized in that: The first model running unit is deployed at the first site, and the model deployment unit is further configured to: Obtaining a first inference latency for the first site to process model inference requests respectively provided by multiple users based on the first pre-trained model, where the multiple users include the first user and a second user; estimating a second inference latency for the second site to process the model inference requests respectively provided by the multiple users using the first pre-trained model; When the second inference latency is less than the first inference latency, the first pre-trained model is migrated from the first site to the second site.

8. A data processing method, characterized in that: The data processing method is applied to a data processing system, wherein the data system includes a model deployment unit and at least one model running unit, wherein the at least one model running unit includes a first model running unit; The method comprises: The model deployment unit deploys the first pre-trained model requested to be deployed by the first user on the first model running unit; The model deployment unit obtains a model deployment request provided by a second user, where the model deployment request is used to request deployment of a second pre-trained model in the data processing system; When the first pre-trained model is the same as the second pre-trained model, the model deployment unit configures the first model operation unit to process the first business inference traffic based on the first pre-trained model, and the first business inference traffic includes the inference traffic of the first user for the first pre-trained model and the inference traffic of the second user for the second pre-trained model.

9. The method according to claim 8, characterized in that The first fine-tuning model requested to be deployed by the first user is deployed in the first model running unit, the first fine-tuning model is trained based on the first pre-trained model, and the model deployment request is further used to request deployment of a second fine-tuning model in the data processing system; The method further comprises: When the second fine-tuning model is the same as the first fine-tuning model, the model deployment unit configures the first model running unit to process second business inference traffic based on the first fine-tuning model, and the second business inference traffic includes the inference traffic of the first user for the first fine-tuning model and the inference traffic of the second user for the second fine-tuning model.

10. The method according to claim 8, characterized in that The method further comprises: When the load generated by the first pre-trained model processing the first business inference traffic is greater than a first load threshold corresponding to the first model running unit, the model deployment unit deploys the second pre-trained model in the second model running unit of the at least one model running unit; The model deployment unit configures the first model running unit to process the first user's inference traffic for the first pre-trained model based on the first pre-trained model, and configures the second model running unit to process the second user's inference traffic for the second pre-trained model based on the second pre-trained model.

11. The method according to claim 8 or 9, characterized in that The model deployment request is further used to request deployment of a second fine-tuning model in the data processing system, the second fine-tuning model being deployed in a second model running unit among the at least one model running unit; The method further comprises: When the load generated by the first pre-trained model processing the first business inference traffic is less than the second load threshold corresponding to the first model running unit, the model deployment unit migrates the second fine-tuning model in the second model running unit to the first model running unit.

12. The method according to any one of claims 8 to 11, characterized in that The method further comprises: The first model running unit uses the first pre-trained model to perform batch inference on the inference traffic of the first user for the first pre-trained model and the inference traffic of the second user for the second pre-trained model.

13. The method according to any one of claims 8 to 12, characterized in that The first model operation unit is deployed at the first site, the at least one model operation unit further includes a third model operation unit, the third model operation unit is deployed at the second site, and the third model operation unit is deployed with the first pre-trained model; The data processing system further includes a scheduling unit, and the method further includes: The scheduling unit obtains first historical reasoning information of the first site processing the reasoning traffic of the second user based on the first pre-trained model, and second historical reasoning information of the second site processing the reasoning traffic of the second user based on the first pre-trained model; The scheduling unit obtains a model inference request provided by the second user, where the model inference request is used to request inference using the first pre-trained model; When the inference delay indicated by the first historical inference information is greater than the inference delay indicated by the second historical inference information, the scheduling unit schedules the third model running unit in the second site to process the model inference request.

14. The method according to any one of claims 8 to 12, characterized in that The first model running unit is deployed at the first site, and the method further includes: The model deployment unit obtains a first inference latency for the first site to process model inference requests respectively provided by multiple users based on the first pre-trained model, where the multiple users include the first user and a second user; The model deployment unit estimates a second inference latency for the second site to process the model inference requests respectively provided by the multiple users using the first pre-trained model; When the second inference latency is less than the first inference latency, the model deployment unit migrates the first pre-trained model from the first site to the second site.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device including a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the computing device cluster performs the method according to any one of claims 8 to 14.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, which, when executed on at least one computing device, enable the at least one computing device to perform the method according to any one of claims 8 to 14.

17. A computer program product comprising instructions, characterized in that When the method is executed on at least one computing device, the method enables the at least one computing device to execute the method according to any one of claims 8 to 14.

Images