Multi-machine task processing method and device, equipment, storage medium and program product

Abstract

The invention provides a multi-machine task processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining a target task request, and determining a plurality of target virtual machines corresponding to the target task request from a plurality of virtual machines corresponding to at least two hosts; the multiple target virtual machines are located in at least two hosts; a physical network card and a physical GPU resource are deployed in each host; the virtual machine comprises a virtual network card and virtual GPU resources; the at least one virtual network card corresponds to the same physical network card, and the at least one virtual GPU resource corresponds to the same physical GPU resource; different virtual machines and different hosts communicate through the target network; and executing the target task request through the plurality of target virtual machines, and outputting a target result corresponding to the target task request. Thus, the target task request is executed based on the computing resources in the hosts, the task processing efficiency can be improved, and meanwhile the task processing real-time performance and the response speed can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to a multi-machine task processing method, apparatus, device, storage medium, and program product. Background Art

[0002] With the continuous development of technologies such as Artificial Intelligence (AI), the scale and complexity of various models have been increasing, and the computing resources required for model inference tasks are also increasing.

[0003] In related technologies, model inference tasks in virtualization scenarios are usually implemented based on the computing resources within a single host. Due to the limited computing resources within a single host, the efficiency of this task processing method in related technologies is low, the real-time performance is not high, and the response speed is also slow. Summary of the Invention

[0004] Multiple aspects of this application provide a multi-machine task processing method, apparatus, device, storage medium, and program product, which can perform task processing based on the computing resources of multiple hosts, improve task processing efficiency, and at the same time improve the real-time performance and response speed of task processing.

[0005] In a first aspect, an embodiment of this application provides a multi-machine task processing method, including:

[0006] Obtain a target task request, and determine multiple target virtual machines corresponding to the target task request from multiple virtual machines corresponding to at least two hosts; the multiple target virtual machines are located in at least two hosts; a physical network card and physical GPU resources are deployed in each host; the virtual machine includes a virtual network card and virtual GPU resources; at least one of the virtual network cards corresponds to the same physical network card, and at least one of the virtual GPU resources corresponds to the same physical GPU resource; different virtual machines and different hosts communicate through a target network;

[0007] Execute the target task request through the multiple target virtual machines, and output a target result corresponding to the target task request.

[0008] In a possible implementation manner, the determining the multiple target virtual machines corresponding to the target task request includes:

[0009] Collect status information of virtual machines in each host;

[0010] Determine multiple target virtual machines corresponding to the target task request according to the target quality of service tags included in the target task request and the status information; the target quality of service tags include at least one of the number of GPUs, priority information, task processing latency, the number of model parameters, and task processing accuracy corresponding to the target task request.

[0011] In a possible implementation manner, the method further includes:

[0012] For each host, virtualize the physical network card in the host to obtain multiple virtual network cards corresponding to the physical network card;

[0013] Configure the virtual network card in each virtual machine corresponding to the host through an input / output memory management unit, so that the virtual machine communicates with the virtual network card through a network driver.

[0014] In a possible implementation manner, at least one containerized application is included in the virtual machine; the method further includes:

[0015] When the status information of the virtual machines in each host meets a preset expansion condition, create a new virtual machine in the host according to a pre-configured virtual machine image, and schedule at least one containerized application to the new virtual machine; the preset expansion condition includes that the GPU utilization rate meets a first preset condition, the average CPU load meets a second preset condition, the memory occupancy ratio meets a third preset condition, or the task request change amount meets a fourth preset condition.

[0016] In a possible implementation manner, the virtual machine is connected to a target cloud service platform through a virtual cloud service network, and receives a configuration instruction issued by the target cloud service platform through the virtual cloud service network.

[0017] In a possible implementation manner, the method further includes:

[0018] Obtain real-time configuration information uploaded by a client; the real-time configuration information includes a target identifier and configuration parameters;

[0019] According to the configuration parameters, perform real-time adjustment on the virtual machine corresponding to the target identifier.

[0020] In a possible implementation manner, the target network is a high-speed network; the physical network card is connected to the high-speed network.

[0021] In a second aspect, an embodiment of the present application provides a multi-machine task processing device, including:

[0022] A determination module, configured to obtain a target task request, and determine a plurality of target virtual machines corresponding to the target task request from a plurality of virtual machines corresponding to at least two hosts; the plurality of target virtual machines are located in at least two hosts; a physical network card and physical GPU resources are deployed in each host; a virtual network card and virtual GPU resources are included in the virtual machine; at least one of the virtual network cards corresponds to the same physical network card, and at least one of the virtual GPU resources corresponds to the same physical GPU resource; different virtual machines and different hosts communicate with each other through a target network;

[0023] An execution module, configured to execute the target task request through a plurality of target virtual machines, and output a target result corresponding to the target task request.

[0024] In a possible implementation manner, the determination module is specifically configured to:

[0025] Collect status information of virtual machines in each host;

[0026] Determine a plurality of target virtual machines corresponding to the target task request according to a target quality of service label included in the target task request and the status information; the target quality of service label includes at least one of the number of GPUs corresponding to the target task request, priority information, task processing delay, number of model parameters, and task processing accuracy.

[0027] In a possible implementation manner, the apparatus is further configured to:

[0028] For each host, virtualize the physical network card in the host to obtain a plurality of virtual network cards corresponding to the physical network card;

[0029] Configure the virtual network card in each virtual machine corresponding to the host through an input / output memory management unit, so that the virtual machine communicates with the virtual network card through a network card driver.

[0030] In a possible implementation manner, at least one containerized application is included in the virtual machine; the apparatus is further configured to:

[0031] When the status information of virtual machines in each host meets a preset expansion condition, create a new virtual machine in the host according to a pre-configured virtual machine image, and schedule at least one containerized application to the new virtual machine; the preset expansion condition includes that the GPU utilization rate meets a first preset condition, the CPU average load meets a second preset condition, the memory occupancy ratio meets a third preset condition, or the task request change amount meets a fourth preset condition.

[0032] In a possible implementation, the virtual machine is connected to the target cloud service platform through a virtual cloud service network and receives a configuration instruction issued by the target cloud service platform through the virtual cloud service network.

[0033] In a possible implementation, the device is further configured to:

[0034] Obtain real-time configuration information uploaded by a client; the real-time configuration information includes a target identifier and configuration parameters;

[0035] Perform real-time adjustment on the virtual machine corresponding to the target identifier according to the configuration parameters.

[0036] In a possible implementation, the target network is a high-speed network; the physical network card is connected to the high-speed network.

[0037] In a third aspect, an embodiment of the present application provides a multi-machine task processing device, including: a memory and a processor;

[0038] The memory stores computer execution instructions;

[0039] The processor executes the computer execution instructions stored in the memory, so that the processor executes the multi-machine task processing method according to any one of the first aspects.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored, and when the computer execution instructions are executed by a processor, they are used to implement the multi-machine task processing method according to any one of the first aspects.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the multi-machine task processing method according to any one of the first aspects.

[0042] In a sixth aspect, an embodiment of the present application provides a multi-machine task processing system, including a server and at least two hosts, and each host includes a plurality of virtual machines;

[0043] The server is configured to obtain a target task request, and determine a plurality of target virtual machines corresponding to the target task request from a plurality of virtual machines corresponding to at least two hosts; the plurality of target virtual machines are located in at least two hosts; a physical network card and physical GPU resources are deployed in each host; the virtual machine includes a virtual network card and virtual GPU resources; at least one of the virtual network cards corresponds to the same physical network card, and at least one of the virtual GPU resources corresponds to the same physical GPU resource; different virtual machines and different hosts communicate through a target network;

[0044] The server executes the target task request through the multiple target virtual machines and outputs a target result corresponding to the target task request.

[0045] In the embodiment of the present application, a target task request is obtained, and multiple target virtual machines corresponding to the target task request are determined from multiple virtual machines corresponding to at least two hosts; the multiple target virtual machines are located in at least two hosts; a physical network card and physical GPU resources are deployed in each host; a virtual network card and virtual GPU resources are included in the virtual machines; at least one virtual network card corresponds to the same physical network card, and at least one virtual GPU resource corresponds to the same physical GPU resource; different virtual machines and different hosts communicate through a target network; the target task request is executed through the multiple target virtual machines, and a target result corresponding to the target task request is output. In the present application, a physical network card and physical GPU resources are deployed in each host, and the virtual machines can directly access the physical network card and physical GPU resources through the virtual network card and virtual GPU resources. Different virtual machines and different hosts communicate through the target network. When a target task request is received, target virtual machines can be determined from the virtual machines of multiple hosts to execute the target task request, and finally a target result corresponding to the target task request is output. In this way, the electronic device can execute the target task request based on the computing resources in multiple hosts, which can improve the efficiency of task processing, and at the same time can also improve the real-time performance and response speed of task processing. Description of the Drawings

[0046] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0047] Figure 1 It is a schematic flowchart of a multi-machine task processing method provided for an exemplary embodiment of the present application;

[0048] Figure 2 It is a schematic flowchart of another multi-machine task processing method provided for an exemplary embodiment of the present application;

[0049] Figure 3 It is a schematic diagram of a system architecture of multi-machine task processing provided for an exemplary embodiment of the present application;

[0050] Figure 4 It is a schematic diagram of the structure of a multi-machine task processing device provided for an exemplary embodiment of the present application;

[0051] Figure 5 It is a schematic diagram of the structure of a multi-machine task processing device provided for an exemplary embodiment of the present application.

[0052] Through the above-mentioned accompanying drawings, specific embodiments of the present application have been shown, and there will be a more detailed description hereinafter. These drawings and written descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed Description of Specific Embodiments

[0053] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0054] The following first explains the terms involved in the present application:

[0055] Elastic GPU Service (EGS): Used to provide Graphics Processing Unit (GPU) acceleration computing capabilities, enabling the on-demand use and elastic scaling of GPU computing resources.

[0056] Pass through: A computer hardware virtualization technology that allows virtual machines to directly access underlying physical hardware devices, such as GPUs and network interfaces.

[0057] Host (or physical machine, host machine) (Host): A physical server used to run a virtual machine manager.

[0058] Virtual Machine (VM): A virtual computer system running on a host machine.

[0059] Kubernetes (K8S): A container orchestration platform used for automating the deployment, scaling, and management of application programs.

[0060] Container Service for Kubernetes (ACK): A K8s container service that can implement container application management and the life cycle management of containerized applications.

[0061] Inference: The process of using a trained model for prediction or classification in fields such as machine learning, deep learning, and artificial intelligence.

[0062] Single Root I / O Virtualization (SR-IOV): Allows a single physical network device (such as a physical Network Interface Card (NIC), network interface card, etc.) to be virtualized at the hardware level, splitting out multiple virtual (function) network cards (Virtual Functions, VF).

[0063] With the rapid development of Internet technology and AI technology, the scale and complexity of various models are constantly increasing, and the demand for computing resources is also growing. For example, in the field of autonomous driving, the latency requirements for model inference are relatively high. In related technologies, the processing of tasks such as model inference is usually based on the computing resources of a single machine. This task processing method has low processing efficiency, and the real-time performance and response speed are not high.

[0064] Based on the fact that the single-machine resources cannot meet the real-time, high-concurrency, and high-precision requirements of model inference, multi-machine inference (distributed inference) has become an important direction. Multi-machine inference distributes computing tasks to multiple servers, achieving efficient utilization of computing resources and significant improvement in performance. It can not only shorten the inference time but also support the simultaneous access of a large number of users, ensuring the stability and reliability of the system. In addition, multi-machine inference also has significant advantages in energy efficiency and scalability, and can flexibly meet the requirements of different application scenarios.

[0065] In the cloud service scenario, due to the lack of an interconnected network among virtual machines in different hosts, in related technologies, task processing in the virtualized cloud service scenario is usually based on a single machine and cannot meet the requirements of multi-machine inference. This single-machine processing method in related technologies limits the performance and efficiency in processing large models and high-concurrency requests. In application scenarios with high requirements for real-time performance and accuracy such as autonomous driving, single-machine inference is difficult to provide sufficient computing power and response speed. In addition, there are solutions for multi-machine inference based on physical machines (bare metal) in related technologies, but there are the following disadvantages in single-machine inference based on physical machines:

[0066] The physical machine cannot access the cloud service network, and various functions of the cloud service cannot be realized; the scalability and flexibility of the physical machine are relatively low, and manual configuration and hardware deployment are required, and the rapid expansion and contraction of resources in the virtual machine scenario cannot be realized; there are differences in hardware resource configuration in the physical machine, and the operation and maintenance cost is relatively high, and the standardized interface and unified resource management in the virtual machine scenario cannot be realized; the startup and configuration process of the physical machine is relatively slow, the deployment complexity is relatively high, and it affects the rapid response ability of the system, and it cannot be applied to the dynamic environment that requires rapid adjustment of resources; the resource isolation of the physical machine is limited, and there is a risk of resource sharing, which affects the security of user data and service instructions; the hardware configuration of the physical machine cannot be adjusted in real time, and the adaptability to changing requirements is relatively low.

[0067] To solve the above problems, the present application provides a multi-machine task processing method, device, equipment, storage medium and program product, which obtains a target task request, and determines a plurality of target virtual machines corresponding to the target task request from a plurality of virtual machines corresponding to at least two hosts; the plurality of target virtual machines are located in at least two hosts; a physical network card and physical GPU resources are deployed in each host; a virtual network card and virtual GPU resources are included in the virtual machine; the plurality of virtual network cards correspond to the same physical network card, and the plurality of virtual GPU resources correspond to the same physical GPU resource; different virtual machines and different hosts communicate through a target network; the target task request is executed by the plurality of target virtual machines, and a target result corresponding to the target task request is output. In the present application, a physical network card and physical GPU resources are deployed in each host, and the virtual machine can directly access the physical network card and physical GPU resources through the virtual network card and virtual GPU resources. Different virtual machines and different hosts communicate through the target network. When the target task request is received, the target virtual machine can be determined from the virtual machines of multiple hosts to execute the target task request, and finally the target result corresponding to the target task request is output. In this way, the electronic device can execute the target task request based on the computing resources in multiple hosts, which can improve the efficiency of task processing, and at the same time can also improve the real-time performance and response speed of task processing.

[0068] The technical solution shown in the present application will be described in detail below through specific embodiments. It should be noted that the following several embodiments can exist independently or be combined with each other. For the same or similar content, it will not be repeated in different embodiments.

[0069] Figure 1 It is a flowchart of a multi-machine task processing method provided for an exemplary embodiment of the present application. Please refer to Figure 1 and the multi-machine task processing method may include:

[0070] S101. Obtain a target task request, and determine multiple target virtual machines corresponding to the target task request from multiple virtual machines corresponding to at least two hosts; the multiple target virtual machines are located in at least two hosts; each host is deployed with a physical network card and physical GPU resources; the virtual machines include virtual network cards and virtual GPU resources; at least one virtual network card corresponds to the same physical network card, and at least one virtual GPU resource corresponds to the same physical GPU resource; different virtual machines and different hosts communicate through a target network.

[0071] The execution subject of the embodiments of the present application can be an electronic device or a multi-machine task processing device provided in the electronic device. The multi-machine task processing device can be implemented by software or by a combination of software and hardware. For the convenience of understanding, hereinafter, the execution subject is taken as an electronic device as an example for description. The electronic device can specifically refer to a cloud platform, a cloud server, a server or a host, etc., and the embodiments of the present application do not limit this.

[0072] In the embodiments of the present application, the target task request can refer to a processing request for tasks such as model inference submitted by a user, for example, it can refer to task processing requests such as autonomous driving, video analysis, and language dialogue models. The target virtual machine can refer to a virtual machine that processes the target task request in a virtualized cloud service scenario. The number of the target virtual machines can be 2, 3 or more, and the multiple target virtual machines are located in at least two hosts to implement multi-machine task inference.

[0073] Each host is deployed with a physical network card NIC and physical GPU resources, where the physical network card can be connected to the target network, and different hosts communicate through the target network. Each virtual machine is deployed with a virtual network card VF and virtual GPU resources (vGPU). In this way, the virtual machine can communicate with the virtual network card VF through the network card driver to access the physical network card, and at the same time communicate with the virtual GPU resources based on the GPU driver to access the physical GPU resources. In this way, through hardware virtualization, the present application can achieve virtual machine pass-through, that is, the virtual machine accesses the physical network card and physical GPU resources of the host to ensure the performance of virtual machine task processing. At the same time, different virtual machines are connected through the target network, which can ensure data synchronization and data sharing and ensure the accuracy in the multi-machine task processing process.

[0074] In this step, the electronic device can receive the target task request submitted by the user through user interaction interfaces such as Open Application Programming Interface (OpenAPI) or virtual cloud service network interface. The target task request may include input data, target calculation parameters, and target Quality of Service (QoS) tags, etc. After that, the electronic device can collect the status information of the virtual machines in each host in the virtualized cloud service scenario. The status information may include GPU utilization, virtual machine idle status, and network bandwidth, etc. The electronic device can determine the target virtual machine corresponding to the target task request in at least two hosts according to the status information, and realize the dynamic allocation of the computing resources corresponding to the target task request.

[0075] S102. Execute the target task request through multiple target virtual machines and output the target result corresponding to the target task request.

[0076] In the embodiment of the present application, the target result may refer to the task inference result corresponding to the target task request. The inference framework and related libraries required for the model inference task are installed and configured inside the target virtual machine, such as the distributed inference framework corresponding to the execution of the distributed inference task and the distributed communication library, tool library, hardware acceleration library, etc. The distributed inference framework can split the inference model into each target virtual machine to realize the distributed computing processing of the input data of the target task request.

[0077] Specifically, after the electronic device determines the target virtual machine corresponding to the target task request, it can execute the target task request through multiple target virtual machines. Specifically, it can parallelly distribute the input data in the target task request to each target virtual machine through the target network; in each target virtual machine, calculate and process the input data based on the target task request to obtain the intermediate calculation result. After that, each target virtual machine performs data synchronization and sharing through the target network. The electronic device can perform aggregation processing on the intermediate calculation results of each target virtual machine to obtain the target result corresponding to the target task request, and output the target result. For example, it can return the target result corresponding to the target task request to the user in the form of a Hypertext Transfer Protocol (HTTP) page response. Of course, it can also return the target result in the form of notification messages such as emails and text messages, or directly write the target result corresponding to the target task request into the target storage path configured by the user. The embodiment of the present application does not limit the output method of the target result.

[0078] In the embodiment of the present application, the electronic device obtains a target task request, and determines a plurality of target virtual machines corresponding to the target task request from a plurality of virtual machines corresponding to at least two hosts; the plurality of target virtual machines are located in at least two hosts; a physical network card and physical GPU resources are deployed in each host; the virtual machine includes a virtual network card and virtual GPU resources; at least one virtual network card corresponds to the same physical network card, and at least one virtual GPU resource corresponds to the same physical GPU resource; different virtual machines and different hosts communicate through a target network; the target task request is executed by the plurality of target virtual machines, and a target result corresponding to the target task request is output. In the present application, a physical network card and physical GPU resources are deployed in each host, and the virtual machine can directly access the physical network card and physical GPU resources through the virtual network card and virtual GPU resources. Different virtual machines and different hosts communicate through the target network. When a target task request is received, target virtual machines can be determined from the virtual machines of multiple hosts to execute the target task request, and finally a target result corresponding to the target task request is output. In this way, the electronic device can execute the target task request based on the computing resources in multiple hosts, which can improve the efficiency of task processing, and at the same time can also improve the real-time performance and response speed of task processing.

[0079] Based on the above embodiment, Figure 2 FIG. is a schematic flowchart of another multi-machine task processing method provided by an exemplary embodiment of the present application. Please refer to Figure 2 The multi-machine task processing method may include:

[0080] S201. For each host, virtualize the physical network card in the host to obtain a plurality of virtual network cards corresponding to the physical network card; configure the virtual network card in each virtual machine corresponding to the host through an input-output memory management unit, so that the virtual machine communicates with the virtual network card through a network card driver.

[0081] In the embodiment of the present application, the input-output memory management unit (IOMMU) may refer to a memory management unit that connects a direct memory access bus (DMA) and a main memory in a virtualized cloud service scenario, and may be used to implement functions such as virtual address mapping, device isolation, and access control.

[0082] Specifically, in each host, the electronic device can virtualize the physical network cards in the host. For example, the SR-IOV technology can be used to virtualize the same physical network card into multiple virtual network cards. Then, based on the component support of the input-output memory management unit (IOMMU), virtual network cards can be configured in each virtual machine of the host, enabling virtual machine passthrough. The virtual machine can directly access the physical network card, allowing the virtual machine to make the most of the hardware performance and obtain computing power similar to that of a physical machine. By performing hardware virtualization on the physical network card, the electronic device enables multiple virtual machines in the same host to efficiently share the same physical network card, that is, the same physical network card corresponding to multiple virtual network cards, which can greatly improve network performance and bandwidth utilization. In addition, each virtual network card is distributed and deployed in a different virtual machine. In the virtualized cloud service scenario, the resource isolation ability of the virtual machine is high, enabling data security isolation between different virtual machines and ensuring network performance and data security in a multi-user environment.

[0083] S202. Obtain a target task request and collect the status information of the virtual machines in each host.

[0084] S203. Determine multiple target virtual machines corresponding to the target task request from multiple virtual machines corresponding to at least two hosts according to the target quality of service label included in the target task request and the status information; the target quality of service label includes at least one of the number of GPUs corresponding to the target task request, priority information, task processing latency, the number of model parameters, and task processing accuracy.

[0085] In the embodiments of the present application, multiple target virtual machines are located in at least two hosts; a physical network card and physical GPU resources are deployed inside each host; a virtual network card and virtual GPU resources are included in the virtual machine; multiple virtual network cards correspond to the same physical network card, and multiple virtual GPU resources correspond to the same physical GPU resource; different virtual machines and different hosts communicate through a target network. The status information may refer to the actual running status information of the virtual machines in multiple hosts, specifically including GPU resource utilization rate, the running status or idle status of each virtual machine, network bandwidth, and communication latency, etc. The target quality of service label may refer to the label information for the quality of service in the target task request, which may include at least one of the number of GPUs corresponding to the target task request, priority information, task processing latency, the number of model parameters, and task processing accuracy. Of course, other label information may also be included, and the embodiments of the present application do not limit this.

[0086] Specifically, after the electronic device obtains the target task request submitted by the client, it can collect the status information of the virtual machines in each host. The electronic device can determine multiple alternative virtual machines that meet the target service quality label based on the status information and in combination with the target service quality label in the target task request. For example, it can select a virtual machine with the number of GPUs meeting the number of GPUs required by the target task request and in an idle state as an alternative virtual machine. The electronic device can select multiple target virtual machines in at least two hosts from the multiple alternative virtual machines to process the target task request. Specifically, it can select the target virtual machines from the alternative virtual machines based on a random selection method, or use other resource selection algorithms to determine the target virtual machines corresponding to the target task request, which can be specifically configured according to actual needs, and the embodiments of the present application do not limit this.

[0087] In a possible implementation manner, the target network is a high-speed network; the physical network card is connected to the high-speed network.

[0088] In the embodiments of the present application, different hosts and different virtual machines are communicatively connected through the target network, and the target network is a high-speed network. Here, high-speed is a defined concept with specific determination criteria. For example, a high-speed network refers to a switched bandwidth network greater than 25 gigabits per second (Gbps), or a network with a communication delay less than 100 microseconds, etc. Of course, the high-speed network can also be determined using other criteria, and the embodiments of the present application do not limit this.

[0089] In the embodiments of the present application, the physical network card is connected to the high-speed network, and different hosts and different virtual machines are communicatively connected through the high-speed network, which can achieve high-bandwidth and low-latency network communication, ensure efficient data exchange and communication during the multi-machine task processing, ensure the processing efficiency of the inference task, and further improve the real-time performance and response speed of the task processing.

[0090] S204. In each target virtual machine, execute the target task request to obtain the intermediate calculation results of each target virtual machine; obtain the intermediate calculation results of each target virtual machine through the target network; perform an aggregation process on the multiple intermediate calculation results to generate and output the target result corresponding to the target task request.

[0091] In the embodiments of the present application, after the electronic device determines multiple target virtual machines corresponding to the target task request, the target task request can be executed by the multiple target virtual machines. Specifically, each target virtual machine can execute the target task request to obtain intermediate calculation results. The multiple target virtual machines perform data exchange and synchronization through a high-speed network. The electronic device can obtain the intermediate calculation results of each target virtual machine through the high-speed network, and then aggregate and process the multiple intermediate calculation results to generate the target result corresponding to the target task request, and finally output the target result corresponding to the target task request through the client. In this way, the electronic device can process the target task request based on the computing resources in multiple hosts, avoiding the problem of insufficient single-machine computing resources, improving the processing efficiency of the model inference task, and also improving the real-time performance and response speed of task processing.

[0092] S205. When the status information of the virtual machines in each host meets the preset expansion condition, create a new virtual machine in the host according to the pre-configured virtual machine image, and schedule at least one containerized application to the new virtual machine; the preset expansion condition includes that the GPU utilization rate meets the first preset condition, the CPU average load meets the second preset condition, the memory occupancy ratio meets the third preset condition, or the task request change amount meets the fourth preset condition.

[0093] In the embodiments of the present application, the containerized architecture (such as ACK, K8s, etc.) is adopted inside the virtual machine for application deployment and management. Each virtual machine includes at least one containerized application, which can improve resource utilization and management convenience. The preset expansion condition can refer to the preset virtual machine expansion trigger condition, and the preset expansion condition can include that the GPU utilization rate meets the first preset condition, the CPU average load meets the second preset condition, the memory occupancy ratio meets the third preset condition, or the task request change amount meets the fourth preset condition; among them, the GPU utilization rate meeting the first preset condition can mean that the duration for which the GPU utilization rate is greater than the preset threshold is greater than the preset duration threshold; the CPU average load meeting the second preset condition can mean that the CPU average load is greater than the preset load threshold; the memory occupancy ratio meeting the third preset condition can mean that the ratio of the available memory to the total memory is less than the preset memory ratio threshold; the task request change amount meeting the fourth preset condition can mean that the increase in the number of task requests is greater than the preset number threshold or the proportion of the increase in the number of task requests is greater than the preset number ratio, etc. Of course, the preset expansion condition can also include other conditions, which can be specifically set based on actual needs, and the embodiments of the present application do not limit this. It should be noted that the different preset expansion conditions are in an "or" relationship, that is, if one of them is met, the electronic device can expand the virtual machine to meet the task processing requirements.

[0094] A virtual machine image may refer to a pre-set virtual machine image file, which may include system files (such as kernels, drivers, etc.), computing components, and container runtimes, etc., and can enable the rapid creation of virtual machine instances. Specifically, an electronic device can collect the status information of virtual machines in multiple hosts through a monitoring and operation and maintenance tool. When the status information meets the preset expansion condition, the electronic device can quickly create a new virtual machine through a pre-configured virtual machine image, and quickly deploy or schedule containerized applications in the new virtual machine to achieve the rapid expansion of the system, while simplifying the deployment process of applications in the virtual machine.

[0095] In a possible implementation manner, the virtual machine is connected to the target cloud service platform through a virtual cloud service network and receives a configuration instruction issued by the target cloud service platform through the virtual cloud service network.

[0096] In the embodiments of the present application, the virtual machine can be connected to the cloud service platform through a virtual cloud service network (Virtual Private Cloud, VPC), and can receive a configuration instruction issued by the target cloud service platform through the virtual cloud service network, so as to implement various configuration functions in the cloud service. In this way, in the virtualized cloud service scenario, the virtual machines in each host can integrate the VPC network, enabling users to make full use of functions such as load balancing, data storage, and security services in the cloud service, with more comprehensive functions, higher scalability, and the ability to quickly start, stop, or migrate virtual machines through the cloud service. In addition, elastic scaling of virtual machines can be achieved based on the cloud service, and dynamic resource requirements can be efficiently processed.

[0097] S206. Obtain the real-time configuration information uploaded by the client; the real-time configuration information includes a target identifier and configuration parameters; according to the configuration parameters, perform real-time adjustment on the virtual machine corresponding to the target identifier.

[0098] In the embodiments of the present application, the real-time configuration information may refer to the specific configuration information of the virtual machine based on the actual needs of the user, which may include a target identifier and configuration parameters. The target identifier may refer to the identifier of the virtual machine that needs to be configured in real time, and the configuration parameters may refer to the parameters of the virtual machine that need to be configured. The electronic device can collect the real-time configuration information uploaded by the client through the VPC or an open application programming interface, and then can perform real-time configuration adjustment on the virtual machine corresponding to the target identifier in the real-time configuration information, so that the parameters of the virtual machine are adjusted to the configuration parameters in the real-time configuration information, or the electronic device can replace the virtual machine image corresponding to the virtual machine based on the real-time configuration information. In this way, the electronic device can perform dynamic configuration on the virtual machine in response to the real-time adjustment requirements of the client in the virtualized cloud service environment, meet the actual needs of users, and have higher applicability in actual application scenarios.

[0099] It should be understood that in various embodiments of the present application, the magnitudes of the sequence numbers of the various processes do not imply the order of execution, and the order of execution of the various processes should be determined according to their functions and internal logics, and should not constitute any limitation to the implementation processes of the embodiments of the present application.

[0100] Based on any one of the above embodiments, Figure 3 FIG. is a schematic diagram of a system architecture for multi-machine task processing provided by an exemplary embodiment of the present application. As Figure 3 shown, the system structure of multi-machine task processing includes a task layer, a service layer, a virtual machine layer, a hardware layer, and a target network. Among them, the task layer may refer to specific model inference tasks, corresponding to different target task requests, and may specifically include autonomous driving, video analysis, and language dialogue, etc. The service layer may correspond to different functional modules of cloud services, and may specifically include an Application Programming Interface (API) gateway, load balancing, security management, and monitoring and operation and maintenance, etc.

[0101] The virtual machine layer may refer to the virtual machines in each host. Each virtual machine includes a virtual network card and virtual GPU resources, and at the same time integrates a VPC network interface. The virtual machine is connected to the virtual network card through a network card driver, and the virtual network card is the hardware virtualization function of the physical network card in the hardware layer; the virtual machine is connected to the virtual GPU resources through a GPU driver, and the virtual GPU resources are the hardware virtualization function of the physical GPU resources in the hardware layer. The VPC network interface is used to connect to the target cloud service platform through the virtual cloud service network to implement various functions of the cloud service platform. The hardware layer refers to multiple hosts, and each host includes a physical network card and physical GPU resources. Different virtual machines and different hosts are connected through the target network, enabling efficient data exchange and communication.

[0102] In the embodiments of the present application, the electronic device is efficiently interconnected through virtual machine passthrough and the target network, enabling collaborative computing and efficient data transmission of the target virtual machines in multiple hosts, improving the overall inference performance of the system, realizing fast calculation of inference tasks, and meeting the actual needs of model inference. Moreover, the virtual machine can provide unified standardized interfaces and management tools, facilitating monitoring, operation and maintenance, and resource configuration, reducing the complexity of operation and maintenance, and enhancing the manageability and maintainability of the system. In addition, the virtual machine can communicate and connect with the cloud service through the VPC, improving the system performance, simplifying the user usage process, enhancing the system functions and user experience. The multi-machine task processing method in the present application, while ensuring high-performance computing capabilities, provides high resource flexibility and dynamic scheduling capabilities through virtualization and container technologies, and can also achieve resource isolation based on virtual machines to ensure data security.

[0103] Figure 4The following is a schematic structural diagram of a multi-machine task processing device provided by an exemplary embodiment of the present application. Please refer to Figure 4 , the multi-machine task processing device 40 includes:

[0104] A determination module 41, configured to obtain a target task request, and determine a plurality of target virtual machines corresponding to the target task request from a plurality of virtual machines corresponding to at least two hosts; the plurality of target virtual machines are located in at least two hosts; a physical network card and physical GPU resources are deployed in each host; the virtual machine includes a virtual network card and virtual GPU resources; at least one virtual network card corresponds to the same physical network card, and at least one virtual GPU resource corresponds to the same physical GPU resource; different virtual machines and different hosts communicate through a target network.

[0105] An execution module 42, configured to execute the target task request through the plurality of target virtual machines and output a target result corresponding to the target task request.

[0106] In a possible implementation manner, the determination module 41 is specifically configured to:

[0107] Collect the status information of the virtual machines in each host;

[0108] Determine a plurality of target virtual machines corresponding to the target task request according to the target quality of service label included in the target task request and the status information; the target quality of service label includes at least one of the number of GPUs, priority information, task processing latency, the number of model parameters, and task processing accuracy corresponding to the target task request.

[0109] In a possible implementation manner, the device 40 is further configured to:

[0110] For each host, virtualize the physical network card in the host to obtain a plurality of virtual network cards corresponding to the physical network card;

[0111] Configure the virtual network card in each virtual machine corresponding to the host through an input / output memory management unit, so that the virtual machine communicates with the virtual network card through a network card driver.

[0112] In a possible implementation manner, the virtual machine includes at least one containerized application; the device 40 is further configured to:

[0113] When the status information of the virtual machines in each host meets a preset expansion condition, create a new virtual machine in the host according to a pre-configured virtual machine image, and schedule at least one containerized application to the new virtual machine; the preset expansion condition includes that the GPU utilization rate meets a first preset condition, the CPU average load meets a second preset condition, the memory occupancy ratio meets a third preset condition, or the task request change amount meets a fourth preset condition.

[0114] In a possible implementation, the virtual machine is connected to the target cloud service platform through the virtual cloud service network and receives the configuration instructions issued by the target cloud service platform through the virtual cloud service network.

[0115] In a possible implementation, the apparatus 40 is further configured to:

[0116] Obtain the real-time configuration information uploaded by the client; the real-time configuration information includes a target identifier and configuration parameters;

[0117] According to the configuration parameters, perform real-time adjustment on the virtual machine corresponding to the target identifier.

[0118] In a possible implementation, the target network is a high-speed network; the physical network card is connected to the high-speed network.

[0119] The multi-machine task processing apparatus 40 provided by the embodiments of the present application can execute the technical solutions shown in the above method embodiments, and the implementation principles and beneficial effects are similar, and will not be elaborated here.

[0120] Figure 5 The following is a schematic structural diagram of a multi-machine task processing device provided by an exemplary embodiment of the present application. Please refer to Figure 5 , the multi-machine task processing device 50 may include a processor 51 and a memory 52. Exemplarily, the processor 51 and the memory 52 are interconnected with each other through a bus 53.

[0121] The memory 52 stores computer execution instructions;

[0122] The processor 51 executes the computer execution instructions stored in the memory 52, so that the processor 51 executes the multi-machine task processing method as shown in the above method embodiments.

[0123] Correspondingly, the embodiments of the present application provide a computer-readable storage medium, and computer execution instructions are stored in the computer-readable storage medium. When the computer execution instructions are executed by a processor, they are used to implement the multi-machine task processing method of the above method embodiments.

[0124] Correspondingly, the embodiments of the present application may further provide a computer program product, including a computer program. When the computer program is executed by a processor, the multi-machine task processing method shown in the above method embodiments can be implemented.

[0125] The embodiments of the present application further provide a multi-machine task processing system, including a server and at least two hosts, and each host includes a plurality of virtual machines;

[0126] The server is used to obtain a target task request, and determine multiple target virtual machines corresponding to the target task request from multiple virtual machines corresponding to at least two hosts; the multiple target virtual machines are located in at least two hosts; a physical network card and physical GPU resources are deployed in each host; the virtual machines include virtual network cards and virtual GPU resources; at least one virtual network card corresponds to the same physical network card, and at least one virtual GPU resource corresponds to the same physical GPU resource; different virtual machines and different hosts communicate through a target network.

[0127] The server executes the target task request through the multiple target virtual machines and outputs a target result corresponding to the target task request.

[0128] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, compact disc read-only memory (CD-ROM), optical memory, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0130] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device realizes the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 steps of the functions specified in one block or multiple blocks.

[0132] In a typical configuration, a computing device includes one or more processors, an input / output interface, a network interface, and memory.

[0133] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0134] Computer-readable media includes both permanent and non-permanent, removable and non-removable media implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0135] It should also be noted that the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising the said element.

[0136] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A multi-machine task processing method, characterized in that: include: Obtain a target task request, and determine a plurality of target virtual machines corresponding to the target task request from a plurality of virtual machines corresponding to at least two hosts; the plurality of target virtual machines are located in at least two hosts; a physical network card and a physical GPU resource are deployed in each host; the virtual machines include a virtual network card and a virtual GPU resource; at least one of the virtual network cards corresponds to the same physical network card, and at least one of the virtual GPU resources corresponds to the same physical GPU resource; different virtual machines and different hosts communicate through a target network; The target task request is executed through multiple target virtual machines, and a target result corresponding to the target task request is output.

2. The method according to claim 1, characterized in that The determining a plurality of target virtual machines corresponding to the target task request comprises: Collect status information of virtual machines in each host; According to the target service quality label and the status information included in the target task request, multiple target virtual machines corresponding to the target task request are determined; the target service quality label includes at least one of the number of GPUs, priority information, task processing delay, model parameter amount and task processing accuracy corresponding to the target task request.

3. The method according to claim 1, characterized in that The method further comprises: For each host, virtualize the physical network card in the host to obtain multiple virtual network cards corresponding to the physical network card; The virtual network card is configured in each virtual machine corresponding to the host through an input / output memory management unit, so that the virtual machine communicates with the virtual network card through a network card driver.

4. The method according to claim 1, characterized in that: The virtual machine includes at least one containerized application; the method further includes: When the status information of the virtual machines in each host meets the preset expansion conditions, a new virtual machine is created in the host according to the preconfigured virtual machine image, and at least one containerized application is scheduled to the new virtual machine; the preset expansion conditions include that the GPU utilization rate meets the first preset condition, the CPU average load meets the second preset condition, the memory occupancy ratio meets the third preset condition, or the task request change meets the fourth preset condition.

5. The method according to claim 1, characterized in that The virtual machine is connected to the target cloud service platform through a virtual cloud service network, and receives configuration instructions issued by the target cloud service platform through the virtual cloud service network.

6. The method according to claim 1, characterized in that The method further comprises: Acquire real-time configuration information uploaded by the client; the real-time configuration information includes a target identifier and configuration parameters; According to the configuration parameters, the virtual machine corresponding to the target identifier is adjusted in real time.

7. The method according to claim 1, characterized in that The executing the target task request by multiple target virtual machines and outputting the target result corresponding to the target task request includes: In each of the target virtual machines, executing the target task request to obtain an intermediate calculation result of each target virtual machine; Obtaining intermediate calculation results of each of the target virtual machines through the target network; Aggregate the plurality of intermediate calculation results to generate and output a target result corresponding to the target task request.

8. The method according to any one of claims 1 to 7, characterized in that: The target network is a high-speed network; the physical network card is connected to the high-speed network.

9. A multi-machine task processing device, characterized in that: include: A determination module is used to obtain a target task request, and determine a plurality of target virtual machines corresponding to the target task request from a plurality of virtual machines corresponding to at least two hosts; the plurality of target virtual machines are located in at least two hosts; a physical network card and a physical GPU resource are deployed in each host; the virtual machines include a virtual network card and a virtual GPU resource; at least one of the virtual network cards corresponds to the same physical network card, and at least one of the virtual GPU resources corresponds to the same physical GPU resource; different virtual machines and different hosts communicate through a target network; The execution module is used to execute the target task request through multiple target virtual machines and output the target result corresponding to the target task request.

10. A multi-machine task processing device, characterized in that: include: Memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the multi-machine task processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the multi-machine task processing method according to any one of claims 1 to 8.

12. A computer program product, characterized in that It includes a computer program, which, when executed by a computer, implements the multi-machine task processing method as described in any one of claims 1 to 8.

13. A multi-machine task processing system, characterized in that: It includes a server and at least two hosts, each of which includes multiple virtual machines; The server is used to obtain a target task request, and determine a plurality of target virtual machines corresponding to the target task request from a plurality of virtual machines corresponding to at least two hosts; the plurality of target virtual machines are located in at least two hosts; a physical network card and a physical GPU resource are deployed in each host; the virtual machine includes a virtual network card and a virtual GPU resource; at least one of the virtual network cards corresponds to the same physical network card, and at least one of the virtual GPU resources corresponds to the same physical GPU resource; different virtual machines and different hosts communicate through a target network; The server executes the target task request through the multiple target virtual machines, and outputs a target result corresponding to the target task request.

Images