Resource adjustment method and device, equipment and storage medium

Abstract

The invention provides a resource adjustment method and apparatus, a device and a storage medium. The method comprises the steps of determining a computing resource usage amount of each reasoning service node; and adjusting the computing resources according to the computing resource usage amount of each reasoning service node. The reasoning service performance and the computing resource utilization rate can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to a resource adjustment method, apparatus, device, and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, the application of reasoning services in various industries has become more and more common. Among them, reasoning service is a service method that uses artificial intelligence model reasoning methods to perform reasoning calculations to achieve rapid processing of data resources. However, the computing resources of most current reasoning services are to allocate a specific number of computing resources to the reasoning service at the time of deployment, that is, the computing resources of the reasoning service are fixed. When an emergency occurs, it will be impossible to cope with it, resulting in poor performance of the reasoning service and low utilization of computing resources. Summary of the invention

[0003] The embodiments of the present application provide a resource adjustment method, apparatus, device and storage medium, which can improve reasoning service performance and computing resource utilization.

[0004] In a first aspect, an embodiment of the present application provides a resource adjustment method, including:

[0005] Determine the computing resource usage of each inference service node;

[0006] The computing resources are adjusted according to the computing resource usage of each of the inference service nodes.

[0007] In a second aspect, an embodiment of the present application provides a resource adjustment device, including:

[0008] A resource determination module, used to determine the computing resource usage of each inference service node;

[0009] The resource adjustment module is used to adjust the computing resources according to the computing resource usage of each of the inference service nodes.

[0010] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0011] A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the resource adjustment method as described in the first aspect embodiment and various implementation methods mentioned above.

[0012] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the resource adjustment method as described in the embodiment of the first aspect and various implementation methods.

[0013] In a fifth aspect, an embodiment of the present application provides a computer program product comprising program instructions. When the program instructions are executed on an electronic device, the electronic device executes the resource adjustment method as described in the embodiment of the first aspect and various implementation methods.

[0014] The technical solution disclosed in the embodiment of the present application determines the computing resource usage of each inference service node, and then dynamically adjusts the computing resources according to the computing resource usage of each inference service node, thereby improving the inference service performance and computing resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of a resource adjustment method provided in an embodiment of the present application;

[0017] Figure 2 A flowchart for adjusting computing resources provided in an embodiment of the present application;

[0018] Figure 3 A flowchart of another resource adjustment method provided in an embodiment of the present application;

[0019] Figure 4 A schematic block diagram of a resource adjustment device provided in an embodiment of the present application;

[0020] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0024] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two, that is, at least two. "At least one" means one or more than one, that is, one or more. "Any" means any one or any several.

[0025] As mentioned earlier, most current inference service computing resources allocate a specific amount of computing resources to the inference service during deployment. This can cause problems when emergencies occur, such as when the workload of the inference service changes dynamically, resulting in poor performance of the inference service and low utilization of computing resources.

[0026] In response to the above technical problems, the present application provides a resource adjustment method, apparatus, device and storage medium to solve the problem of poor performance of reasoning services and low computing resource utilization when the computing resources allocated to the reasoning service are fixed values.

[0027] The technical solutions provided by the embodiments of the present application are described in detail below through some embodiments. The embodiments described below can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0028] Figure 1A flow chart of a resource adjustment method provided for an embodiment of the present application. The resource adjustment method of an embodiment of the present application can be performed by a resource adjustment device, which can be composed of hardware and / or software and can be integrated in an electronic device. The electronic device can be selected as any hardware device with a data processing function, such as a server, a virtual machine, a computer, and other types of devices, etc., and the type of electronic device is not specifically limited here. In addition, a cluster for deploying and scheduling reasoning services is deployed in the above-mentioned electronic device, such as a Kubernetes cluster (K8S cluster), etc., then the resource adjustment method provided in the present application can be applied to the above-mentioned cluster for deploying and scheduling reasoning services, that is, the resource adjustment method is executed by the cluster.

[0029] like Figure 1 As shown, the method may include the following steps:

[0030] S101, determining the computing resource usage of each inference service node.

[0031] In this application, an inference service node refers to a machine that is deployed with an inference model, such as an inference server or an inference engine, etc. The inference model can be any trained AI model such as a machine learning model or a deep learning model, and there is no restriction on the inference model.

[0032] Furthermore, each of the above-mentioned inference service nodes may be deployed in a distributed manner, and may also be deployed in other manners, such as stand-alone deployment or cluster deployment, etc., which are not specifically limited here.

[0033] In addition, the computing resources of each inference service node may include a central processing unit (CPU), a graphics processing unit (GPU), memory, network, and the number of inference requests.

[0034] In some optional embodiments, when an inference request sent by a user through a client is received, the computing resource usage of each inference service node in the inference service system can be detected to determine the computing resource usage of each inference service node; alternatively, during the process of any inference service node processing an inference request, the computing resource usage of each inference service node in the inference service system can be detected to determine the computing resource usage of each inference service node; alternatively, in addition to the aforementioned two situations, the computing resource usage of each inference service node in the inference service system can be detected in real time to determine the computing resource usage of each inference service node.

[0035] It should be understood that the above-mentioned reasoning service system may be composed of multiple reasoning service nodes. Of course, in addition to the reasoning service nodes, the reasoning service system may also include other components, such as a reasoning scheduler.

[0036] Considering that computing resources may include central processing unit (CPU), graphics processing unit (GPU), memory, network, and number of inference requests, the computing resource usage of each inference service node may include CPU occupancy, GPU occupancy, memory occupancy, network occupancy, and inference request queue length, etc.

[0037] S102: Adjust computing resources according to the computing resource usage of each inference service node.

[0038] In some optional embodiments, it is possible to determine whether the computing resource usage of each inference service node is too high or too low based on the computing resource usage of each inference service node. When it is determined that the computing resource usage of any inference service node is too high or too low, it means that the current computing resource allocation will cause a waste or insufficient computing resource, affecting the performance of the inference service, then the computing resource dynamic adjustment mechanism can be triggered to dynamically adjust the computing resources of the inference service system to increase the usage of computing resources and improve the performance of the inference service.

[0039] Optionally, the above-mentioned dynamic adjustment mechanism of computing resources can be achieved by increasing or decreasing the number of inference service nodes, or adjusting the computing power of the inference service nodes. That is, when it is determined that the computing resource usage of any inference service node is too high, the usage of computing resources can be increased by increasing the number of inference service nodes or improving the computing power of the inference service nodes, thereby improving the performance of the inference service. Correspondingly, when it is determined that the computing resource usage of any inference service node is too low, the usage of computing resources can be increased by reducing the number of inference service nodes or reducing the computing power of the inference service nodes, thereby improving the performance of the inference service.

[0040] The technical solution disclosed in the embodiment of the present application determines the computing resource usage of each inference service node, and then dynamically adjusts the computing resources according to the computing resource usage of each inference service node, thereby improving the inference service performance and computing resource utilization.

[0041] Combine the following Figure 2 , further explaining step S102 in the above embodiment. Figure 2 As shown, step S102 may include the following steps: S102-1 to S102-3:

[0042] S102-1, determine whether the computing resource usage of each inference service node is greater than the resource usage threshold, if so, execute S102-2, otherwise execute S102-3.

[0043] S102-2: If the computing resource usage of any inference service node is greater than the resource usage threshold, the computing resources are adjusted according to a first adjustment method.

[0044] S102-3: If the computing resource usage of any inference service node is less than the resource usage threshold, the computing resources are adjusted according to the second adjustment method.

[0045] The resource usage threshold can be flexibly set according to the dynamic adjustment requirements of computing resources, and no restrictions are imposed on it here.

[0046] Considering that there are many types of computing resources in the inference service node, the present application can set a resource usage threshold for all computing resources, or set a resource usage threshold for each computing resource to achieve personalized monitoring of different computing resources.

[0047] In some optional embodiments, the computing resource usage of each inference service node can be compared with the corresponding resource usage threshold to determine whether there is any inference service node whose computing resource usage is too high or too low. When the computing resource usage of any inference service node is greater than the corresponding resource usage threshold, it means that the computing resources are tight at this time, then the computing resource dynamic adjustment mechanism can be triggered to adjust the computing resources according to the first adjustment method; when the computing resource usage of any inference service node is less than the corresponding resource usage threshold, it means that there are unused computing resources, that is, idle computing resources, then the computing resource dynamic adjustment mechanism can be triggered to adjust the computing resources according to the second adjustment method, so that the inference service can be ensured to run efficiently and stably, while maximizing the utilization of computing resources. When the computing resource usage of each inference service node is equal to the corresponding resource usage threshold, it means that the computing resources are moderate at this time, then there is no need to trigger the computing resource dynamic adjustment mechanism, and the above-mentioned inference service can continue to be used for inference calculation.

[0048] Optionally, the above-mentioned adjustment of computing resources according to the first adjustment method may include: adding a new reasoning service node; or increasing the number of reasoning workflows in the reasoning service node, wherein the reasoning service node is a reasoning service node whose computing resource usage is greater than the resource usage threshold. In this way, when the workload of any reasoning service exceeds the threshold, the number of reasoning service nodes or reasoning workflows can be automatically increased to improve the parallel processing capability.

[0049] The above-mentioned adjustment of computing resources according to the second adjustment method may include: reducing the inference service nodes; or reducing the number of inference workflows in the inference service nodes, wherein the inference service nodes are inference service nodes whose computing resource usage is less than the resource usage threshold. In this way, when the workload of any inference service is reduced, it is possible to release redundant inference service nodes or reduce the number of inference workflows, which can save computing resources and thus improve the utilization rate of computing resources.

[0050] In other words, the first adjustment method mentioned above in the present application may include a horizontal expansion method and a workflow expansion method; the second adjustment method mentioned above may include a horizontal reduction method and a workflow reduction method.

[0051] Then, computing resources are adjusted according to the first adjustment method, specifically including: adding new reasoning service nodes based on the original reasoning service nodes according to the horizontal expansion method; or, adding new reasoning workflows based on the original number of reasoning workflows of the reasoning service nodes according to the workflow expansion method, wherein the reasoning service node is an reasoning service node whose computing resource usage is greater than the resource usage threshold.

[0052] And, computing resources are adjusted according to the second adjustment method, specifically including: reducing the inference service nodes according to the horizontal reduction method; or reducing the number of inference workflows in the inference service node according to the workflow reduction method, wherein the inference service node is an inference service node whose computing resource usage is less than the resource usage threshold.

[0053] It should be understood that the reasoning workflow in the above-mentioned reasoning service node in this application refers to encapsulating the task of processing reasoning requests into a workflow, which is called the reasoning workflow and can be expressed as a pipeline.

[0054] The embodiments of the present application can realize automatic expansion and contraction of the reasoning service by dynamically adjusting the computing resources according to the computing resource usage of each reasoning service node to cope with concurrent requests under different workloads. This can improve the reasoning service performance and computing resource utilization on the one hand, and improve the scalability and stability of the reasoning service on the other hand, and reduce the response delay of the reasoning service.

[0055] Based on the above embodiments, before, after or during the adjustment of computing resources, it is possible to receive an inference request sent by a user. In this case, the present application can also determine a target inference service node for processing the inference request according to the inference request, and perform inference calculation for the inference request through the target inference service node. Figure 3 , the above contents described in the resource adjustment method provided in the embodiment of the present application are specifically explained.

[0056] like Figure 3As shown, the method may include the following steps:

[0057] S201, upon receiving an inference request, determining a target inference service node.

[0058] S202: Send the reasoning request to the target reasoning service node, so that the target reasoning service node performs reasoning calculation for the reasoning request.

[0059] In this application, the inference request sent by the user may include the target inference model identifier, the data to be processed, and the business scenario corresponding to the inference request. Among them, the target inference model identifier refers to the unique information of the target inference model, such as the name of the target inference model. For example, assuming that the inference request is an image recognition request, then the image recognition request may include the name of the icon recognition model, the image data to be processed, and the image recognition field.

[0060] Considering that the present application is based on a dynamic adjustment mechanism of computing resources, an inference request sent by a user through a client may be received before, during, or after the computing resources are dynamically adjusted.

[0061] Then, after receiving the inference request sent by the user, the inference scheduler in the inference service system can determine the target inference service node from multiple inference service nodes in the inference service system according to the preset scheduling strategy. Then, the inference scheduler allocates the inference request to the target inference service node so that the target inference service node performs inference calculation on the inference request sent by the inference scheduler.

[0062] In some optional embodiments, the target inference service node performs inference calculation on the inference request sent by the inference scheduler. Specifically, upon receiving the inference request sent by the inference scheduler, the target inference service node first parses the inference request to obtain information such as the target inference model identifier, the data to be processed, and the business scenario corresponding to the inference request from the inference request. Then, according to the target inference model identifier and the business scenario corresponding to the inference request, the corresponding target inference model is loaded from the multiple deployed inference models, and then the data to be processed is input into the target inference model, so that the target inference model performs inference calculation on the data to be processed to obtain the inference result.

[0063] Furthermore, the target reasoning service node can return the reasoning result to the reasoning scheduler, so that the reasoning scheduler can feed back the reasoning result to the client used by the user, so that the user can perform subsequent processing according to the reasoning result, such as parsing the reasoning result or performing other operations.

[0064] In some optional embodiments, the above-mentioned determination of the target inference service node from multiple inference service nodes in the inference service system according to a preset scheduling strategy through an inference scheduler in the inference service system may include: obtaining the amount of computing resources to be used of each inference service node; and determining the target inference service node according to the amount of computing resources to be used of each inference service node.

[0065] The amount of computing resources to be used may include: the number of available reasoning workflows, available resources of a central processing unit, and available resources of a graphics processing unit.

[0066] It should be understood that the above-mentioned number of available reasoning workflows refers to reasoning workflows in an idle state.

[0067] Then, the above-mentioned determination of the target reasoning service node based on the amount of computing resources to be used of each reasoning service node may be based on the number of available reasoning workflows, the amount of available resources of the central processing unit and / or the amount of available resources of the graphics processing unit of each reasoning service node.

[0068] As an optional implementation, the present application determines the target reasoning service node according to the number of available reasoning workflows, the available resources of the central processing unit and / or the available resources of the graphics processing unit of each reasoning service node, which may include one of the following implementations:

[0069] Implementation method 1: From the number of available reasoning workflows of all reasoning service nodes, select the reasoning service node with the largest number of available reasoning workflows, and determine the reasoning service node as the target reasoning service node.

[0070] For example, assuming that there are 4 reasoning service nodes in the reasoning service system, when the number of available reasoning workflows of reasoning service node 1 is 5, the number of available reasoning workflows of reasoning service node 2 is 6, the number of available reasoning workflows of reasoning service node 3 is 1, and the number of available reasoning workflows of reasoning service node 4 is 3, it can be determined that the number of available reasoning workflows of reasoning service node 2 is the largest, and at this time, reasoning service node 2 is used as the target reasoning service node.

[0071] Optionally, when the number of available reasoning workflows of all reasoning service nodes is equal, one of the reasoning service nodes may be randomly selected as the target reasoning service node.

[0072] Implementation method 2: Filter out the inference service node with the largest amount of available graphics processor resources from the available graphics processor resources of all inference service nodes, and determine the inference service node as the target inference service node.

[0073] Implementation method three: select the inference service node with the largest amount of available CPU resources from the available CPU resources of all inference service nodes, and determine the inference service node as the target inference service node.

[0074] Taking into account that the graphics processing unit GPU can perform large-scale parallel computing, and the computing power of the central processing unit CPU is relatively balanced and not suitable for large-scale parallel computing, the priority of the graphics processing unit GPU in the inference service should be higher than that of the central processing unit CPU. Therefore, this application prioritizes the amount of available resources of the graphics processing unit of all inference service nodes to determine the target inference service node.

[0075] However, in actual application, the available graphics processor resources of all inference service nodes may be equal. In this case, the inference service node with the largest available CPU resources can be selected from the available CPU resources of all inference service nodes, and the inference service node can be determined as the target inference service node.

[0076] Optionally, when the available resources of the central processing units of all the inference service nodes are equal, one of the inference service nodes may be randomly selected as the target inference service node.

[0077] Implementation method 4: Determine the target inference service node based on the number of available inference workflows, available CPU resources, and available GPU resources of each inference service node.

[0078] Optionally, the fourth implementation may include the following steps:

[0079] In step 1, the inference service node with the largest number of available inference workflows can be firstly selected from the number of available inference workflows of all inference service nodes. When the inference service node with the largest number of available inference workflows is selected, the selected inference service node can be used as the target inference service node. When the number of available inference workflows of all inference service nodes is equal, step 2 is executed.

[0080] Step 2: Filter out the inference service node with the largest amount of available GPU resources from the available GPU resources of all inference service nodes. When the inference service node with the largest amount of available GPU resources is filtered out, the filtered inference service node can be used as the target inference service node. When the available GPU resources of all inference service nodes are equal, execute step 3.

[0081] Step 3: Filter out the inference service node with the largest amount of available CPU resources from the available CPU resources of all inference service nodes. When the inference service node with the largest amount of available CPU resources is filtered out, the filtered inference service node can be used as the target inference service node. When the available CPU resources of all inference service nodes are equal, execute step 4.

[0082] Step 4: Randomly select any one of the inference service nodes as the target inference service node from all the inference service nodes.

[0083] In some optional embodiments, when the target inference service node is performing inference calculations for an inference request, the present application may detect changes in computing resources in the inference service system, such as the addition of a large number of other inference tasks leading to tight computing resources, etc. At this time, a dynamic adjustment mechanism of computing resources can be triggered to dynamically adjust the computing resources of the inference service system.

[0084] Considering that the inference service system in this application may also include a gateway, after receiving the inference request sent by the user, the gateway can authenticate the user and control the flow according to the inference request. Afterwards, the gateway routes the inference request to the inference scheduler, so that the inference scheduler determines the target inference service node from multiple inference service nodes in the inference service system according to the preset scheduling strategy. In this way, the data security of the inference service node can be protected and the stability of the inference service system can be maintained.

[0085] The technical solution disclosed in the embodiment of the present application can improve the inference service performance and computing resource utilization by determining the computing resource usage of each inference service node and then dynamically adjusting the computing resources according to the computing resource usage of each inference service node. In addition, the target inference service node is selected according to the amount of computing resources to be used of each inference service node, which can ensure that the inference request is processed efficiently and accurately, while maximizing the utilization of computing resources.

[0086] Please refer to the attached Figure 4 , a resource adjustment device proposed in an embodiment of the present application is described. Figure 4 As shown, the resource adjustment device 300 includes: a resource determination module 310 and a resource adjustment module 320 .

[0087] The resource determination module 310 is used to determine the computing resource usage of each inference service node;

[0088] The resource adjustment module 320 is used to adjust the computing resources according to the computing resource usage of each of the inference service nodes.

[0089] In an optional implementation of the embodiment of the present application, the resource adjustment module 320 includes:

[0090] A first determining unit, determining whether the computing resource usage of each of the inference service nodes is greater than a resource usage threshold;

[0091] An adjustment unit is used to adjust the computing resources according to a first adjustment method if the computing resource usage of any of the inference service nodes is greater than the resource usage threshold; if the computing resource usage of any of the inference service nodes is less than the resource usage threshold, adjust the computing resources according to a second adjustment method.

[0092] An optional implementation method of an embodiment of the present application is an adjustment unit, which is specifically used to: add a new reasoning service node; or increase the number of reasoning workflows in the reasoning service node, wherein the reasoning service node is an reasoning service node whose computing resource usage is greater than the resource usage threshold.

[0093] An optional implementation method of an embodiment of the present application is an adjustment unit, which is specifically used to: reduce the reasoning service node; or reduce the number of reasoning workflows in the reasoning service node, wherein the reasoning service node is an reasoning service node whose computing resource usage is less than the resource usage threshold.

[0094] In an optional implementation of the embodiment of the present application, the resource adjustment device 300 further includes:

[0095] A node determination module, used to determine a target reasoning service node when receiving a reasoning request;

[0096] The request processing module is used to send the reasoning request to the target reasoning service node, so that the target reasoning service node performs reasoning calculation for the reasoning request.

[0097] In an optional implementation of the embodiment of the present application, the node determination module includes:

[0098] A data acquisition unit, used to acquire the amount of computing resources to be used of each of the inference service nodes;

[0099] The second determining unit is used to determine the target reasoning service node according to the amount of computing resources to be used of each of the reasoning service nodes.

[0100] In an optional implementation of the embodiment of the present application, the amount of computing resources to be used includes: the number of available reasoning workflows, the amount of available resources of the central processing unit, and the amount of available resources of the graphics processing unit;

[0101] Accordingly, the second determining unit is specifically configured to:

[0102] The target inference service node is determined according to the number of available inference workflows, the available resources of the central processing unit and / or the available resources of the graphics processing unit of each of the inference service nodes.

[0103] It should be understood that the device embodiment and the aforementioned method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 4 The device 300 shown can perform Figure 1 The corresponding method embodiments, and the aforementioned and other operations and / or functions of each module in the device 300 are respectively to implement Figure 1 For the sake of brevity, the corresponding processes in each method are not repeated here.

[0104] In the above, the device 300 of the embodiment of the present application is described from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the first aspect method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the first aspect method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above-mentioned first aspect method embodiment in conjunction with its hardware.

[0105] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 400 may include: a memory 410 and a processor 420, wherein the memory 410 is used to store a computer program and transmit the program code to the processor 420. In other words, the processor 420 may call and run the computer program from the memory 410 to implement the resource adjustment method described in the first aspect of the embodiment of the present application.

[0106] For example, the processor 420 may be configured to execute the resource adjustment method according to instructions in the computer program.

[0107] In some embodiments of the present application, the processor 420 may include but is not limited to:

[0108] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0109] In some embodiments of the present application, the memory 410 includes but is not limited to:

[0110] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0111] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 410 and executed by the processor 420 to complete the resource adjustment method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0112] like Figure 5 As shown, the electronic device 400 may further include:

[0113] The transceiver 430 may be connected to the processor 420 or the memory 410 .

[0114] The processor 420 may control the transceiver 430 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include an antenna, and the number of antennas may be one or more.

[0115] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0116] The present application also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to execute the resource adjustment method described in the first aspect above.

[0117] The present application also provides a computer program product including program instructions. When the program instructions are executed on an electronic device, the electronic device executes the resource adjustment method described in the first aspect above.

[0118] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0119] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0120] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0121] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0122] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0123] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A resource adjustment method, characterized in that: include: Determine the computing resource usage of each inference service node; The computing resources are adjusted according to the computing resource usage of each of the inference service nodes.

2. The method according to claim 1, characterized in that The adjusting the computing resources according to the computing resource usage of each of the inference service nodes includes: Determining whether the computing resource usage of each of the inference service nodes is greater than a resource usage threshold; If the computing resource usage of any of the inference service nodes is greater than the resource usage threshold, adjusting the computing resources according to a first adjustment method; If the computing resource usage of any of the inference service nodes is less than the resource usage threshold, the computing resources are adjusted according to the second adjustment method.

3. The method according to claim 2, characterized in that The adjusting the computing resources according to the first adjustment method includes: Add a new inference service node; or, The number of inference workflows in the inference service node is increased, wherein the inference service node is an inference service node whose computing resource usage is greater than the resource usage threshold.

4. The method according to claim 2, characterized in that: The adjusting the computing resources according to the second adjustment method includes: reducing the inference service nodes; or, The number of inference workflows in the inference service node is reduced, wherein the inference service node is an inference service node whose computing resource usage is less than the resource usage threshold.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: When receiving an inference request, determining a target inference service node; The inference request is sent to the target inference service node, so that the target inference service node performs inference calculation for the inference request.

6. The method according to claim 5, characterized in that The determining of the target reasoning service node includes: Obtaining the amount of computing resources to be used of each of the inference service nodes; A target reasoning service node is determined according to the amount of computing resources to be used of each of the reasoning service nodes.

7. The method according to claim 6, characterized in that The amount of computing resources to be used includes: the number of available reasoning workflows, the amount of available resources of the central processing unit, and the amount of available resources of the graphics processing unit; Correspondingly, determining the target reasoning service node according to the amount of computing resources to be used of each of the reasoning service nodes includes: The target inference service node is determined according to the number of available inference workflows, the available resources of the central processing unit and / or the available resources of the graphics processing unit of each of the inference service nodes.

8. A resource adjustment device, characterized in that: include: A resource determination module, used to determine the computing resource usage of each inference service node; The resource adjustment module is used to adjust the computing resources according to the computing resource usage of each of the inference service nodes.

9. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 7.

11. A computer program product comprising program instructions, characterized in that When the program instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 7.

Images