Video memory resource allocation method and device

Abstract

The application discloses a display memory resource allocation method and device. The method comprises the following steps: in response to a creation instruction of creating an inference task, a connection is established with a graphic processor virtual resource pool; a first display memory resource value required by the inference task is estimated, and display memory resources of the first display memory resource value are allocated to the inference task from the graphic processor virtual resource pool; the inference task is run, a change state of a display memory resource value occupied in the inference task running process is monitored, and a maximum display memory resource value required in the inference task running process is determined according to the change state; the display memory resources of the first display memory resource value are returned to the graphic processor virtual resource pool, and display memory resources of the maximum display memory resource value are re-allocated to the inference task from the graphic processor virtual resource pool. The application solves the technical problem that related technologies cannot reasonably allocate display memory resources, thereby reducing the waste of display memory resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of emerging information technology, in particular, to a display memory resource allocation method and device. BACKGROUND

[0002] With the continuous development of virtualization technology and cloud computing technology, GPU (Graphic Processing Unit) has strong graphics image processing capability, so that GPU virtualization technology becomes the terminal of research. Different tasks are allocated different display memory resources by virtualizing GPU resources, so as to meet the needs of display memory isolation and on-demand allocation of GPU virtualization.

[0003] Display memory resource allocation is mainly limited by a static initial fixed value allocated by the host. However, the demand for display memory resources is different for different tasks. So far, relevant technical personnel have not proposed a suitable display memory resource allocation method to effectively avoid the situation of display memory resource waste or insufficient display memory resources.

[0004] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0005] The embodiments of the present application provide a display memory resource allocation method and device to at least solve the technical problem that the related art cannot reasonably allocate display memory resources, thereby reducing display memory resource waste.

[0006] According to an aspect of an embodiment of the present application, a display memory resource allocation method is provided, comprising: in response to a creation instruction of creating an inference task, establishing a connection with a graphic processor virtual resource pool; estimating a first display memory resource value required by the inference task, and allocating display memory resources of the first display memory resource value from the graphic processor virtual resource pool to the inference task; running the inference task, monitoring the change state of the display memory resource value occupied in the inference task running process, and determining the maximum display memory resource value required in the inference task running process according to the change state; returning the display memory resources of the first display memory resource value to the graphic processor virtual resource pool, and re-allocating display memory resources of the maximum display memory resource value from the graphic processor virtual resource pool to the inference task.

[0007] Optionally, the connection with the graphic processor virtual resource pool comprises: calling a query interface in a dynamic link library to establish a connection with a virtual graphic processor driver module; and establishing a connection with the graphic processor virtual resource pool through the virtual graphic processor driver module.

[0008] Optionally, the first memory resource value estimated to be occupied by the inference task includes: a first sub-memory resource value estimated to be occupied by model parameters in the inference task, a second sub-memory resource value estimated to be occupied by gradients of the model parameters, a third sub-memory resource value estimated to be occupied by an optimizer, and a fourth sub-memory resource value estimated to be occupied by intermediate activation values; and the first sub-memory resource value, the second sub-memory resource value, the third sub-memory resource value, and the fourth sub-memory resource value are summed to obtain the first memory resource value.

[0009] Optionally, before the first memory resource value is allocated to the inference task from the virtual resource pool of the graphics processor, the method further includes: determining an available memory resource value in the virtual resource pool of the graphics processor; when the first memory resource value does not exceed the available memory resource value, allocating the first memory resource value to the inference task from the virtual resource pool of the graphics processor; and when the first memory resource value exceeds the available memory resource value, re-determining the first memory resource value.

[0010] Optionally, the change state of the memory resource value occupied by the inference task during the running process is monitored, including: periodically collecting the memory resource value occupied by the inference task during the running process according to a plurality of different time intervals to obtain a plurality of memory resource occupation value sequences changing with time.

[0011] Optionally, the maximum memory resource value to be occupied by the inference task during the running process is determined according to the change state, including: storing the plurality of memory resource occupation value sequences in a queue format, and comparing the values in each memory resource occupation value sequence according to a first-in first-out principle; when the same values in the plurality of memory resource occupation value sequences are found, it is determined whether the same values meet a preset condition, wherein the preset condition includes: in each memory resource occupation value sequence, the slope between the previous value of the same value and the same value is positive, and the slope between the next value of the same value and the same value is negative; if yes, the same value is determined to be the maximum memory resource value; if not, the next same value is determined, and it is determined whether the next same value meets the preset condition, until the same value meeting the preset condition is determined.

[0012] Optionally, the maximum GPU memory resource value occupied during the inference task running process is determined according to the change state, including: for each GPU memory resource occupation value sequence, each value in the GPU memory resource occupation value sequence is mapped according to a target mapping function to obtain a target mapping sequence, wherein the target mapping function is determined according to the change rate of the GPU memory resource occupied during the inference task running process; the plurality of target mapping sequences are stored in a linked list format, wherein the linked list format includes a plurality of nodes, each node includes a data field and a pointer field, the data field is used to store a mapping value and a mapping value frequency in the target mapping sequence, and the pointer field is used to store an address pointing to a next node; the plurality of target mapping sequences are traversed to determine the maximum value in each target mapping sequence; if the maximum values in each target mapping sequence are all the same, the maximum value is determined as the maximum GPU memory resource value.

[0013] According to another aspect of the embodiments of the present application, a GPU memory resource allocation device is also provided, including: a connection module configured to establish a connection with a GPU virtual resource pool in response to a creation instruction of creating an inference task; a first allocation module configured to estimate a first GPU memory resource value occupied by the inference task, and allocate GPU memory resources of the first GPU memory resource value from the GPU virtual resource pool to the inference task; a determination module configured to run the inference task, monitor a change state of a GPU memory resource value occupied during the inference task running process, and determine a maximum GPU memory resource value occupied during the inference task running process according to the change state; and a second allocation module configured to return the GPU memory resources of the first GPU memory resource value to the GPU virtual resource pool, and allocate GPU memory resources of the maximum GPU memory resource value from the GPU virtual resource pool to the inference task.

[0014] According to another aspect of the embodiments of the present application, a non-volatile storage medium is also provided, including a stored program, wherein a device in which the non-volatile storage medium is located executes the GPU memory resource allocation method by running the program.

[0015] According to another aspect of the embodiments of the present application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the GPU memory resource allocation method by running the computer program.

[0016] In the embodiment of the present application, in response to the creation instruction of creating the inference task, a connection is established with the graphics processor virtual resource pool; a first video memory resource value occupied by the inference task is estimated, and video memory resources of the first video memory resource value are allocated to the inference task from the graphics processor virtual resource pool; the inference task is run, the change state of the video memory resource value occupied in the inference task running process is monitored, and the maximum video memory resource value required to be occupied in the inference task running process is determined according to the change state; the video memory resources of the first video memory resource value are returned to the graphics processor virtual resource pool, and the video memory resources of the maximum video memory resource value are re-allocated to the inference task from the graphics processor virtual resource pool. Wherein, different types of image processors are virtualized into an image processor virtual resource pool, and at the same time, the inference task is simulated and the dynamic change of the video memory resource value in the inference task running process is monitored, so as to facilitate determining the maximum video memory resource value required to be occupied in the inference task running process according to the current inference task, avoiding the situation that the video memory utilization is too low or too high when the video memory is allocated on demand in the image processor virtualization background, thereby realizing the technical effect of fully utilizing the video memory resources, and further solving the technical problem that the related art cannot reasonably allocate the video memory resources, thereby reducing the waste of video memory resources. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings used to provide further understanding of the present application, and constitute a part of the present application, the illustrative embodiments of the present application and its description are used to explain the present application, and do not constitute improper limitation on the present application. In the drawings:

[0018] Figure 1 is a schematic diagram of an optional video memory resource allocation method according to an embodiment of the present application;

[0019] Figure 2 is a correspondence diagram of an optional target mapping value and a storage unit according to an embodiment of the present application;

[0020] Figure 3 is a structure diagram of an optional video memory resource allocation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable the personnel in the technical field to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second" and the like in the description and in the claims of the present application and the drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged, where appropriate, so that the embodiments of the present application described herein can be carried out in other than the order shown or described herein. Furthermore, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, processes, methods, systems, products, or devices that comprise a list of steps or units are not necessarily limited to those clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products, or devices.

[0023] Embodiment 1

[0024] According to the embodiments of the present application, a video memory resource allocation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0025] Figure 1 is a flowchart of an optional video memory resource allocation method according to the embodiments of the present application, as shown in Figure 1 The method comprises at least steps S102-S108, wherein:

[0026] Step S102, in response to the creation instruction of creating an inference task, a connection is established with a graphics processor virtual resource pool.

[0027] In the technical solution provided in the above step S102 of the present application, the creation instruction of creating an inference task by the user terminal is received, the simulation inference task is started, and based on the graphics processor virtualization background, a plurality of different GPU cards are virtualized into a graphics processor virtual resource pool, and a communication connection is established with the creation instruction of creating an inference task by the user.

[0028] Step S104, estimating the first video memory resource value occupied by the inference task, and allocating the first video memory resource value of the video memory resource from the graphics processor virtual resource pool for the inference task.

[0029] In the technical solution provided in the foregoing step S104 of the present application, the yolov5s model is used to perform an inference task, and a first video memory resource value occupied by the inference task is estimated, wherein the maximum consumption of video memory is about 3GB (i.e., the maximum setting of batchsize is 32), and when the video memory resource available in the graphics processor virtual resource pool is sufficient to support the first video memory resource value occupied by the current inference task, the graphics processor virtual resource pool is applied to allocate the video memory resource of the first video memory resource value to the inference task, wherein the first video memory resource value includes but is not limited to 3GB.

[0030] In step S106, the inference task is run, the change state of the video memory resource value occupied by the inference task during the running process is monitored, and the maximum video memory resource value required to be occupied by the inference task during the running process is determined according to the change state.

[0031] In the technical solution provided in the foregoing step S106 of the present application, when the available video memory resource value of the graphics processor virtual resource pool is sufficient to support the completion of the inference running of the current simulation inference task, the inference task is formally started, and the information of the GPU is read by using the built-in function of the python third-party library GPUtil module, wherein the read information includes: acquisition time, query times, GPU ID, GPU memory usage, etc., and the change state of the video memory resource value occupied by the simulation inference task during the running process is monitored through the above data.

[0032] Specifically, GPUtil is a Python module for obtaining GPU status from NVIDIA GPUs using nvidia-smi, GPUtil finds all GPUs on the computer, determines their availability and returns an ordered list of available GPUs. Among them, the availability is determined based on the current memory consumption and load of each GPU. In addition, the built-in function get_gpu_info obtains the information of all GPUs through GPUtil: the current time time, the GPU card number GPU ID, the total amount of GPU GPU.memoryTotal, the usage of GPU GPU.memoryUsed, and the utilization rate of GPU GPU.memoryUtil.

[0033] In step S108, the video memory resource of the first video memory resource value is returned to the graphics processor virtual resource pool, and the video memory resource of the maximum video memory resource value is re-allocated to the inference task from the graphics processor virtual resource pool.

[0034] In the technical solution provided in the foregoing step S108 of the present application, after the first video memory resource value is determined, the video memory resource of the first video memory resource value required to be occupied by the current virtual inference task model is returned to the graphics processor virtual resource pool, and the video memory resource of the first video memory resource value is used as a reference value for the video memory resource required to be occupied by the user during the running process of the inference task.

[0035] In the technical solution provided in steps S102-S108 of the present application, a connection is established with the graphics processor virtual resource pool in response to a creation instruction for creating an inference task; a first video memory resource value occupied by the inference task is estimated, and video memory resources of the first video memory resource value are allocated to the inference task from the graphics processor virtual resource pool; the inference task is run, the change state of the video memory resource value occupied in the inference task running process is monitored, and the maximum video memory resource value required to be occupied in the inference task running process is determined according to the change state; the video memory resources of the first video memory resource value are returned to the graphics processor virtual resource pool, and video memory resources of the maximum video memory resource value are re-allocated to the inference task from the graphics processor virtual resource pool. Wherein, different types of image processors are virtualized into an image processor virtual resource pool, and at the same time, the dynamic change of the video memory resource value in the inference task running process is monitored by simulating the inference task, so as to facilitate determining the maximum video memory resource value required to be occupied in the inference task running process according to the current inference task, avoiding the situation that the video memory utilization is too low or too high when the video memory is allocated on demand in the background of virtualizing the image processor, thereby realizing the technical effect of fully utilizing the video memory resources, and further solving the technical problem that the related art cannot reasonably allocate the video memory resources, thereby reducing the waste of video memory resources.

[0036] The above method in the embodiment will be further introduced below.

[0037] As an optional implementation, in the technical solution provided in step S102 of the present application, the method comprises: calling a query interface in a dynamic link library to establish a connection with a virtual graphics processor driver module; and establishing a connection with a graphics processor virtual resource pool through the virtual graphics processor driver module.

[0038] In this embodiment, the ctypes protocol is used to load the dynamic link library through the cdll object, wherein the dynamic link library can use the libnvidiaml.so dynamic library file provided by the NVIDIA manufacturer, and the query interface in the dynamic link library is called to establish a communication connection with the virtual graphics processor driver module, wherein the query interface is a set of nvml (nvidia management library) functions built in libnvidia_ml.so; and a communication connection is established between the virtual graphics processor driver module and the graphics processor virtual resource pool. Specifically, ctypes is an external function library of Python, which provides data types compatible with C language and allows functions in DLL or shared library to be called. The ctypes exports the cdll object, and by operating the attributes of the cdll object, the dynamic link library of external C language can be loaded.

[0039] As an optional implementation, in the technical solution provided by the step S104 of the present application, the method comprises: estimating a first sub-video memory resource value occupied by model parameters in the inference task, a second sub-video memory resource value occupied by gradients of the model parameters, a third sub-video memory resource value occupied by the optimizer, and a fourth sub-video memory resource value occupied by intermediate activation values; summing the first sub-video memory resource value, the second sub-video memory resource value, the third sub-video memory resource value, and the fourth sub-video memory resource value to obtain the first video memory resource value.

[0040] In this embodiment, the video memory resource consumption of the inference task based on the pytorch framework comprises: a first sub-video memory resource value occupied by model parameters, a second sub-video memory resource value occupied by gradients of the model parameters, a third sub-video memory resource value occupied by the optimizer, and a fourth sub-video memory resource value occupied by intermediate activation values. The first video memory resource value occupied by the inference task is estimated by summing the above-mentioned several types of video memory resource values.

[0041] As an optional implementation, in the technical solution provided by the step S104 of the present application, before the first video memory resource value of the inference task is allocated from the virtual resource pool of the graphics processor, the method further comprises: determining an available video memory resource value in the virtual resource pool of the graphics processor; when the first video memory resource value does not exceed the available video memory resource value, allocating the first video memory resource value of the inference task from the virtual resource pool of the graphics processor; when the first video memory resource value exceeds the available video memory resource value, re-determining the first video memory resource value.

[0042] In this embodiment, the available video memory resource value in the virtual resource pool of the graphics processor is obtained, and it is determined whether the available video memory resource value meets the preliminary screening condition, wherein the preliminary screening condition is whether the available video memory resource value in the virtual resource pool of the graphics processor is sufficient to support the current inference model. When the first video memory resource value is less than the available video memory resource value, the first video memory resource value of the inference task is allocated from the virtual resource pool of the graphics processor; otherwise, the first video memory resource value occupied by the inference task is re-estimated.

[0043] As an optional implementation, in the technical solution provided by the step S106 of the present application, the method comprises: periodically collecting the video memory resource value occupied by the inference task in the running process according to a plurality of different time intervals to obtain a plurality of video memory resource occupation value sequences varying with time.

[0044] In this embodiment, a plurality of different time intervals are adopted, and the occupied GPU resource values during the inference task running are periodically collected respectively, and the NVIDI-SMI command is refreshed to save the dynamic GPU resource values according to the above time intervals, so as to obtain a plurality of GPU resource occupation value sequences changing with time. For example, two sets of control parameters are taken as time intervals of 1 second and 2 seconds, that is, the time interval parameters are set as delay = 1 or delay = 2. In actual application, the specific value of the time interval can be set according to the actual scene.

[0045] The GPU resources in the inference task mainly include the following three parts: model weight, input and output, and intermediate result. For a fully connected layer network, the model weight mainly refers to the parameters of the model itself, such as the parameters of the convolution layer is a high kernel convolution kernel, the parameters of the fully connected layer is a two-dimensional matrix, the activation function layer and the pooling layer do not contain parameters; and the intermediate result can be understood as, in the forward calculation, since the output of the previous layer corresponds to the input of the next layer, the intermediate result of the two connected layers also needs to occupy GPU resources; the input and output occupy less GPU resources. Therefore, the maximum GPU resource value occupied during the running of the inference task of the same model is determined to be the same, and with the gradual completion of the inference task running, the GPU resource value presents a change trend of growth-plateau-growth-again plateau, so the GPU resource value occupied during the inference task running will be stably maintained at the maximum value, and finally all are released with the simulation inference task running technology. Therefore, based on the change rule of the above GPU resource value, the maximum GPU resource value occupied during the inference task running is determined.

[0046] As an optional implementation, in the technical solution provided by the above step S106 of the application, the method comprises: storing a plurality of GPU resource occupation value sequences in a queue format, and comparing the values in each GPU resource occupation value sequence according to the first-in first-out principle; when the same values in the plurality of GPU resource occupation value sequences are found, it is judged whether the same values meet the preset condition, wherein the preset condition comprises: in each GPU resource occupation value sequence, the slope between the previous value of the same value and the same value is positive, and the slope between the next value of the same value and the same value is negative; if it is satisfied, the same value is determined as the maximum GPU resource value; if it is not satisfied, the next same value is determined, and it is determined whether the next same value meets the preset condition, until the same value meeting the preset condition is determined.

[0047] In this embodiment, the multiple sets of video memory resource occupation value sequences are stored in a queue data format, and the values in the multiple sets of video memory resource occupation value sequences are compared based on the first-in first-out principle of queue storage. If the values in the multiple sets of video memory resource occupation value sequences are different, the video memory resource value is removed. If the values in the multiple sets of video memory resource occupation value sequences are the same, the value is determined as the same value, and whether the curve slope between the same value and the previous value and the next value satisfies the change from positive to negative is determined. If it is satisfied, the same value is determined as the maximum video memory resource value. If it is not satisfied, whether the next equal value satisfies the slope change rule is determined until the obtained video memory resource value satisfies the slope change rule, and the value is determined as the maximum video memory resource value.

[0048] As an optional implementation, in the technical solution provided in the above step S106 of the application, the method comprises: for each video memory resource occupation value sequence, performing mapping processing on each value in the video memory resource occupation value sequence according to a target mapping function to obtain a target mapping sequence, wherein the target mapping function is determined according to the change rate of the occupied video memory resource in the inference task running process; storing the multiple target mapping sequences in a linked list format, wherein the linked list format comprises multiple nodes, each node comprises a data field and a pointer field, the data field is used to store the mapping value and the occurrence frequency of the mapping value in the target mapping sequence, and the pointer field is used to store the address of the next node; traversing the multiple target mapping sequences to determine the maximum value in each target mapping sequence; and if the maximum values in each target mapping sequence are all the same, the maximum value is determined as the maximum video memory resource value.

[0049] In this embodiment, for the two sets of experimental data S1 and S2, the following mapping function can be designed:

[0050] V=f(M)

[0051] Wherein, V represents the value obtained by mapping the video memory value collected in the experimental data according to a certain rule; M represents a certain video memory value collected in the experimental data; f represents the mapping rule. Therefore, for each video memory resource occupation value sequence, each value in the video memory resource occupation value sequence is mapped according to the above mapping function, and the target mapping sequence can be obtained, and the target mapping value is stored in the same storage unit, wherein the correspondence between the storage unit and the target mapping value is referred to Figure 2The storage units can be designed in a linked list format, in which the mapping value corresponding to the V1 value in S1 and the occurrence frequency of the mapping value in the target mapping sequence are stored in the linked list L1, and the mapping value corresponding to the V2 value in S2 and the occurrence frequency of the mapping value in the target mapping sequence are stored in the linked list L2. The plurality of linked lists L1 and L2 are traversed respectively, the nodes with a V value of 0 are deleted, the nodes with the largest display memory resource value in L1 and L2 are determined, and it is determined whether the maximum values are the same. If the maximum values are the same, the maximum value is determined to be the maximum display memory resource value. Otherwise, the sample quantity is increased to continue the determination.

[0052] In the above steps, the dynamic display memory resource change value in the simulation inference task is monitored, and the maximum display memory resource value most suitable for the current inference task model is calculated according to the sampling value matching, which helps the user to apply appropriate GPU virtualization display memory resource for the current inference task, avoids the low or high display memory utilization that is prone to occur in the GPU virtualization background, and fully utilizes the GPU virtualization resource.

[0053] Embodiment 2

[0054] According to the embodiments of the present application, a display memory resource allocation device for implementing the display memory resource allocation method in Embodiment 1 is further provided, Figure 3 is a structural diagram of an optional display memory resource allocation device according to the embodiments of the present application, as Figure 3 shown, the display memory resource allocation device at least includes a connection module 31, a first allocation module 32, a determination module 33 and a second allocation module 34, wherein:

[0055] The connection module 31 is configured to establish a connection with the graphics processor virtual resource pool in response to a creation instruction for creating an inference task.

[0056] Specifically, the connection module 31 receives a creation instruction for creating an inference task by a user end, starts a simulation inference task, and based on the graphics processor virtualization background, virtualizes a plurality of different GPU cards into a graphics processor virtual resource pool, and establishes a communication connection with the creation instruction for creating an inference task by the user.

[0057] As an optional implementation, the connection module 31 calls a query interface in a dynamic link library to establish a connection with a virtual graphics processor driver module; and establishes a connection with the graphics processor virtual resource pool through the virtual graphics processor driver module.

[0058] In this embodiment, the dynamic link library is loaded through the cdll object in the ctypes protocol, wherein the dynamic link library can adopt the libnvidia_ml.so dynamic library file provided by the NVIDIA manufacturer, and a query interface in the dynamic link library is called to establish a communication connection with the virtual graphics processor driver module, wherein the query interface is a set of nvml (nvidia management library) functions built in the libnvidia_ml.so; and then the virtual graphics processor driver module establishes a communication connection with the graphics processor virtual resource pool. Specifically, ctypes is an external function library of Python, which provides data types compatible with C language and allows functions in DLL or shared library to be called. The cdll object is exported through ctypes, and by operating the attributes of the cdll object, the dynamic link library of external C language can be loaded

[0059] The first allocation module 32 is configured to estimate a first video memory resource value required by the inference task, and allocate video memory resources of the first video memory resource value from the graphics processor virtual resource pool to the inference task.

[0060] Specifically, the yolov5s model is used for the inference task, and the first video memory resource value required by the inference task is estimated, wherein the maximum consumption of video memory is about 3GB (i.e., the maximum batchsize is set to 32), and when the video memory resources that can be applied for in the graphics processor virtual resource pool are sufficient to support the first video memory resource value required by the current inference task, the graphics processor virtual resource pool is applied to allocate video memory resources of the first video memory resource value to the inference task, wherein the first video memory resource value includes but is not limited to 3GB.

[0061] As an optional implementation, the first allocation module 32 estimates a first sub-video memory resource value required by model parameters in the inference task, a second sub-video memory resource value required by gradients of the model parameters, a third sub-video memory resource value required by an optimizer, and a fourth sub-video memory resource value required by intermediate activation values; and sums the first sub-video memory resource value, the second sub-video memory resource value, the third sub-video memory resource value, and the fourth sub-video memory resource value to obtain the first video memory resource value.

[0062] In this embodiment, the video memory resource consumption of the inference task based on the pytorch framework includes: a first sub-video memory resource value required by model parameters, a second sub-video memory resource value required by gradients of the model parameters, a third sub-video memory resource value required by an optimizer, and a fourth sub-video memory resource value required by intermediate activation values; and the first sub-video memory resource value, the second sub-video memory resource value, the third sub-video memory resource value, and the fourth sub-video memory resource value are summed to estimate the first video memory resource value required by the inference task.

[0063] As an optional implementation, before the first allocation module 32 allocates the first GPU memory resource value of GPU memory resource for the inference task from the graphic processor virtual resource pool, it further includes: determining the available GPU memory resource value in the graphic processor virtual resource pool; when the first GPU memory resource value does not exceed the available GPU memory resource value, allocating the first GPU memory resource value of GPU memory resource for the inference task from the graphic processor virtual resource pool; when the first GPU memory resource value exceeds the available GPU memory resource value, re-determining the first GPU memory resource value.

[0064] In this embodiment, the available GPU memory resource value in the graphic processor virtual resource pool is obtained, and it is determined whether the available GPU memory resource value meets the preliminary screening condition, wherein the preliminary screening condition is whether the available GPU memory resource value in the graphic processor virtual resource pool is sufficient to support the current inference model, and when the first GPU memory resource value is less than the available GPU memory resource value, the first GPU memory resource value of GPU memory resource is allocated for the inference task from the graphic processor virtual resource pool; otherwise, the first GPU memory resource value occupied by the inference task is re-estimated.

[0065] The determination module 33 is configured to run the inference task, monitor the change state of the GPU memory resource value occupied in the inference task running process, and determine the maximum GPU memory resource value required in the inference task running process according to the change state.

[0066] Specifically, when the available GPU memory resource value of the graphic processor virtual resource pool is sufficient to support the current simulation inference task to complete the inference running, the inference task is formally started, and the information of the GPU is read by using the built-in function of the python third-party library GPUtil module, wherein the read information includes: acquisition time, query times, GPU ID, GPU memory usage, etc., and the change state of the GPU memory resource value occupied in the simulation inference task running process is monitored through the above data.

[0067] Specifically, GPUtil is a Python module for using nvidia-smi to obtain the GPU state from NVIDIA GPU, GPUtil finds all GPUs on the computer, determines their availability and returns an ordered list of available GPUs. Among them, the availability is determined based on the current memory consumption and load of each GPU. In addition, the built-in function get_gpu_info obtains the information of all GPUs through GPUtil: the current time time, the GPU card number GPU ID, the total amount of GPU GPU.memoryTotal, the usage of GPU GPU.memoryUsed, and the utilization rate of GPU GPU.memoryUtil.

[0068] The main memory resource in the inference task includes the following three parts: model weight, input and output, and intermediate result. For a fully connected layer network, the model weight mainly refers to the parameters of the model itself, such as the parameter of a convolution layer is a high kernel convolution kernel, the parameter of a fully connected layer is a two-dimensional matrix, and the activation function layer and the pooling layer do not contain parameters; the intermediate result can be understood as, in the forward calculation, since the output of the previous layer corresponds to the input of the next layer, the intermediate result of the two connected layers also needs to occupy the memory resource; the input and output occupy a small memory resource. Thus, for the same model, the maximum memory resource value required in the inference task running process is the same, and as the inference task running is gradually completed, the memory resource value presents a trend of growth-plateau-growth-plateau again, thus the memory resource value required in the inference task running process is stably maintained at the maximum value, and finally all are released with the simulation inference task running. Therefore, based on the change rule of the memory resource value, the maximum memory resource value required in the inference task running process is determined.

[0069] As an optional implementation, the determining module 33 can determine the maximum memory resource value required in the inference task running process according to the following rules: store a plurality of memory resource occupation value sequences in a queue format, and compare the values in each memory resource occupation value sequence according to the first-in first-out principle; when the same value is found in the plurality of memory resource occupation value sequences, it is judged whether the same value meets a preset condition, wherein the preset condition includes: in each memory resource occupation value sequence, the slope between the previous value of the same value and the same value is positive, and the slope between the next value of the same value and the same value is negative; if it is met, the same value is determined as the maximum memory resource value; if it is not met, the next same value is determined, and it is determined whether the next same value meets the preset condition, until the same value meeting the preset condition is determined.

[0070] In this embodiment, a plurality of groups of memory resource occupation value sequences are stored in a queue data format, the values in each memory resource occupation value sequence are compared based on the first-in first-out principle of queue storage, if the values in the plurality of memory resource occupation value sequences are different, the memory resource value is removed; if the same value is found in the plurality of memory resource occupation value sequences, the value is determined as the same value, it is judged whether the curve slope between the same value and the previous value and the next value thereof meets the change from positive to negative, if it is met, the same value is determined as the maximum memory resource value; if it is not met, it is continued to be judged whether the next equal value meets the slope change rule, until the obtained memory resource value meets the slope change rule, then the value is determined as the maximum memory resource value.

[0071] As an optional implementation, the determining module 33 can also determine the maximum GPU memory resource value required by the inference task running process according to the following rules: for each GPU memory resource occupation value sequence, each value in the GPU memory resource occupation value sequence is mapped according to a target mapping function to obtain a target mapping sequence, wherein the target mapping function is determined according to the change rate of the GPU memory resource occupied in the inference task running process; the plurality of target mapping sequences are stored in a linked list format, wherein the linked list format includes a plurality of nodes, each node includes a data field and a pointer field, the data field is used to store the mapping value and the occurrence frequency of the mapping value in the target mapping sequence, and the pointer field is used to store the address of the next node; the plurality of target mapping sequences are traversed to determine the maximum value in each target mapping sequence; if the maximum values in each target mapping sequence are all the same, the maximum value is determined as the maximum GPU memory resource value.

[0072] In this embodiment, for the two groups of experimental data S1 and S2 sampled, the following mapping function can be designed:

[0073] V = f(M)

[0074] Wherein, V represents the value obtained by mapping the GPU memory value collected in the experimental data according to a certain rule; M represents a certain GPU memory value collected in the experimental data; f represents the mapping rule. Therefore, for each GPU memory resource occupation value sequence, each value in the GPU memory resource occupation value sequence is mapped according to the above mapping function, and the target mapping sequence can be obtained, and the target mapping value is stored in the same storage unit, wherein the correspondence between the storage unit and the target mapping value is referred to Figure 2 The storage unit can be designed in a linked list format, at this time, the mapping value corresponding to V1 value in S1 and the occurrence frequency of the mapping value in the target mapping sequence are stored in the linked list L1, and the mapping value corresponding to V2 value in S2 and the occurrence frequency of the mapping value in the target mapping sequence are stored in the linked list L2, the plurality of linked lists L1 and L2 are traversed respectively, the nodes with V value of 0 are deleted, the nodes with the maximum GPU memory resource value in L1 and L2 are determined, and whether the maximum values are the same is determined, if equal, the maximum value is determined as the maximum GPU memory resource value; otherwise, increase the sample size and continue to determine.

[0075] The second allocation module 34 is configured to return the GPU memory resource of the first GPU memory resource value to the GPU virtual resource pool, and re-allocate the GPU memory resource of the maximum GPU memory resource value from the GPU virtual resource pool for the inference task.

[0076] After determining the first GPU memory resource value, the GPU memory resource of the first GPU memory resource value required by the current virtual inference task model is returned to the GPU virtual resource pool, and the second allocation module 34 takes the GPU memory resource of the first GPU memory resource value as a reference value of the GPU memory resource required by the user to apply the inference task running process.

[0077] It should be noted that each module in the display memory resource allocation apparatus in the embodiments of the present application corresponds to each implementation step of the display memory resource allocation method in Embodiment 1. Since Embodiment 1 has been described in detail, the details not embodied in this embodiment can be referred to Embodiment 1, and will not be described in detail here.

[0078] Embodiment 3

[0079] According to the embodiments of the present application, a non-volatile storage medium is also provided, which includes a stored program. The device in which the non-volatile storage medium is located executes the display memory resource allocation method in Embodiment 1 by running the program.

[0080] Optionally, the device in which the non-volatile storage medium is located executes the following steps by running the program: in response to a creation instruction for creating an inference task, establishing a connection with a graphics processor virtual resource pool; estimating a first display memory resource value required by the inference task, and allocating display memory resources of the first display memory resource value from the graphics processor virtual resource pool to the inference task; running the inference task, monitoring the change state of the display memory resource value occupied in the inference task running process, and determining the maximum display memory resource value required in the inference task running process according to the change state; returning the display memory resources of the first display memory resource value to the graphics processor virtual resource pool, and re-allocating display memory resources of the maximum display memory resource value from the graphics processor virtual resource pool to the inference task.

[0081] According to the embodiments of the present application, a processor is also provided, which is used to run a program. When the program is running, the display memory resource allocation method in Embodiment 1 is executed.

[0082] Optionally, when the program is running, the following steps are implemented: in response to a creation instruction for creating an inference task, establishing a connection with a graphics processor virtual resource pool; estimating a first display memory resource value required by the inference task, and allocating display memory resources of the first display memory resource value from the graphics processor virtual resource pool to the inference task; running the inference task, monitoring the change state of the display memory resource value occupied in the inference task running process, and determining the maximum display memory resource value required in the inference task running process according to the change state; returning the display memory resources of the first display memory resource value to the graphics processor virtual resource pool, and re-allocating display memory resources of the maximum display memory resource value from the graphics processor virtual resource pool to the inference task.

[0083] According to the embodiments of the present application, an electronic device is also provided, which includes a memory and a processor. The memory stores a computer program, and the processor is configured to execute the display memory resource allocation method in Embodiment 1 by the computer program.

[0084] Optionally, the processor is configured to implement the following steps by computer program execution: in response to a creation instruction of creating an inference task, establishing a connection with a graphics processor virtual resource pool; estimating a first video memory resource value occupied by the inference task, allocating video memory resources of the first video memory resource value from the graphics processor virtual resource pool to the inference task; running the inference task, monitoring a change state of a video memory resource value occupied in a running process of the inference task, and determining a maximum video memory resource value required to be occupied in the running process of the inference task according to the change state; returning the video memory resources of the first video memory resource value to the graphics processor virtual resource pool, and re-allocating video memory resources of the maximum video memory resource value from the graphics processor virtual resource pool to the inference task.

[0085] The above sequence numbers of the embodiments of the application are only for description, and do not represent advantages or disadvantages of the embodiments.

[0086] In the above embodiments of the application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0087] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the unit embodiment described above is only schematic. For example, the division of the units can be a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between units can be indirect coupling or communication connection through some interface, electrical or other form.

[0088] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0089] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0090] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0091] The above is only the preferred embodiment of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method for allocating video memory resources, characterized in that, The method comprises: in response to a creation instruction for creating an inference task, establishing a connection with a graphics processor virtual resource pool; estimating a first video memory resource value required by the inference task, and allocating video memory resources of the first video memory resource value from the graphics processor virtual resource pool to the inference task; running the inference task, monitoring the change state of the video memory resource value occupied by the inference task during the running process, and determining the maximum video memory resource value required by the inference task during the running process according to the change state, wherein a plurality of video memory resource occupation value sequences changing with time are obtained by periodically collecting the video memory resource value occupied by the inference task during the running process according to a plurality of different time intervals; the plurality of video memory resource occupation value sequences are stored in a queue format, and the values in each of the video memory resource occupation value sequences are compared according to the first-in first-out principle; when the same values are found in the plurality of video memory resource occupation value sequences, it is judged whether the same values meet a preset condition, wherein the preset condition includes: in each of the video memory resource occupation value sequences, the slope between the previous value of the same value and the same value is positive, and the slope between the next value of the same value and the same value is negative; if it is satisfied, the same value is determined as the maximum video memory resource value; returning the video memory resources of the first video memory resource value to the graphics processor virtual resource pool, and re-allocating video memory resources of the maximum video memory resource value from the graphics processor virtual resource pool to the inference task.

2. The method of claim 1, wherein, Establishing a connection with a graphics processor virtual resource pool comprises: calling a query interface in a dynamic link library to establish a connection with a virtual graphics processor driver module; establishing a connection with the graphics processor virtual resource pool through the virtual graphics processor driver module.

3. The method of claim 1, wherein, Estimating the first video memory resource value required by the inference task comprises: estimating a first sub-video memory resource value required by model parameters in the inference task, a second sub-video memory resource value required by gradients of the model parameters, a third sub-video memory resource value required by an optimizer, and a fourth sub-video memory resource value required by intermediate activation values; summing the first sub-video memory resource value, the second sub-video memory resource value, the third sub-video memory resource value, and the fourth sub-video memory resource value to obtain the first video memory resource value.

4. The method of claim 1, wherein, Before allocating the video memory resources of the first video memory resource value from the graphics processor virtual resource pool to the inference task, the method further comprises: determining the available video memory resource value in the graphics processor virtual resource pool; when the first video memory resource value does not exceed the available video memory resource value, allocating the video memory resources of the first video memory resource value from the graphics processor virtual resource pool to the inference task; when the first video memory resource value exceeds the available video memory resource value, re-determining the first video memory resource value.

5. The method of claim 1, wherein, Determining the maximum video memory resource value required by the inference task during the running process according to the change state comprises: if the same value does not meet the preset condition, continue to determine the next same value and determine whether the next same value meets the preset condition until the same value that meets the preset condition is determined.

6. The method of claim 1, wherein, The maximum video memory resource value to be occupied during the inference task running is determined according to the change state, and the method comprises the following steps: For each of the video memory resource occupation value sequences, a target mapping sequence is obtained by performing mapping processing on each value in the video memory resource occupation value sequence according to a target mapping function, wherein the target mapping function is determined according to the change rate of the video memory resource occupied during the inference task running; A plurality of target mapping sequences are stored in a linked list format, wherein the linked list format comprises a plurality of nodes, each node comprises a data field and a pointer field, the data field is used to store a mapping value and the frequency of occurrence of the mapping value in the target mapping sequence, and the pointer field is used to store the address of the next node; Each of the target mapping sequences is traversed to determine the maximum value in each of the target mapping sequences; If the maximum values in each of the target mapping sequences are all the same, the maximum value is determined as the maximum video memory resource value.

7. A device for allocating video memory resources, characterized in that Comprise: The connection module is used for establishing a connection with a graphics processor virtual resource pool in response to a creation instruction for creating an inference task; The first allocation module is used for estimating a first video memory resource value to be occupied by the inference task, and allocating video memory resources of the first video memory resource value from the graphics processor virtual resource pool for the inference task; The determination module is used for running the inference task, monitoring the change state of the video memory resource value occupied during the inference task running, and determining the maximum video memory resource value to be occupied during the inference task running according to the change state, wherein a plurality of video memory resource occupation value sequences changing with time are obtained by periodically collecting the video memory resource value occupied during the inference task running according to a plurality of different time intervals; the plurality of video memory resource occupation value sequences are stored in a queue format, and the values in each of the video memory resource occupation value sequences are compared according to the first-in first-out principle; when the same values are found in the plurality of video memory resource occupation value sequences, it is judged whether the same values meet a preset condition, wherein the preset condition comprises: in each of the video memory resource occupation value sequences, the slope between the previous value of the same value and the same value is positive, and the slope between the next value of the same value and the same value is negative; if it is satisfied, the same value is determined as the maximum video memory resource value; The second allocation module is used for returning the video memory resources of the first video memory resource value to the graphics processor virtual resource pool, and re-allocating video memory resources of the maximum video memory resource value from the graphics processor virtual resource pool for the inference task.

8. A non-volatile storage medium, comprising: The non-volatile storage medium comprises a stored program, wherein the device in which the non-volatile storage medium is located executes the video memory resource allocation method of any one of claims 1 to 6 by running the program.

9. An electronic device, comprising: Comprise: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the video memory resource allocation method of any one of claims 1 to 6 by the computer program.

Images