Video memory allocation method and device

Abstract

The present disclosure provides a method and device for allocating video memory, which relates to the field of communication technology. The above method includes: detecting the actual first video memory parameter of the video memory in the server during operation, and the server is deployed with multiple models; predicting the second video memory parameter of the video memory based on the first video memory parameter; predicting the user request information of the target user in a future preset time period for the server; and allocating video memory for the future preset time period based on the first video memory parameter, the second video memory parameter and the user request information. The present disclosure can address the resource demand conflicts of multiple models by developing the memory and CPU computing power that has not been fully used, so as to deploy multiple models more compactly on a computing device, thereby achieving the coexistence and concurrency of multiple models.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular to a method and device for allocating video memory. Background Art

[0002] Video memory is the graphics card memory, also known as the frame buffer. Its function is to store rendering data that has been processed or is about to be retrieved by the graphics chip. Like the computer's memory, video memory is a component used to store graphics information to be processed.

[0003] The capacity of video memory on a server is limited and often requires time-sharing and multiplexing. How can we fully utilize the video memory on the server based on user requests? Summary of the Invention

[0004] In view of this, the purpose of the present disclosure is to provide a method and device for allocating video memory, which can specifically solve the existing problems.

[0005] Based on the above-mentioned purpose, in a first aspect, the present disclosure proposes a method for allocating video memory, comprising: detecting an actual first video memory parameter of the video memory in the server during operation, wherein the server is deployed with multiple models; predicting a second video memory parameter for the video memory based on the first video memory parameter; predicting user request information of a target user in a future preset time period for the server; and allocating video memory for the future preset time period based on the first video memory parameter, the second video memory parameter and the user request information.

[0006] In the second aspect, a video memory allocation device is also provided, including: a detection unit, configured to detect an actual first video memory parameter of the video memory in the server during operation, and the server is deployed with multiple models; a parameter prediction unit, configured to predict a second video memory parameter for the video memory based on the first video memory parameter; a request prediction unit, configured to predict user request information of a target user in a future preset time period for the server; and an allocation unit, configured to allocate video memory for the future preset time period based on the first video memory parameter, the second video memory parameter and the user request information.

[0007] In a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0008] In a fourth aspect, a computer-readable storage medium is further provided, on which a computer program is stored, and the computer program is executed by a processor to implement any method described in the first aspect.

[0009] In a fifth aspect, a computer program product is also provided, comprising a computer program, wherein the computer program is executed by a processor to implement any one of the methods described in the first aspect.

[0010] In summary, the present disclosure has at least the following beneficial effects: It addresses resource demand conflicts among multiple models by exploiting underutilized memory and CPU computing power, enabling more compact deployment of multiple models on a single computing device, thus enabling the coexistence and concurrency of multiple models. Furthermore, based on the requested load, it adaptively offloads some computations to memory and, when necessary, uses the CPU for computation, reducing the communication overhead caused by constant model switching. This reduces the additional communication overhead caused by switching between multiple models and maximizes the computing power of the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0012] Figure 1 A flowchart of a method for allocating video memory according to an embodiment of the present disclosure is shown;

[0013] Figure 2 Another flow chart of a method for allocating video memory according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram of a video memory allocation device according to an embodiment of the present disclosure is shown;

[0015] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown;

[0016] Figure 5 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0017] The present disclosure will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0018] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] Figure 1The method for allocating video memory of the present disclosure is shown. In an embodiment of the present disclosure, the method includes:

[0020] Step S101 : detecting an actual first video memory parameter of the video memory in the server during operation, wherein the server is deployed with multiple models.

[0021] In this embodiment, the execution subject of the video memory allocation method (such as the server) can detect the first video memory parameter of the video memory in the server during the operation of the server.

[0022] The first video memory parameter is the video memory parameter used in some scenarios and situations on the server. This parameter can be various parameters, such as memory consumption and computation time on the CPU and GPU. The memory consumption is the amount of space occupied by the video memory.

[0023] Models can be various formulas, functions, or deep neural networks. For example, they can be language models.

[0024] Step S102: predicting second video memory parameters for the video memory according to the first video memory parameters.

[0025] In this embodiment, the execution entity may predict the second video memory parameter based on the first video memory parameter in various ways. For example, the execution entity may input the first video memory parameter into a preset video memory parameter prediction model or formula to obtain the second video memory parameter.

[0026] The second video memory parameter is a video memory parameter of the server in other scenarios and situations other than the above scenarios and situations.

[0027] Step S103: predicting user request information of the target user in a future preset time period for the server.

[0028] In this embodiment, the execution entity may use various methods to predict user request information for the target user in a preset future time period for the server. For example, the execution entity may predict the target user's user request information based on the target user's request sending cycle. User request information is the request sent by the user.

[0029] Step S104 : Allocating video memory for the future preset time period according to the first video memory parameter, the second video memory parameter, and user request information.

[0030] In this embodiment, the execution entity may use various methods to allocate video memory for the future preset time period based on the video memory operating parameters, the video memory operating prediction parameters, and the user request information. For example, the execution entity may input the video memory operating parameters, the video memory operating prediction parameters, and the user request information into a video memory allocation model to obtain a video memory allocation method output by the model.

[0031] For example, video memory allocation may include the following three allocation methods: calculation on the CPU, calculation on the GPU without using a KV cache (key-value cache), and calculation on the GPU with using a KV cache.

[0032] This embodiment addresses the resource demand conflicts between multiple models by exploiting underutilized memory and CPU computing power to more compactly deploy multiple models on a single computing device, enabling the coexistence and concurrency of multiple models. Based on the requested load, it adaptively offloads some computations to memory and, when necessary, uses the CPU to perform computations, reducing the communication overhead associated with constant model switching. This reduces the additional communication overhead caused by switching between multiple models and maximizes the computing power of the device.

[0033] In some optional implementations of any embodiment of the present disclosure, the actual first video memory parameters of the video memory in the server are detected during operation, including: for the server, the models to be loaded in each layer constitute a computing unit; for each computing unit, the actual first video memory parameters are detected during operation.

[0034] In these implementations, the execution entity can divide the loaded models into layers and organize them into computational units. That is, each layer can correspond to a computational unit. For example, a computational unit can be a nn.module in PyTorch.

[0035] In some optional implementations of any one of the embodiments of the present disclosure, the first video memory parameter is a video memory parameter for the current configuration information of the server, and the second video memory parameter is a video memory parameter for configuration information other than the current configuration information; predicting the second video memory parameter for the video memory based on the first video memory parameter includes: using a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory.

[0036] Optionally, the second video memory parameter includes possible video memory consumption other than the video memory consumption corresponding to the current configuration information of the server; the use of a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: determining the possible video memory consumption based on the size of the parameters in the calculation unit, the size of the batch processing, the highest video memory peak of the intermediate results of the model to be loaded, and the space occupied by the KV cache.

[0037] Optionally, the second video memory parameter includes possible communication overhead other than the communication overhead corresponding to the current configuration information of the server; the use of a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: fitting the first video memory parameter with a linear model, and determining the coefficients of the parameters in the possible communication overhead through the fitting results; determining the possible communication overhead based on the fixed overhead required for each transmission startup and the corresponding coefficient, the amount of data transmitted and the corresponding coefficient.

[0038] Optionally, the second video memory parameter includes possible calculation time other than the calculation time corresponding to the current configuration information of the server; the use of a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: performing interpolation calculations through a first interpolation function and a second interpolation function to obtain a calculation result, the first interpolation function uses a batch size as a variable, the second interpolation function uses a request sequence length of the input model and a request sequence length output from the model as variables, the interpolation calculation of the first interpolation function uses at least one first video memory parameter, and the difference calculation of the second interpolation function uses at least one second video memory parameter; according to the calculation result, the possible calculation time is determined.

[0039] Specifically, the first video memory parameter may include:

[0040] (1) Memory usage: This includes the storage space required for the computational unit's parameters, the peak memory usage of intermediate results, and the size of the KV cache. The peak memory usage R is the peak memory usage of the computational unit during the computation.

[0041] (2) Transmission time of unit data: This includes the time for bidirectional copying between the CPU and GPU. The data transmitted here can be user input, model parameters, or calculation results.

[0042] (3) Calculation time: Based on different batch sizes and whether there is an available KV cache, collect data profiling to obtain the required time consumption.

[0043] During the profiling process, it is not possible to traverse all possible situations. Therefore, for specific computing requests, it is necessary to estimate the secondary memory parameters. In other words, estimate the possible memory consumption, possible communication overhead, and possible computing time under other configurations.

[0044] The possible video memory consumption is estimated using the following formula:

[0045] S=M+b(R+K)

[0046] Where M is the parameter size of the computing unit, which can be calculated as the number of parameters * the space occupied by each parameter. b is the batch size. R and K are the space occupied by the calculation results and the KV cache, respectively. After the server receives a request, the upper-layer scheduler decides whether to aggregate the requests for calculation. The batch size refers to the number of requests to be aggregated.

[0047] The possible communication overhead, i.e., the transmission time per unit time, can be estimated using the following formula:

[0048] T=α·T0+β·D

[0049] Here, T0 is assumed to be the fixed overhead required for each transmission startup, D is the amount of data transmitted, and α and β are coefficients fitted according to the profiling results.

[0050] The calculation time may be estimated using the following formula:

[0051] C=f1(b)·f2(L i ,L o )

[0052] Where b is the batch size, Li and Lo represent the input sequence length and the expected output sequence length, respectively. Li and Lo exist for each request of each model. f1 and f2 are both interpolation functions, f1 is the first interpolation function, and f2 is the second interpolation function.

[0053] Use interpolation methods at the profiling points to obtain the value that should correspond to the specific input value.

[0054] In some optional implementations of any embodiment of the present disclosure, predicting user request information in a future preset time period for the server includes: receiving first user request information from a target terminal; calling a trained request prediction model to process the first user request information to predict second user request information of the target terminal in a future preset time period, and the request prediction model contains model parameters related to the time dependency between user request information.

[0055] The training step may include: using historical request data to train the request prediction model to adjust the model parameters of the request prediction model, the historical request data includes the historical first request information received earlier and the historical second request information received later, and the model parameters are related to the time dependency between the historical first request information and the historical second request information.

[0056] The goal is to adaptively offload some computations to memory based on the request load, and to use the CPU for computation when necessary to reduce the communication overhead caused by constant model switching. To achieve this, it is necessary to know the request list within a certain time window. This list can be obtained through the request scheduler. In addition, this disclosure also designs an optional module for predicting future requests, thereby providing richer request information.

[0057] The steps to build a request predictor are as follows:

[0058] (1) Data collection: record historical request data into the database;

[0059] (2) Model selection: Select a prediction model, such as the long short-term memory network (LSTM);

[0060] (3) Model training: Use historical data to train the model and adjust parameters to capture the temporal dependencies and changing patterns between computing requests;

[0061] (4) Model evaluation: Use the trained model to make the requested predictions.

[0062] In some optional implementations of any embodiment of the present disclosure, the first video memory parameter and the second video memory parameter both include calculation time and inter-device transmission time; allocating video memory for the future preset time period based on the first video memory parameter, the second video memory parameter and user request information includes: allocating video memory for the future preset time period with the goal of minimizing the product of the inverse of the number of computing units in the server and a target sum, the target sum being the sum of the sum of the calculation time in the video memory parameters and the sum of the inter-device transmission time.

[0063] In some optional implementations of any embodiment of the present disclosure, the constraint conditions of the target include at least one of the following: each computing unit corresponds to only one allocation strategy, the sum of the video memory requirements of each computing unit under the corresponding allocation strategy is not greater than the video memory capacity of the video memory, the sum of the calculation completion time of the target request and the inter-device transmission time of the computing unit pair of the target request is less than a preset timeout time, and the computing unit pair is a computing unit with a dependent relationship.

[0064] The goal of memory planning is to determine whether to place a compute unit on the GPU for execution. If computing on the GPU, it also determines whether to allocate a KV cache. In other words, a compute unit has three possible states: ① computing on the CPU, ② computing on the GPU but not using the KV cache, and ③ computing on the GPU with the KV cache.

[0065] Memory planning can be transformed into an optimization problem. The goal is to minimize the average completion time for all user requests within a certain time window without starving them. The specific steps are as follows:

[0066] 1) There are n computing units, and a decision variable x is defined for each computing unit. i , where x i It can take values ​​{1,2,3}, representing three possible states (memory allocation strategies).

[0067] 2) Let C i (x i ) is the calculation unit in x i The completion time in the state, the objective function can be expressed as:

[0068]

[0069] 3) Add the following constraints: a) Each computation unit can only choose one state, that is, x i Can only take

[0070] One of {1, 2, 3}; b) For computational units i and j that have dependencies (e.g., computational unit j needs to input the result of computational unit i), if they are on different devices, there will be additional transmission time, i.e., the inter-device transmission time T i,j , the sum of the target request's computation completion time and the target request's inter-device transmission time is less than the preset timeout threshold; c) Computing unit i will have a specific video memory requirement S according to the selected state i (x i ), requiring all activated computing units to correspond to S i (x i ) is not greater than the video memory capacity.

[0071] 4) The final optimization objective function is:

[0072]

[0073] Among them, I(x i ,x j ) is the characteristic function, when x i and x jWhen in the same device, the value is 1 (such as

[0074] x i =1,x j =2 or 3), otherwise it takes the value 0 (such as x i =2 or 3, x j =2 or 3).

[0075] The constraints are:

[0076]

[0077] ∑S i (x i )≤Video memory capacity

[0078] All activated computational units i

[0079] ΣC i (x i )+ΣT i,j ·I(x i ,x j )≤preset timeout All computing units i involved in a specific computing request's dependency pair (i, j)

[0080] A specific compute request is a target request. Through the above steps, memory planning can be converted into an integer programming problem, which can then be solved using branch-and-bound or plane methods. A dependency pair is a pair of compute units that have a dependency relationship, for example, compute unit A requires the output of compute unit B.

[0081] Figure 2 A method for allocating video memory according to an embodiment of the present disclosure is shown.

[0082] The embodiment of the present disclosure provides a video memory allocation device, which is used to execute the video memory allocation method described in the above embodiment, such as Figure 3 As shown, the device includes: a detection unit 301, configured to detect an actual first video memory parameter of the video memory in the server during operation, and the server is deployed with multiple models; a parameter prediction unit 302, configured to predict a second video memory parameter for the video memory based on the first video memory parameter; a request prediction unit 303, configured to predict user request information of a target user in a future preset time period for the server; and an allocation unit 304, configured to allocate video memory for the future preset time period based on the first video memory parameter, the second video memory parameter and the user request information.

[0083] The video memory allocation device provided in the above-mentioned embodiment of the present disclosure and the video memory allocation method provided in the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0084] The embodiment of the present disclosure further provides an electronic device corresponding to the video memory allocation method provided in the above embodiment, so as to execute the above video memory allocation method.

[0085] Please refer to Figure 4 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Figure 4 As shown, the electronic device 40 includes: a processor 400, a memory 401, a bus 402 and a communication interface 403, and the processor 400, the communication interface 403 and the memory 401 are connected via the bus 402; the memory 401 stores a computer program that can be run on the processor 400, and when the processor 400 runs the computer program, it executes the method provided in any of the aforementioned embodiments of the present disclosure.

[0086] Memory 401 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. Communication between the system network element and at least one other network element is achieved through at least one communication interface 403 (which may be wired or wireless), and may use the Internet, a wide area network, a local area network, a metropolitan area network, etc.

[0087] Bus 402 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. Memory 401 is used to store programs, and processor 400 executes the programs upon receiving execution instructions. The video memory allocation method disclosed in any of the aforementioned embodiments of the present disclosure may be applied to processor 400 or implemented by processor 400.

[0088] The processor 400 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method may be completed by hardware integrated logic circuits or software instructions in the processor 400. The processor 400 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 401 , and the processor 400 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.

[0089] The electronic device provided by the embodiment of the present disclosure and the method for allocating video memory provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.

[0090] The present disclosure also provides a computer-readable storage medium corresponding to the method for allocating video memory provided in the above embodiment. Figure 5 The computer-readable storage medium shown is a CD 50 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method for allocating video memory provided by any of the aforementioned embodiments is executed.

[0091] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0092] The computer-readable storage medium provided by the above-mentioned embodiment of the present disclosure and the method for allocating video memory provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0093] It should be noted that:

[0094] In the above text, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0095] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0096] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, which are only specific implementation methods of the present disclosure. However, the present disclosure is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present disclosure and the claims, which are all within the protection of the present disclosure.

Claims

1. A method for allocating video memory, characterized in that: For the server, including: For a video memory in the server, detecting an actual first video memory parameter during operation, wherein the server is deployed with multiple models; predicting a second video memory parameter for the video memory according to the first video memory parameter; For the server, predicting user request information of the target user in a future preset time period; Allocating video memory for the future preset time period according to the first video memory parameter, the second video memory parameter, and user request information; The first video memory parameter is a video memory parameter for the current configuration information of the server, and the second video memory parameter is a video memory parameter for configuration information other than the current configuration information; The predicting a second video memory parameter for the video memory according to the first video memory parameter includes: Using a preset formula to process the first video memory parameter to obtain a second video memory parameter of the video memory; The first video memory parameter and the second video memory parameter both include calculation time and inter-device transmission time; The allocating video memory for the future preset time period according to the first video memory parameter, the second video memory parameter and the user request information includes: The video memory allocation for the future preset time period is performed with the goal of minimizing the product of the inverse of the number of computing units in the server and the target sum, where the target sum is the sum of the sum of the calculation time in the video memory parameters and the sum of the transmission time between devices.

2. The method according to claim 1, characterized in that The detecting of an actual first video memory parameter of the video memory in the server during operation includes: For the server, the models to be loaded in each layer constitute a computing unit; For each computing unit, the actual first video memory parameter during the running process is detected.

3. The method according to claim 1, characterized in that The second video memory parameter includes video memory consumption other than video memory consumption corresponding to current configuration information of the server; The using a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: The video memory consumption is determined according to the size of the parameters in the computing unit, the size of the batch processing, the highest video memory peak value of the intermediate results of the model to be loaded, and the space occupied by the key-value KV cache.

4. The method according to claim 1, wherein The second video memory parameter includes a communication overhead other than a communication overhead corresponding to the current configuration information of the server; The using a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: Fitting the first video memory parameters using a linear model, and determining coefficients of parameters in the communication overhead based on the fitting results; The communication overhead is determined according to the fixed overhead required for each transmission startup and the corresponding coefficient, the amount of data transmitted and the corresponding coefficient.

5. The method according to claim 1, wherein The second video memory parameter includes a calculation time other than a calculation time corresponding to the current configuration information of the server; The using a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: performing interpolation calculations using a first interpolation function and a second interpolation function to obtain a calculation result, wherein the first interpolation function uses a batch size as a variable, and the second interpolation function uses a request sequence length input to the model and a request sequence length output from the model as variables, the interpolation calculation of the first interpolation function uses at least one first memory parameter, and the difference calculation of the second interpolation function uses at least one second memory parameter; The calculation time consumption is determined according to the calculation result.

6. The method according to claim 1, wherein The predicting of user request information in a future preset time period for the server includes: receiving first user request information from a target terminal; The trained request prediction model is called to process the first user request information to predict the second user request information of the target terminal in a future preset time period, and the request prediction model contains model parameters related to the time dependency between user request information.

7. The method according to claim 1, characterized in that The constraints of the target include at least one of the following: Each computing unit corresponds to only one allocation strategy, the sum of the video memory requirements of each computing unit under the corresponding allocation strategy is not greater than the video memory capacity of the video memory, the sum of the calculation completion time of the target request and the inter-device transmission time of the computing unit pair of the target request is less than the preset timeout time, and the computing unit pair is a computing unit with a dependent relationship.

8. A video memory allocation device, characterized in that: include: a detection unit configured to detect an actual first video memory parameter of a video memory in a server during operation, wherein the server is deployed with a plurality of models; a parameter prediction unit, configured to predict a second video memory parameter for the video memory based on the first video memory parameter; a request prediction unit configured to predict, for the server, user request information of a target user in a future preset time period; an allocation unit configured to allocate video memory for the future preset time period according to the first video memory parameter, the second video memory parameter and user request information; The first video memory parameter is a video memory parameter for the current configuration information of the server, and the second video memory parameter is a video memory parameter for configuration information other than the current configuration information; The parameter prediction unit is further configured to perform the step of predicting the second video memory parameter for the video memory based on the first video memory parameter in the following manner: Using a preset formula to process the first video memory parameter to obtain a second video memory parameter of the video memory; The first video memory parameter and the second video memory parameter both include calculation time and inter-device transmission time; The allocation unit is further configured to perform the video memory allocation for the future preset time period according to the first video memory parameter, the second video memory parameter, and the user request information in the following manner: The video memory allocation for the future preset time period is performed with the goal of minimizing the product of the inverse of the number of computing units in the server and the target sum, where the target sum is the sum of the sum of the calculation time in the video memory parameters and the sum of the transmission time between devices.

Images