Video memory allocation method and device

Abstract

The invention provides a video memory allocation method and device, and relates to the technical field of communication. The method comprises the following steps: for a video memory in a server, detecting an actual first video memory parameter in an operation process, and deploying a plurality of models in the server; predicting a second video memory parameter for the video memory according to the first video memory parameter; predicting user request information of a target user in a future preset time period for the server; and according to the first video memory parameter, the second video memory parameter and user request information, performing video memory allocation for the future preset time period. According to the method and the device, aiming at resource demand conflicts of multiple models, the multiple models can be more compactly deployed on one computing device by developing the memory which is not fully used and the computing power of the CPU, so that coexistence and concurrency of the multiple models are realized.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular to a method and device for allocating video memory. Background Art

[0002] Video memory is the graphics card memory, also known as frame buffer, which is used to store rendering data that has been processed or is about to be extracted by the graphics chip. Like the computer's memory, video memory is a component used to store graphics information to be processed.

[0003] The capacity of the video memory on the server is limited and often needs to be time-shared. How to make full use of the video memory on the server according to user requests. Summary of the invention

[0004] In view of this, the purpose of the present disclosure is to provide a method and device for allocating video memory, which can specifically solve the existing problems.

[0005] Based on the above purpose, in a first aspect, the present disclosure proposes a method for allocating video memory, comprising: for the video memory in the server, detecting an actual first video memory parameter during operation, wherein the server is deployed with multiple models; predicting a second video memory parameter for the video memory based on the first video memory parameter; for the server, predicting user request information of a target user in a future preset time period; and allocating video memory for the future preset time period based on the first video memory parameter, the second video memory parameter and the user request information.

[0006] In a second aspect, a device for allocating video memory is also provided, comprising: a detection unit, configured to detect an actual first video memory parameter of the video memory in the server during operation, wherein the server is deployed with multiple models; a parameter prediction unit, configured to predict a second video memory parameter for the video memory based on the first video memory parameter; a request prediction unit, configured to predict user request information of a target user in a future preset time period for the server; and an allocation unit, configured to allocate video memory for the future preset time period based on the first video memory parameter, the second video memory parameter and the user request information.

[0007] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0008] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program is executed by a processor to implement any method described in the first aspect.

[0009] In a fifth aspect, a computer program product is also provided, including a computer program, wherein the computer program is executed by a processor to implement any method described in the first aspect.

[0010] In general, the present disclosure has at least the following beneficial effects: it can address resource demand conflicts among multiple models, develop memory and CPU computing capabilities that have not been fully utilized, and deploy multiple models more compactly on a computing device to achieve coexistence and concurrency of multiple models. And according to the load of the request, it can adaptively offload part of the calculation to the memory, and use the CPU for calculation when necessary to reduce the communication overhead caused by the continuous switching of models, reduce the additional communication overhead caused by the switching of multiple models, and tap the computing power of the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in accordance with the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0012] Figure 1 A flowchart of a method for allocating video memory according to an embodiment of the present disclosure is shown;

[0013] Figure 2 Another flow chart of a method for allocating video memory according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram of a video memory allocation device according to an embodiment of the present disclosure is shown;

[0015] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown;

[0016] Figure 5 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0017] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0018] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] Figure 1The method for allocating video memory of the present disclosure is shown. In an embodiment of the present disclosure, the method includes:

[0020] Step S101, detecting an actual first video memory parameter of the video memory in the server during operation, wherein the server is deployed with multiple models.

[0021] In this embodiment, the execution subject of the video memory allocation method (such as the server) can detect the first video memory parameter of the video memory in the server during the operation of the server.

[0022] The first video memory parameter is the video memory parameter in some scenarios and situations on the server. The parameters here can be various parameters, such as the storage consumption and computing time on the CPU and GPU. The storage consumption is the size of the space occupied by the video memory.

[0023] The model can be any formula, function or deep neural network. For example, it can be a language model.

[0024] Step S102: predicting a second video memory parameter for the video memory according to the first video memory parameter.

[0025] In this embodiment, the execution subject may predict the second video memory parameter based on the first video memory parameter in various ways. For example, the execution subject may input the first video memory parameter into a preset video memory parameter prediction model or formula to obtain the second video memory parameter.

[0026] The second video memory parameter is a video memory parameter of the server in other scenarios and situations other than the above scenarios and situations.

[0027] Step S103: predicting, for the server, user request information of the target user in a future preset time period.

[0028] In this embodiment, the execution subject may predict the user request information of the target user in a preset time period in the future for the server in various ways. For example, the execution subject may predict the user request information of the target user according to the cycle of the target user sending the request. The user request information is the request sent by the user.

[0029] Step S104: allocating video memory for the future preset time period according to the first video memory parameter, the second video memory parameter and user request information.

[0030] In this embodiment, the execution subject may use various methods to perform video memory allocation for the future preset time period according to the video memory operation parameters, video memory operation prediction parameters and user request information. For example, the execution subject may input the video memory operation parameters, video memory operation prediction parameters and user request information into a video memory allocation model, thereby obtaining a video memory allocation method output by the model.

[0031] For example, video memory allocation may include the following three allocation methods: calculation on the CPU, calculation on the GPU without using a KV cache (key-value cache), and calculation on the GPU with using a KV cache.

[0032] This embodiment can address the resource demand conflicts of multiple models by developing the memory and CPU computing power that have not been fully used, so that multiple models can be deployed more compactly on a computing device to achieve the coexistence and concurrency of multiple models. According to the load of the request, part of the calculation is adaptively unloaded to the memory, and the CPU is used for calculation when necessary to reduce the communication overhead caused by the continuous switching of models, thereby reducing the additional communication overhead caused by the switching of multiple models and tapping the computing power of the device.

[0033] In some optional implementations of any embodiment of the present disclosure, the actual first video memory parameters of the video memory in the server are detected during operation, including: for the server, the models to be loaded in each layer constitute a computing unit; for each computing unit, the actual first video memory parameters are detected during operation.

[0034] In these implementations, the execution subject can divide the loaded models into layers to form computing units. That is, each layer can correspond to a computing unit. For example, a computing unit can be a nn.module in pytorch.

[0035] In some optional implementations of any one of the embodiments of the present disclosure, the first video memory parameter is a video memory parameter for current configuration information of the server, and the second video memory parameter is a video memory parameter for configuration information other than the current configuration information; predicting the second video memory parameter for the video memory based on the first video memory parameter includes: using a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory.

[0036] Optionally, the second video memory parameter includes possible video memory consumption other than the video memory consumption corresponding to the current configuration information of the server; and the using of a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: determining the possible video memory consumption based on the size of the parameters in the calculation unit, the size of the batch processing, the highest video memory peak of the intermediate results of the model to be loaded, and the space occupied by the KV cache.

[0037] Optionally, the second video memory parameter includes possible communication overhead other than the communication overhead corresponding to the current configuration information of the server; the use of a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: fitting the first video memory parameter with a linear model, and determining the coefficients of the parameters in the possible communication overhead through the fitting results; determining the possible communication overhead based on the fixed overhead required for each transmission startup and the corresponding coefficient, the amount of data transmitted and the corresponding coefficient.

[0038] Optionally, the second video memory parameter includes possible calculation time other than the calculation time corresponding to the current configuration information of the server; the use of a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: performing interpolation calculations through a first interpolation function and a second interpolation function to obtain a calculation result, the first interpolation function uses a batch size as a variable, the second interpolation function uses a request sequence length of an input model and a request sequence length output from the model as variables, the interpolation calculation of the first interpolation function uses at least one first video memory parameter, and the difference calculation of the second interpolation function uses at least one second video memory parameter; according to the calculation result, determining the possible calculation time.

[0039] Specifically, the first video memory parameter may include:

[0040] (1) Video memory consumption: includes the storage space required for the parameters of the computing unit, the highest video memory peak value of the intermediate results, and the space size of the KV cache. The highest video memory peak value R is the peak value of the video memory used by the computing unit during the calculation process.

[0041] (2) Transmission time of unit data: This includes the time for bidirectional copying between the CPU and GPU. The data transmitted here can be user input, model parameters, or calculation results.

[0042] (3) Calculation time: Based on the batch size and whether the KV cache is available, collect data profiling to obtain the time required.

[0043] During the profiling process, it is not possible to traverse all possible situations. Therefore, for specific computing requests, it is necessary to estimate the second video memory parameters, that is, to estimate the possible video memory consumption, possible communication overhead, and possible computing time under other configurations.

[0044] The possible video memory consumption is estimated using the following formula:

[0045] S=M+b(R+K)

[0046] Among them, M is the parameter size of the computing unit, which can be the number of parameters * the space occupied by each parameter, b is the batch size, R and K are the space occupied by the calculation results and KV cache respectively. After the server receives the request, the upper-layer scheduling will decide whether to aggregate them together for calculation. The batch size refers to the number of aggregated requests.

[0047] The possible communication overhead, i.e. the transmission time per unit time, can be estimated using the following formula:

[0048] T=α·T0+β·D

[0049] Among them, T0 is assumed to be the fixed overhead required for each transmission startup, D is the amount of data transmitted, and α and β are coefficients after fitting according to the profiling results.

[0050] The possible calculation time can be estimated using the following formula:

[0051] C=f1(b)·f2(L i ,L o )

[0052] Where b is the batch size, Li and Lo represent the input sequence length and the expected output sequence length respectively, each request of each model has Li and Lo, f1 and f2 are both interpolation functions, f1 is the first interpolation function, and f2 is the second interpolation function.

[0053] Use interpolation methods on the already profiled points to get the value that the specific input value should correspond to.

[0054] In some optional implementations of any embodiment of the present disclosure, predicting user request information in a future preset time period for the server includes: receiving first user request information from a target terminal; calling a trained request prediction model to process the first user request information to predict second user request information of the target terminal in a future preset time period, wherein the request prediction model contains model parameters related to the time dependency between user request information.

[0055] The training step may include: using historical request data to train the request prediction model to adjust the model parameters of the request prediction model, the historical request data including historical first request information received earlier and historical second request information received later, and the model parameters are related to the time dependency between the historical first request information and the historical second request information.

[0056] The goal is to adaptively offload part of the calculation to the memory according to the load of the request, and use the CPU for calculation when necessary to reduce the communication overhead caused by the constant switching of the model. To this end, it is necessary to know the list of requests within a certain time window. This list can be obtained through the request scheduler. In addition, the present disclosure also designs an optional module for predicting future requests to provide richer request information.

[0057] The steps to construct a request predictor are as follows:

[0058] (1) Data collection: record historical request data into the database;

[0059] (2) Model selection: Select a prediction model, such as the long short-term memory network (LSTM);

[0060] (3) Model training: Use historical data to train the model and adjust parameters to capture the time dependencies and change patterns between computing requests;

[0061] (4) Model evaluation: Use the trained model to make requested predictions.

[0062] In some optional implementations of any one of the embodiments of the present disclosure, the first video memory parameter and the second video memory parameter both include calculation time and inter-device transmission time; allocating video memory for the future preset time period based on the first video memory parameter, the second video memory parameter and user request information includes: allocating video memory for the future preset time period with the goal of minimizing the product of the inverse of the number of computing units in the server and a target sum, the target sum being the sum of the sum of the calculation time in the video memory parameters and the sum of the transmission time between devices.

[0063] In some optional implementations of any embodiment of the present disclosure, the constraint conditions of the target include at least one of the following: each computing unit corresponds to only one allocation strategy, the sum of the video memory requirements of each computing unit under the corresponding allocation strategy is not greater than the video memory capacity of the video memory, the sum of the calculation completion time of the target request and the inter-device transmission time of the computing unit pair of the target request is less than a preset timeout time, and the computing unit pair is a computing unit with a dependent relationship.

[0064] The goal of video memory planning is to determine whether to place the computing unit on the GPU for execution. If computing on the GPU, it further determines whether to allocate KV cache. In other words, a computing unit has three optional states: ① computing on the CPU, ② computing on the GPU but not using KV cache, and ③ computing on the GPU and using KV cache.

[0065] Video memory planning can be transformed into an optimization problem. The optimization goal is to achieve the minimum average completion time for all user requests within a certain time window without being "starved to death". The specific steps are as follows:

[0066] 1) Suppose there are n computing units, and define a decision variable x for each computing unit i , where x i The value can be {1,2,3}, representing 3 possible states (video memory allocation strategies).

[0067] 2) Let C i (x i ) is the calculation unit in x i The completion time under the state, the objective function can be expressed as:

[0068]

[0069] 3) Add the following constraints: a) Each computation unit can only choose one state, namely x i Can only take

[0070] {1,2,3}; b) For computing units i and j that have dependencies (e.g., computing unit j needs to input the result of computing unit i), if they are on different devices, there will be additional transmission time, that is, the inter-device transmission time T i,j , the sum of the target request's computation completion time and the target request's inter-device transmission time of the computing unit is less than the preset timeout threshold; c) computing unit i will have a specific video memory requirement S according to the selected state i (x i ), requiring all activated computing units to correspond to S i (x i ) is not greater than the video memory capacity.

[0071] 4) The final optimization objective function is:

[0072]

[0073] Among them, I(x i ,x j ) is an indicative function. When x i and x jWhen in the same device, the value is 1 (such as

[0074] x i =1,x j =2 or 3), otherwise it takes the value 0 (such as x i =2 or 3, x j =2 or 3).

[0075] The constraints are:

[0076]

[0077] ∑S i (x i )≤Video memory capacity

[0078] All activated computational units i

[0079] Σ C i (x i )+ Σ T i,j ·I(x i ,x j )≤ the preset timeout time for all computing units i of a specific computing request involving dependency pairs (i, j)

[0080] A specific computing request is a target request. Through the above steps, the memory planning can be converted into an integer programming problem, and then solved using the branch and bound method or the plane method. A dependent pair is a computing unit pair that is a computing unit with a dependent relationship, such as computing unit A needs the output of computing unit B.

[0081] Figure 2 A method for allocating video memory according to an embodiment of the present disclosure is shown.

[0082] The embodiment of the present disclosure provides a video memory allocation device, which is used to execute the video memory allocation method described in the above embodiment, such as Figure 3 As shown, the device includes: a detection unit 301, configured to detect an actual first video memory parameter of the video memory in the server during operation, and the server is deployed with multiple models; a parameter prediction unit 302, configured to predict a second video memory parameter for the video memory according to the first video memory parameter; a request prediction unit 303, configured to predict user request information of a target user in a future preset time period for the server; an allocation unit 304, configured to allocate video memory for the future preset time period according to the first video memory parameter, the second video memory parameter and the user request information.

[0083] The video memory allocation device provided in the above-mentioned embodiment of the present disclosure and the video memory allocation method provided in the embodiment of the present disclosure are based on the same inventive concept, and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0084] The embodiment of the present disclosure also provides an electronic device corresponding to the video memory allocation method provided in the above embodiment, so as to execute the above video memory allocation method. The embodiment of the present disclosure is not limited.

[0085] Please refer to Figure 4 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Figure 4 As shown, the electronic device 40 includes: a processor 400, a memory 401, a bus 402 and a communication interface 403, and the processor 400, the communication interface 403 and the memory 401 are connected via the bus 402; the memory 401 stores a computer program that can be run on the processor 400, and the processor 400 executes the method provided in any of the aforementioned embodiments of the present disclosure when running the computer program.

[0086] The memory 401 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 403 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0087] The bus 402 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 401 is used to store programs, and the processor 400 executes the programs after receiving execution instructions. The video memory allocation method disclosed in any implementation of the aforementioned embodiment of the present disclosure may be applied to the processor 400, or implemented by the processor 400.

[0088] The processor 400 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 400 or the instruction in the form of software. The above processor 400 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 401, and the processor 400 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.

[0089] The electronic device provided by the embodiment of the present disclosure and the method for allocating video memory provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.

[0090] The present disclosure also provides a computer-readable storage medium corresponding to the video memory allocation method provided in the above embodiment. Figure 5 The computer-readable storage medium shown is a CD 50 on which a computer program (ie, a program product) is stored. When the computer program is executed by a processor, the method for allocating video memory provided in any of the aforementioned embodiments will be executed.

[0091] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0092] The computer-readable storage medium provided by the above-mentioned embodiment of the present disclosure and the method for allocating video memory provided by the embodiment of the present disclosure are based on the same inventive concept, and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0093] It should be noted that:

[0094] In the above text, the term "comprises", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0096] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, which are only specific implementation modes of the present disclosure. However, the present disclosure is not limited to the above-mentioned specific implementation modes. The above-mentioned specific implementation modes are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the purpose of the present disclosure and the scope of protection of the claims, all of which are within the protection of the present disclosure.

Claims

1. A method for allocating video memory, characterized in that: For the server, including: For the video memory in the server, detecting an actual first video memory parameter during operation, wherein the server is deployed with a plurality of models; Predicting a second video memory parameter for the video memory according to the first video memory parameter; For the server, predicting user request information of the target user in a future preset time period; The video memory for the future preset time period is allocated according to the first video memory parameter, the second video memory parameter and the user request information.

2. The method according to claim 1, characterized in that The detecting of an actual first video memory parameter of the video memory in the server during operation includes: For the server, the models to be loaded in each layer constitute a computing unit; For each computing unit, the actual first video memory parameter during the running process is detected.

3. The method according to claim 2, characterized in that The first video memory parameter is a video memory parameter for the current configuration information of the server, and the second video memory parameter is a video memory parameter for other configuration information other than the current configuration information; The predicting a second video memory parameter for the video memory according to the first video memory parameter includes: The first video memory parameter is processed by using a preset formula to obtain a second video memory parameter of the video memory.

4. The method according to claim 3, characterized in that The second video memory parameter includes possible video memory consumption other than the video memory consumption corresponding to the current configuration information of the server; The using a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: The possible video memory consumption is determined according to the size of the parameters in the computing unit, the size of the batch processing, the highest video memory peak value of the intermediate results of the model to be loaded, and the space occupied by the key-value KV cache.

5. The method according to claim 3, characterized in that: The second video memory parameter includes possible communication overhead other than the communication overhead corresponding to the current configuration information of the server; The using a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: Fitting the first video memory parameter using a linear model, and determining coefficients of parameters in the possible communication overhead according to the fitting results; The possible communication overhead is determined according to the fixed overhead required for each transmission startup and the corresponding coefficient, the amount of data transmitted and the corresponding coefficient.

6. The method according to claim 3, characterized in that The second video memory parameter includes possible calculation time other than the calculation time corresponding to the current configuration information of the server; The using a preset formula to process the first video memory parameter to obtain the second video memory parameter of the video memory includes: Performing interpolation calculations through a first interpolation function and a second interpolation function to obtain a calculation result, wherein the first interpolation function takes a batch size as a variable, and the second interpolation function takes a request sequence length input to the model and a request sequence length output from the model as variables, the interpolation calculation of the first interpolation function uses at least one first video memory parameter, and the difference calculation of the second interpolation function uses at least one second video memory parameter; The possible calculation time consumption is determined according to the calculation result.

7. The method according to claim 1, characterized in that The predicting of user request information in a future preset time period for the server includes: Receiving first user request information from a target terminal; The trained request prediction model is called to process the first user request information to predict the second user request information of the target terminal in a future preset time period, wherein the request prediction model contains model parameters related to the time dependency between user request information.

8. The method according to claim 1, characterized in that The first video memory parameter and the second video memory parameter both include calculation time and inter-device transmission time; The allocating video memory for the future preset time period according to the first video memory parameter, the second video memory parameter and the user request information includes: The video memory allocation for the future preset time period is performed with the goal of minimizing the product of the inverse of the number of computing units in the server and the target sum, where the target sum is the sum of the sum of the calculation time in the video memory parameters and the sum of the transmission time between devices.

9. The method according to claim 8, characterized in that The constraint conditions of the target include at least one of the following: Each computing unit corresponds to only one allocation strategy, the sum of the video memory requirements of each computing unit under the corresponding allocation strategy is not greater than the video memory capacity of the video memory, the sum of the calculation completion time of the target request and the inter-device transmission time of the computing unit pair of the target request is less than the preset timeout time, and the computing unit pair is a computing unit with a dependent relationship.

10. A video memory allocation device, characterized in that: include: A detection unit is configured to detect an actual first video memory parameter of the video memory in the server during operation, wherein the server is deployed with a plurality of models; a parameter prediction unit, configured to predict a second video memory parameter for the video memory according to the first video memory parameter; A request prediction unit, configured to predict, for the server, user request information of a target user in a future preset time period; The allocation unit is configured to allocate video memory for the future preset time period according to the first video memory parameter, the second video memory parameter and user request information.

Images