Video memory management method and device, equipment and storage medium

Abstract

The invention discloses a video memory management method and device, equipment and a storage medium, and relates to the technical field of memory management, and the method comprises the steps: carrying out the statistics of a plurality of video memory applications of a target pre-training model in a model reasoning process through a preset recording function, so as to determine the maximum video memory consumption; determining an available video memory of the target pre-training model based on the maximum video memory consumption, and determining a plurality of first video memory applications whose video memory demands are not greater than a preset video memory request threshold and a plurality of second video memory applications whose video memory demands are greater than the preset video memory request threshold, determining a plurality of first video memory allocation spaces corresponding to the plurality of first video memory applications and a plurality of second video memory allocation spaces corresponding to the plurality of second video memory applications through a preset video memory interval; therefore, frequent application and release of the memory can be reduced, memory fragments are avoided, and the reasoning performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of memory management technology, and in particular to a video memory management method, device, equipment and storage medium. Background Art

[0002] With the development of science and technology, pre-trained models are widely used in various industries, such as vLLM, TensorRT-LLM, etc. Among them, vLLM is widely used for its PagedAttention technology and continuous batch processing. In addition to pre-allocating kv cache, the current vLLM model relies on the pytorch framework for input and output management of other operators. It uses the memory management solution of pytorch. The memory management of pytorch will allocate a large memory according to the required memory, and then cut this large memory into blocks and return the required part. When new memory is required later, it will search from the current remaining part. If there is a memory of suitable size, it will be returned, otherwise a new memory block will be opened, and so on.

[0003] However, it is not easy to control the size of large memory blocks allocated by the existing technology. If the allocation is too large, it is easy to generate a lot of memory fragments, affecting the computing utilization. If the allocation is too small, frequent memory requests are required, which has a greater impact on performance. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a video memory management method, device, equipment and storage medium, which can reduce the frequent application and release of memory as much as possible, avoid memory fragmentation, and improve reasoning performance. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a video memory management method, comprising:

[0006] Using a preset recording function to count a number of video memory applications of the target pre-trained model during the model inference process, and determining a maximum video memory consumption according to the number of video memory applications;

[0007] Determine the available video memory corresponding to the target pre-trained model based on the maximum video memory consumption, and differentiate the plurality of video memory applications according to a preset video memory request threshold to obtain a plurality of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a plurality of second video memory applications whose video memory requirements are greater than the preset video memory request threshold;

[0008] Matching the first video memory requirements corresponding to the first video memory applications with a preset video memory interval to determine the first target video memory intervals corresponding to the first video memory requirements, and allocating space from the available video memory for the first video memory requirements corresponding to the first video memory applications based on the maximum values ​​of the intervals corresponding to the first target video memory intervals;

[0009] A plurality of second video memory requirements corresponding to the plurality of second video memory applications are determined, and video memory combinations are performed based on the preset video memory interval, so as to allocate space from the available video memory for matching the plurality of second video memory requirements with the plurality of second video memory requirements according to the video memory combinations.

[0010] Optionally, counting a number of video memory applications of the target pre-trained model in the model inference process by a preset recording function, and determining a maximum video memory consumption according to the number of video memory applications, includes:

[0011] Insert a preset recording function into the target pre-trained model, and use the preset recording function to count several video memory requests of the target pre-trained model during the model inference process;

[0012] A number of video memory requirements corresponding to the number of video memory applications are counted, and a maximum video memory consumption is determined based on the number of video memory requirements.

[0013] Optionally, determining the available video memory corresponding to the target pre-trained model based on the maximum video memory consumption, and distinguishing the plurality of video memory applications according to a preset video memory request threshold to obtain a plurality of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a plurality of second video memory applications whose video memory requirements are greater than the preset video memory request threshold, includes:

[0014] Determine the product of a preset total video memory and a preset usage coefficient, and calculate the difference between a value corresponding to the product and a value corresponding to the maximum video memory consumption, so as to use the obtained difference as the available video memory corresponding to the target pre-trained model;

[0015] Comparing the plurality of video memory requirements respectively corresponding to the plurality of video memory applications with a preset video memory request threshold to obtain a corresponding comparison result;

[0016] According to the comparison result, a plurality of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a plurality of second video memory applications whose video memory requirements are greater than the preset video memory request threshold are screened out from the plurality of video memory applications.

[0017] Optionally, matching the first video memory requirements corresponding to the first video memory applications with a preset video memory interval to determine the first target video memory intervals corresponding to the first video memory requirements, and allocating space from the available video memory to the first video memory corresponding to the first video memory applications based on the maximum values ​​of the intervals corresponding to the first target video memory intervals, includes:

[0018] Determining a number of first video memory requirements corresponding to the number of first video memory applications;

[0019] Matching a current first video memory requirement among the plurality of first video memory requirements with a preset video memory interval to determine a first target video memory interval matched by the current first video memory requirement;

[0020] Determine a maximum value of an interval corresponding to the first target video memory interval, and use the maximum value of the interval as a target first video memory requirement corresponding to the current first video memory requirement;

[0021] Allocating space from the available video memory for the first video memory corresponding to the target first video memory requirement to match the current first video memory requirement;

[0022] The next first video memory demand is used as the current first video memory demand, and the step of matching the current first video memory demand among the plurality of first video memory demands with a preset video memory interval to determine a first target video memory interval matched by the current first video memory demand is performed, until the plurality of first video memory demands are successfully matched with corresponding first video memory allocation spaces from the available video memory to obtain a plurality of first video memory allocation spaces.

[0023] Optionally, the determining a number of second video memory requirements corresponding to the number of second video memory applications, and performing video memory combination based on the preset video memory range, so as to allocate space for the number of second video memory requirements corresponding to the number of second video memory requirements from the available video memory according to the video memory combination, includes:

[0024] Determining a number of second video memory requirements corresponding to the number of second video memory applications;

[0025] Performing a video memory combination corresponding to a current second video memory requirement among the plurality of second video memory requirements based on the preset video memory range, so as to obtain a target video memory combination that meets the current second video memory requirement;

[0026] Allocating space from the available video memory for the second video memory corresponding to the target video memory combination matching the current second video memory demand;

[0027] The next second video memory demand is used as the current second video memory demand, and the step of performing a video memory combination corresponding to the current second video memory demand among the plurality of second video memory demands based on the preset video memory interval to obtain a target video memory combination that meets the current second video memory demand is jumped to, until the plurality of second video memory demands are successfully matched with corresponding second video memory allocation spaces from the available video memory to obtain a plurality of second video memory allocation spaces.

[0028] Optionally, after determining the plurality of second video memory requirements corresponding to the plurality of second video memory applications and performing video memory combination based on the preset video memory range so as to allocate space for the plurality of second video memory requirements corresponding to the plurality of second video memory requirements from the available video memory according to the video memory combination, the method further includes:

[0029] The starting offsets corresponding to the plurality of first video memory allocation spaces and the plurality of second video memory allocation spaces are calculated, so as to read data based on the starting offsets.

[0030] Optionally, after determining the plurality of second video memory requirements corresponding to the plurality of second video memory applications and performing video memory combination based on the preset video memory range so as to allocate space for the plurality of second video memory requirements corresponding to the plurality of second video memory requirements from the available video memory according to the video memory combination, the method further includes:

[0031] If there are a number of target spaces that have been released in the number of first video memory allocation spaces and the number of second video memory allocation spaces, marking the number of target spaces as being in a released state to obtain a number of released spaces;

[0032] Merging the plurality of released spaces to obtain a merged released space;

[0033] If a new video memory application is received, video memory matching is performed for the new existing application based on the merged freed space.

[0034] In a second aspect, the present application discloses a video memory management device, comprising:

[0035] A video memory consumption determination module, used to count a number of video memory applications of a target pre-trained model during a model inference process through a preset recording function, and determine a maximum video memory consumption based on the number of video memory applications;

[0036] a video memory application distinguishing module, configured to determine the available video memory corresponding to the target pre-trained model based on the maximum video memory consumption, and distinguish the plurality of video memory applications according to a preset video memory request threshold, so as to obtain a plurality of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a plurality of second video memory applications whose video memory requirements are greater than the preset video memory request threshold;

[0037] A first video memory allocation module is used to match a plurality of first video memory requirements corresponding to the plurality of first video memory applications with a preset video memory interval to determine a plurality of first target video memory intervals corresponding to the plurality of first video memory requirements, and allocate space from the available video memory for the plurality of first video memory applications to match the plurality of first video memory applications based on the maximum values ​​of the intervals corresponding to the plurality of first target video memory intervals;

[0038] The second video memory allocation module is used to determine a number of second video memory requirements corresponding to the number of second video memory applications, and to perform video memory combinations based on the preset video memory interval, so as to match the number of second video memory requirements with the corresponding number of second video memory allocation spaces from the available video memory according to the video memory combinations.

[0039] In a third aspect, the present application discloses an electronic device, comprising:

[0040] Memory, used to store computer programs;

[0041] The processor is used to execute the computer program to implement the video memory management method as described above.

[0042] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned video memory management method when executed by a processor.

[0043] In the present application, firstly, a preset recording function is used to count a number of video memory applications of a target pre-trained model in a model inference process, and a maximum video memory consumption is determined according to the number of video memory applications. Then, based on the maximum video memory consumption, the available video memory corresponding to the target pre-trained model is determined, and the number of video memory applications are differentiated according to a preset video memory request threshold to obtain a number of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a number of second video memory applications whose video memory requirements are greater than the preset video memory request threshold. Finally, a number of first video memory requirements corresponding to the number of first video memory applications are matched with a preset video memory interval to determine a number of first target video memory intervals corresponding to the number of first video memory requirements, and based on the maximum value of the interval corresponding to the number of first target video memory intervals, a number of first video memory allocation spaces are matched from the available video memory for the number of first video memory applications to correspond to the number of first video memory applications. At the same time, a number of second video memory requirements corresponding to the number of second video memory applications are determined, and video memory combinations are performed based on the preset video memory interval to match a number of second video memory allocation spaces from the available video memory for the number of second video memory requirements to correspond to the number of second video memory requirements according to the video memory combination.

[0044] It can be seen that, through the method of the present application, a number of video memory applications of the target pre-trained model in the model reasoning process can be counted through a preset recording function, and the maximum video memory consumption can be determined according to the determined number of existing applications, and then the corresponding available video memory can be determined according to the maximum video memory consumption, and a number of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a number of second video memory applications whose video memory requirements are greater than the preset video memory request threshold can be distinguished according to the preset video memory request threshold. Then, a number of first video memory requirements corresponding to the number of first video memory applications can be matched with the preset video memory interval to determine the corresponding number of first target video memory intervals, and the first video memory allocation space corresponding to the maximum value of the interval corresponding to the number of first target video memory intervals can be matched for the first video memory application from the available video memory; in another case, a number of second video memory requirements corresponding to the number of second video memory applications can be determined, and video memory combinations can be performed based on the preset video memory interval, so as to match the corresponding second video memory allocation space for the number of second video memory requirements from the available video memory according to the obtained video memory combination. In this way, based on the statistical information, space can be uniformly allocated for subsequent video memory use. According to the characteristics of each video memory requirement, the uniformly allocated large block of video memory can be cut into different video memory modules for easy use, keeping video memory fragmentation at a low level and avoiding memory fragmentation, thereby improving model inference performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flow chart of a video memory management method provided by an embodiment of the present invention;

[0047] Figure 2 A flow chart of a specific video memory management method provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the structure of a video memory management device provided by an embodiment of the present invention;

[0049] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0052] In the prior art, it is not easy to control the size of a large memory block. If the allocation is too large, it is easy to generate a lot of memory fragments, affecting computing utilization. If the allocation is too small, frequent memory requests are required, which has a greater impact on performance.

[0053] In order to overcome the above technical problems, the present application discloses a video memory management method, apparatus, device and storage medium, which can minimize the frequent application and release of memory, avoid memory fragmentation, and improve reasoning performance.

[0054] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0055] See also Figure 1 As shown, an embodiment of the present invention discloses a video memory management method, including:

[0056] Step S11, counting a number of video memory requests of the target pre-trained model during the model inference process through a preset recording function, and determining a maximum video memory consumption based on the number of video memory requests.

[0057] In this embodiment, it is necessary to use a preset recording function to count several video memory requests of the target pre-trained model in the model inference process, and determine the maximum video memory consumption based on the several video memory requests. Specifically, it is necessary to insert a preset recording function in the target pre-trained model. In this embodiment, the target pre-trained model is a vLLM model, and the vLLM model includes two modes, graph and eager. The graph mode is based on cuda graph technology and builds a large model in the form of a computational graph. It is executed after compilation. The inference delay of this mode is low, but the video memory consumption is large. The eager mode is similar to PyTorch's operator-by-operator execution. The inference delay of this mode is relatively high, but it has the characteristics of instant execution, easy debugging, and strong flexibility, and the video memory consumption is small, which is easier to access for new hardware. It should be noted that the vLLM model will perform profile run model inference before allocating kv cache memory in eager mode. Therefore, a preset recording function can be inserted at the location where the PyTorch underlying layer calls the board runtime interface to allocate and release video memory. Then, the preset recording function is used to count several video memory applications of the target pre-trained model during the model inference process, and then the actual video memory application and release situation is determined. For example, 4 32-byte applications, 10 1024-byte applications, and 2 16-byte releases occur during the model inference process.

[0058] It is further necessary to explain that after determining a number of video memory applications, it is necessary to determine a number of video memory requirements corresponding to the respective video memory applications, and then it is necessary to determine the maximum video memory consumption based on the determined video memory requirements. Specifically, the model's reasoning process is based on the longest input supported by the model, so the statistical video memory usage is the maximum video memory consumption that can be achieved during the actual reasoning of the model. Therefore, the peak video memory usage of the reasoning process can be calculated based on the determined video memory requirements, and the peak video memory usage of the reasoning process can be determined as the maximum video memory consumption. In this way, the maximum video memory consumption can be determined by recording a number of video memory applications statistically counted by the function, thereby ensuring the accuracy of the available video memory allocated to the pre-trained model.

[0059] Step S12: determining the available video memory corresponding to the target pre-trained model based on the maximum video memory consumption, and distinguishing the plurality of video memory applications according to a preset video memory request threshold to obtain a plurality of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a plurality of second video memory applications whose video memory requirements are greater than the preset video memory request threshold.

[0060] In this embodiment, it is necessary to determine the available video memory corresponding to the target pre-trained model based on the determined maximum video memory consumption. Specifically, it is necessary to determine the product of the preset total video memory and the preset usage coefficient, and then subtract the corresponding value of the maximum video memory consumption from the corresponding value of the obtained product to obtain the difference between the corresponding value of the product and the corresponding value of the maximum video memory consumption, and then use the obtained difference as the value of the available video memory corresponding to the target pre-trained model.

[0061] Furthermore, it is necessary to distinguish a plurality of video memory applications according to a video memory request threshold value. When the video memory demand corresponding to a video memory application is relatively small, the video memory allocation space can be directly matched to the video memory application according to the maximum value of the interval where the video memory demand is located. When the video memory demand corresponding to a video memory application is relatively large, the video memory needs to be combined to match the video memory allocation space to the application to avoid wasting video memory. Therefore, it is necessary to distinguish a plurality of video memory applications according to the video memory request threshold value. In this embodiment, the video memory request threshold value is set to 1MB, and then a plurality of video memory demands corresponding to a plurality of video memory applications and a preset video memory request threshold value are compared, so as to select a plurality of first video memory applications whose video memory demands are not greater than the preset video memory request threshold value and a plurality of second video memory applications whose video memory demands are greater than the preset video memory request threshold value from the plurality of video memory applications according to the comparison result. In this way, the video memory applications can be distinguished according to the preset video memory request threshold value, so as to perform video memory allocation with higher utilization rate in the process of video memory allocation.

[0062] Step S13, matching the first video memory requirements corresponding to the first video memory applications with preset video memory intervals to determine the first target video memory intervals corresponding to the first video memory requirements, and allocating space from the available video memory for the first video memory requirements corresponding to the first video memory applications based on the maximum values ​​of the intervals corresponding to the first target video memory intervals.

[0063] In the present embodiment, it is necessary to perform video memory allocation according to the determined first video memory application. Specifically, it is necessary to determine a plurality of first video memory requirements corresponding to the plurality of first video memory applications, and match the current first video memory requirement among the plurality of first video memory requirements with the preset video memory interval to determine the first target video memory interval matched by the current first video memory requirement. Since the first video memory requirements corresponding to the first video memory applications are all less than 1MB, and there may be multiple first video memory applications, the plurality of first video memory applications may be matched in sequence, and the preset video memory interval is a pre-set video memory interval, for example, 1KB and below is one level, 1KB to 64KB is one level, 64KB to 128KB is one level, and so on. After determining the first target video memory interval matched by the current first video memory demand, it is necessary to determine the interval maximum value corresponding to the first target video memory interval, and use the interval maximum value as the target first video memory demand corresponding to the current first video memory demand. For example, if the video memory interval matched by the current first video memory demand is 1KB to 64KB, then the interval maximum value 64KB is directly used as the target first video memory demand corresponding to the current first video memory demand. Then, when performing video memory allocation, 64KB of available space is allocated to the first video memory demand. Finally, the next first video memory demand is used as the current first video memory demand, and the process jumps to the step of matching the current first video memory demand among the plurality of first video memory demands with the preset video memory interval to determine the first target video memory interval matched by the current first video memory demand, so as to perform video memory allocation for the next first video memory demand, until the plurality of first video memory demands successfully match the corresponding first video memory allocation space from the available video memory. For example, if there are 20 first video memory demands, and the 20 first video memory demands are all less than 1KB, then 20KB is opened, and 1KB is allocated to the first video memory applications corresponding to the 20 first video memory demands respectively. In this way, since the video memory demand corresponding to the first video memory application is relatively small, even if there is a certain amount of fragmentation during the allocation process, it will not cause excessive waste, and the maximum value of the interval corresponding to the video memory interval is directly used as the target first video memory demand corresponding to the current first video memory demand, which can effectively save the amount of calculation during allocation.

[0064] Step S14, determining a plurality of second video memory requirements corresponding to the plurality of second video memory applications, and performing video memory combination based on the preset video memory range, so as to allocate space from the available video memory for matching the plurality of second video memory requirements with the plurality of second video memory requirements according to the video memory combination.

[0065] In this embodiment, it is necessary to perform video memory allocation according to the determined second video memory application. Specifically, it is necessary to determine a plurality of second video memory requirements corresponding to a plurality of second video memory applications, and perform a video memory combination corresponding to a current second video memory requirement among the plurality of second video memory requirements based on a preset video memory interval, so as to obtain a target video memory combination that meets the current second video memory requirement. Since the first video memory requirements corresponding to the second video memory applications are not less than 1MB, and there may be multiple second video memory applications, and when the video memory requirement is above 1MB, it is not possible to continue to allocate according to the upper limit of the level, because at this time a single video memory request is large, and still allocating according to the upper limit will cause serious fragmentation, and the video memory utilization rate will become very low. Therefore, at the level above 1MB, combined allocation will be performed in sequence, and it is necessary to match the second video memory allocation space corresponding to the target video memory combination to the current second video memory requirement from the available video memory, for example, a 64MB and a 30MB, 54MB are combined together to form a 128MB B is allocated once, and a certain amount of redundant space is still allowed here, and considering that address alignment can improve the reasoning performance to a certain extent, there will be more possible combinations when allocating at a large level. In order to minimize fragmentation, large video memory requirements should be combined first. For example, if the maximum single video memory requirement counted is 5GB, then the upper limit of this level is 8GB. Therefore, it is necessary to combine the requirements above 1MB with this 5GB from large to small to fill in this 8GB allocation, and then combine the remaining allocation requirements according to this principle; finally, it is necessary to take the next second video memory requirement as the current second video memory requirement, and jump to the step of performing a video memory combination corresponding to the current second video memory requirement among the several second video memory requirements based on the preset video memory interval to obtain a target video memory combination that meets the current second video memory requirement, until the several second video memory requirements successfully match the corresponding second video memory allocation space from the available video memory to obtain several second video memory allocation spaces. In this way, while meeting the video memory usage requirements, the impact of fragmentation is minimized as much as possible.

[0066] It should be further explained that the starting offsets corresponding to several first video memory allocation spaces and several second video memory allocation spaces need to be calculated to read data based on the starting offsets. For example, 1KB has 20 times, so the 1KB gear is in the range from the starting address to 20KB, and the 1KB-64KB gear is in the range from 20KB to 724KB, and so on. Later, when the pytorch framework needs video memory, it will select between these blocks according to the required size and record it.

[0067] It can be seen that, through the method of the present application, a number of video memory applications of the target pre-trained model in the model reasoning process can be counted through a preset recording function, and the maximum video memory consumption can be determined according to the determined number of existing applications, and then the corresponding available video memory can be determined according to the maximum video memory consumption, and a number of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a number of second video memory applications whose video memory requirements are greater than the preset video memory request threshold can be distinguished according to the preset video memory request threshold. Then, a number of first video memory requirements corresponding to the number of first video memory applications can be matched with the preset video memory interval to determine the corresponding number of first target video memory intervals, and the first video memory allocation space corresponding to the maximum value of the interval corresponding to the number of first target video memory intervals can be matched for the first video memory application from the available video memory; in another case, a number of second video memory requirements corresponding to the number of second video memory applications can be determined, and video memory combinations can be performed based on the preset video memory interval, so as to match the corresponding second video memory allocation space for the number of second video memory requirements from the available video memory according to the obtained video memory combination. In this way, based on the statistical information, space can be uniformly allocated for subsequent video memory use. According to the characteristics of each video memory requirement, the uniformly allocated large block of video memory can be cut into different video memory modules for easy use, keeping video memory fragmentation at a low level and avoiding memory fragmentation, thereby improving model inference performance.

[0068] Based on the above embodiments, it can be known that while memory allocation is being performed, memory release may also occur. Therefore, this embodiment provides a detailed description of how to release memory and reuse memory. Figure 2 As shown, an embodiment of the present invention discloses a video memory management method, including:

[0069] Step S21: If there are some target spaces that have been released in some first video memory allocation spaces and some second video memory allocation spaces, mark the some target spaces as being in a released state to obtain some released spaces.

[0070] In this embodiment, if it is found that there are several target spaces that have been released in several first video memory allocation spaces and several second video memory allocation spaces, it is necessary to mark the several target spaces that have been released and mark them as released. It should be noted that pytorch usually releases memory when a role ends or when certain interfaces are explicitly called. However, it should be noted that when the pytorch end releases, the underlying memory will not be truly released. Instead, a mark is made for each occupied memory and it is marked as released, thereby obtaining several released spaces.

[0071] Step S22: Merge the plurality of released spaces to obtain a merged released space.

[0072] In this embodiment, it is necessary to merge several released memories into a large memory to obtain the released space after the merger, and it is necessary to build a map mapping with the starting address and length of each occupied memory as the key value. If it is detected that pytorch calls the interface to release this memory, it will be marked as released and all the currently released memories will be merged.

[0073] Step S23: If a new video memory application is received, video memory matching is performed for the new existing application based on the space released after merging.

[0074] In this embodiment, if a new video memory application is received, it is necessary to find the corresponding merged released space according to the constructed map mapping to match the video memory for the new existing application, and release all the merged released space at once when finally exiting the model inference.

[0075] It should be noted that if a new video memory application is received, the amount of video memory corresponding to the new video memory application needs to be determined. Because there may be multiple new video memory applications in a short period of time, a video memory application queue can be constructed, and then the received new video memory application is added to the video memory application queue in chronological order. Then, video memory matching is performed for the new video memory application in sequence according to the order in the queue. When performing video memory matching for the new video memory application, the video memory demand corresponding to the new video memory application needs to be determined. If the video memory demand is greater than the current released space after merging, the current released space after merging cannot meet the video memory demand corresponding to the current new video memory application. Therefore, a video memory application queue to be matched needs to be set. When the current released space after merging cannot meet the video memory demand corresponding to the current new video memory application, the current new video memory application can be transferred to the video memory application queue to be matched, and then video memory matching is performed for the next new video memory application in the video memory application queue. While matching, the video memory applications in the video memory application queue to be matched are added back to the video memory application queue in sequence. In this way, it can be ensured that the space released after the merger will first satisfy the video memory requests that meet the conditions, avoiding the waste of video memory.

[0076] It can be seen that in this embodiment, if there are several target spaces that have been released in several first video memory allocation spaces and several second video memory allocation spaces, the several target spaces are marked as released to obtain several released spaces, and then the several released spaces can be merged to obtain merged released spaces. Finally, if a new video memory application is received, video memory matching is performed for the new existing application based on the merged released spaces. In this way, the number of times the interface is called to allocate and release video memory can be reduced, the reasoning performance is improved, and the video memory reuse is fully considered, thereby reducing the problem of video memory fragmentation.

[0077] See also Figure 3 As shown, an embodiment of the present invention discloses a video memory management device, including:

[0078] The video memory consumption determination module 11 is used to count a number of video memory applications of the target pre-trained model during the model inference process through a preset recording function, and determine the maximum video memory consumption according to the number of video memory applications;

[0079] A video memory application distinguishing module 12 is used to determine the available video memory corresponding to the target pre-trained model based on the maximum video memory consumption, and distinguish the plurality of video memory applications according to a preset video memory request threshold to obtain a plurality of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a plurality of second video memory applications whose video memory requirements are greater than the preset video memory request threshold;

[0080] A first video memory allocation module 13 is used to match the first video memory requirements corresponding to the first video memory applications with a preset video memory interval to determine the first target video memory intervals corresponding to the first video memory requirements, and allocate space from the available video memory for the first video memory applications to match the first video memory requirements based on the maximum values ​​of the intervals corresponding to the first target video memory intervals;

[0081] The second video memory allocation module 14 is used to determine a number of second video memory requirements corresponding to the number of second video memory applications, and to perform video memory combinations based on the preset video memory interval, so as to match a number of second video memory allocation spaces corresponding to the number of second video memory requirements from the available video memory according to the video memory combinations.

[0082] In some embodiments, the video memory consumption determination module 11 may specifically include:

[0083] A video memory application statistics unit, used to insert a preset recording function into the target pre-trained model, and to count a number of video memory applications of the target pre-trained model during the model inference process through the preset recording function;

[0084] The maximum video memory consumption determination unit is used to count a number of video memory requirements corresponding to the number of video memory applications, and determine the maximum video memory consumption based on the number of video memory requirements.

[0085] In some embodiments, the video memory application distinguishing module 12 may specifically include:

[0086] An available video memory determination unit, used to determine the product of a preset total video memory and a preset usage coefficient, and calculate a difference between a value corresponding to the product and a value corresponding to the maximum video memory consumption, so as to use the obtained difference as the available video memory corresponding to the target pre-trained model;

[0087] A threshold comparison unit, used to compare the plurality of video memory requirements respectively corresponding to the plurality of video memory applications with a preset video memory request threshold to obtain a corresponding comparison result;

[0088] The video memory application distinguishing unit is used to select, from the plurality of video memory applications, a plurality of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a plurality of second video memory applications whose video memory requirements are greater than the preset video memory request threshold according to the comparison result.

[0089] In some embodiments, the first video memory allocation module 13 may specifically include:

[0090] A first video memory requirement determining unit, configured to determine a plurality of first video memory requirements corresponding to the plurality of first video memory applications;

[0091] A first video memory interval matching unit, configured to match a current first video memory requirement among the plurality of first video memory requirements with a preset video memory interval to determine a first target video memory interval matched by the current first video memory requirement;

[0092] a target video memory requirement determining unit, configured to determine a maximum value of an interval corresponding to the first target video memory interval, and use the maximum value of the interval as a target first video memory requirement corresponding to the current first video memory requirement;

[0093] A first video memory allocation space determining unit, configured to match a first video memory allocation space corresponding to the target first video memory demand to the current first video memory demand from the available video memory;

[0094] The first step jump unit is used to take the next first video memory demand as the current first video memory demand, and jump to the step of matching the current first video memory demand among the plurality of first video memory demands with a preset video memory interval to determine a first target video memory interval matched by the current first video memory demand, until the plurality of first video memory demands are successfully matched with corresponding first video memory allocation spaces from the available video memory to obtain a plurality of first video memory allocation spaces.

[0095] In some embodiments, the second video memory allocation module 14 may specifically include:

[0096] A second video memory requirement determining unit, configured to determine a plurality of second video memory requirements corresponding to the plurality of second video memory applications;

[0097] a video memory combination determining unit, configured to perform a video memory combination corresponding to a current second video memory requirement among the plurality of second video memory requirements based on the preset video memory interval, so as to obtain a target video memory combination that meets the current second video memory requirement;

[0098] A second video memory allocation space determining unit, configured to match a second video memory allocation space corresponding to the target video memory combination for the current second video memory demand from the available video memory;

[0099] The second step jump unit is used to take the next second video memory demand as the current second video memory demand, and jump to the step of performing a video memory combination corresponding to the current second video memory demand among the plurality of second video memory demands based on the preset video memory interval to obtain a target video memory combination that meets the current second video memory demand, until the plurality of second video memory demands are successfully matched with corresponding second video memory allocation spaces from the available video memory to obtain a plurality of second video memory allocation spaces.

[0100] In some embodiments, the video memory management device may further include:

[0101] The data reading unit is used to calculate the starting offsets corresponding to the plurality of first video memory allocation spaces and the plurality of second video memory allocation spaces, so as to read data based on the starting offsets.

[0102] In some embodiments, the video memory management device may further include:

[0103] A control state marking unit, used for marking the target spaces as being in a released state if there are target spaces that have been released in the first video memory allocation spaces and the second video memory allocation spaces, so as to obtain the released spaces;

[0104] A space merging unit, used for merging the plurality of released spaces to obtain a merged released space;

[0105] The video memory matching unit is used for matching the new existing video memory application based on the merged released space if a new video memory application is received.

[0106] Furthermore, the present application also discloses an electronic device. Figure 4 This is a structural diagram of an electronic device according to an exemplary embodiment, and the content in the diagram cannot be regarded as any limitation on the scope of use of this application. The electronic device may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the video memory management method disclosed in any of the aforementioned embodiments. In addition, the electronic device in this embodiment may specifically be an electronic computer.

[0107] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0108] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0109] The operating system 221 is used to manage and control various hardware devices on the electronic device and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the video memory management method executed by the electronic device disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0110] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed video memory management method. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, and no further description will be given here.

[0111] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0112] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0113] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0114] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0115] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video memory management method, characterized in that: include: Using a preset recording function to count a number of video memory applications of the target pre-trained model during the model inference process, and determining a maximum video memory consumption according to the number of video memory applications; Determine the available video memory corresponding to the target pre-trained model based on the maximum video memory consumption, and differentiate the plurality of video memory applications according to a preset video memory request threshold to obtain a plurality of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a plurality of second video memory applications whose video memory requirements are greater than the preset video memory request threshold; Matching the first video memory requirements corresponding to the first video memory applications with a preset video memory interval to determine the first target video memory intervals corresponding to the first video memory requirements, and allocating space from the available video memory for the first video memory requirements corresponding to the first video memory applications based on the maximum values ​​of the intervals corresponding to the first target video memory intervals; A plurality of second video memory requirements corresponding to the plurality of second video memory applications are determined, and video memory combinations are performed based on the preset video memory interval, so as to allocate space from the available video memory for matching the plurality of second video memory requirements with the plurality of second video memory requirements according to the video memory combinations.

2. The video memory management method according to claim 1, characterized in that: The method of counting a number of video memory applications of the target pre-trained model in the model inference process by using a preset recording function, and determining a maximum video memory consumption according to the number of video memory applications, includes: Insert a preset recording function into the target pre-trained model, and use the preset recording function to count several video memory applications of the target pre-trained model during the model inference process; A number of video memory requirements corresponding to the number of video memory applications are counted, and a maximum video memory consumption is determined based on the number of video memory requirements.

3. The video memory management method according to claim 2, characterized in that: The determining the available video memory corresponding to the target pre-trained model based on the maximum video memory consumption, and distinguishing the plurality of video memory applications according to a preset video memory request threshold, so as to obtain a plurality of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a plurality of second video memory applications whose video memory requirements are greater than the preset video memory request threshold, includes: Determine the product of a preset total video memory and a preset usage coefficient, and calculate the difference between a value corresponding to the product and a value corresponding to the maximum video memory consumption, so as to use the obtained difference as the available video memory corresponding to the target pre-trained model; Comparing the plurality of video memory requirements respectively corresponding to the plurality of video memory applications with a preset video memory request threshold to obtain a corresponding comparison result; According to the comparison result, a plurality of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a plurality of second video memory applications whose video memory requirements are greater than the preset video memory request threshold are screened out from the plurality of video memory applications.

4. The video memory management method according to claim 1, characterized in that: The matching of the first video memory requirements corresponding to the first video memory applications with a preset video memory interval to determine the first target video memory intervals corresponding to the first video memory requirements, and allocating space from the available video memory to the first video memory corresponding to the first video memory applications based on the maximum values ​​of the intervals corresponding to the first target video memory intervals, includes: Determining a number of first video memory requirements corresponding to the number of first video memory applications; Matching a current first video memory requirement among the plurality of first video memory requirements with a preset video memory interval to determine a first target video memory interval matched by the current first video memory requirement; Determine a maximum value of an interval corresponding to the first target video memory interval, and use the maximum value of the interval as a target first video memory requirement corresponding to the current first video memory requirement; Allocating space from the available video memory for the first video memory corresponding to the target first video memory requirement to match the current first video memory requirement; The next first video memory demand is used as the current first video memory demand, and the step of matching the current first video memory demand among the plurality of first video memory demands with a preset video memory interval to determine a first target video memory interval matched by the current first video memory demand is performed, until the plurality of first video memory demands are successfully matched with corresponding first video memory allocation spaces from the available video memory to obtain a plurality of first video memory allocation spaces.

5. The video memory management method according to claim 1, characterized in that: The determining of the plurality of second video memory requirements corresponding to the plurality of second video memory applications and combining video memory based on the preset video memory interval, so as to allocate space for the plurality of second video memory requirements corresponding to the plurality of second video memory requirements from the available video memory according to the video memory combination, includes: Determining a number of second video memory requirements corresponding to the number of second video memory applications; Performing a video memory combination corresponding to a current second video memory requirement among the plurality of second video memory requirements based on the preset video memory range, so as to obtain a target video memory combination that meets the current second video memory requirement; Allocating space from the available video memory for the second video memory corresponding to the target video memory combination matching the current second video memory demand; The next second video memory demand is used as the current second video memory demand, and the step of performing a video memory combination corresponding to the current second video memory demand among the plurality of second video memory demands based on the preset video memory interval to obtain a target video memory combination that meets the current second video memory demand is jumped to, until the plurality of second video memory demands are successfully matched with corresponding second video memory allocation spaces from the available video memory to obtain a plurality of second video memory allocation spaces.

6. The video memory management method according to claim 1, characterized in that: After determining the plurality of second video memory requirements corresponding to the plurality of second video memory applications, and performing video memory combination based on the preset video memory interval, so as to allocate space from the available video memory for matching the plurality of second video memory requirements with the plurality of second video memory requirements according to the video memory combination, the method further includes: The starting offsets corresponding to the plurality of first video memory allocation spaces and the plurality of second video memory allocation spaces are calculated, so as to read data based on the starting offsets.

7. The video memory management method according to any one of claims 1 to 6, characterized in that: After determining the plurality of second video memory requirements corresponding to the plurality of second video memory applications, and performing video memory combination based on the preset video memory interval, so as to allocate space from the available video memory for matching the plurality of second video memory requirements with the plurality of second video memory requirements according to the video memory combination, the method further includes: If there are a number of target spaces that have been released in the number of first video memory allocation spaces and the number of second video memory allocation spaces, marking the number of target spaces as being in a released state to obtain a number of released spaces; Merging the plurality of released spaces to obtain a merged released space; If a new video memory application is received, video memory matching is performed for the new existing application based on the merged freed space.

8. A video memory management device, characterized in that: include: A video memory consumption determination module, used to count a number of video memory applications of a target pre-trained model during a model inference process through a preset recording function, and determine a maximum video memory consumption based on the number of video memory applications; a video memory application distinguishing module, configured to determine the available video memory corresponding to the target pre-trained model based on the maximum video memory consumption, and distinguish the plurality of video memory applications according to a preset video memory request threshold, so as to obtain a plurality of first video memory applications whose video memory requirements are not greater than the preset video memory request threshold and a plurality of second video memory applications whose video memory requirements are greater than the preset video memory request threshold; A first video memory allocation module is used to match a plurality of first video memory requirements corresponding to the plurality of first video memory applications with a preset video memory interval to determine a plurality of first target video memory intervals corresponding to the plurality of first video memory requirements, and allocate space from the available video memory for the plurality of first video memory applications to match the plurality of first video memory applications based on the maximum values ​​of the intervals corresponding to the plurality of first target video memory intervals; The second video memory allocation module is used to determine a number of second video memory requirements corresponding to the number of second video memory applications, and to perform video memory combinations based on the preset video memory interval, so as to match the number of second video memory requirements with the corresponding number of second video memory allocation spaces from the available video memory according to the video memory combinations.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the video memory management method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the video memory management method according to any one of claims 1 to 7 is implemented.

Images