Video memory allocation processing method, device, equipment and system

Abstract

The present specification provides a method, device, equipment and system for processing video memory allocation, which performs hash operation on model parameters in an intelligent learning model, and determines whether the model parameters of the model to be deployed are repeated with the model parameters that have been deployed by comparing the hash values ​​of the model parameters. If they are repeated, there is no need to allocate new physical video memory. The repeated model parameters are mapped to the corresponding physical video memory by means of virtual pointers to achieve sharing of the same content. The same model parameters do not need to be saved repeatedly, and data sharing of the same content is achieved, which greatly saves physical video memory space, and can further achieve deployment of more instances within limited video memory space, thereby improving system performance.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to a method, device, equipment and system for video memory allocation and processing. Background Art

[0002] With the development of computer Internet technology, the application scenarios of intelligent models are increasing. Generally, trained intelligent models can be deployed on computers. As the service content increases, more and more intelligent models are required. More and more nodes need to deploy multiple intelligent models. Each model needs to occupy the physical video memory of the computer, but the capacity of the computer's physical video memory is limited. Therefore, it is necessary to reasonably allocate and manage the computer's video memory. Summary of the invention

[0003] The purpose of the embodiments of this specification is to provide a method, device, equipment and system for video memory allocation processing, which reduces the occupation of video memory space and improves the utilization rate of video memory.

[0004] On the one hand, an embodiment of the present specification provides a method for processing video memory allocation, the method comprising:

[0005] Get the model parameter set of the model to be deployed;

[0006] Performing a hash operation on each of the acquired model parameter sets to obtain a parameter hash value of each model parameter set;

[0007] Matching the parameter hash values ​​of each model parameter set with the video memory mapping table in sequence to determine whether each model parameter set of the model to be deployed is the same as the deployed model parameters in the video memory mapping table; the video memory mapping table includes parameter hash values ​​of multiple deployed model parameters and physical video memory addresses corresponding to the deployed model parameters;

[0008] If it is determined that the model parameter set is the same as the deployed model parameters in the video memory mapping table, a virtual video memory pointer is allocated to the model parameter set, and the physical video memory address of the deployed model parameters that are the same as the model parameter set is mapped to the virtual video memory pointer.

[0009] On the other hand, this specification provides a video memory allocation processing device, the device comprising:

[0010] A parameter acquisition module is used to obtain a set of model parameters of the model to be deployed;

[0011] A hash operation module, used to perform a hash operation on each of the acquired model parameter sets to obtain a parameter hash value of each model parameter set;

[0012] A parameter duplication checking module is used to match the parameter hash values ​​of each model parameter set with the video memory mapping table in turn to determine whether each model parameter set of the model to be deployed is the same as the deployed model parameters in the video memory mapping table; the video memory mapping table includes parameter hash values ​​of multiple deployed model parameters and physical video memory addresses corresponding to the deployed model parameters;

[0013] A video memory allocation module is used to allocate a virtual video memory pointer to the model parameter set if it is determined that the model parameter set is the same as the deployed model parameters in the video memory mapping table, and map the physical video memory address of the deployed model parameters that are the same as the model parameter set to the virtual video memory pointer.

[0014] On the other hand, an embodiment of the present specification provides a video memory allocation processing device, including at least one processor and a memory for storing processor executable instructions, and the processor implements the above-mentioned video memory allocation processing method when executing the instructions.

[0015] In another aspect, an embodiment of the present specification provides a graphics memory allocation processing system, the system comprising: a plurality of graphics processors, a global model parameter management module, each graphics processor comprising a plurality of processes or containers, each process or container comprising at least one model to be deployed, wherein:

[0016] An inter-process communication module is provided in each process or container, and the global parameter management module is used to execute the above-mentioned video memory allocation processing method. The inter-process communication module is used to query whether the model parameters of each model to be deployed are repeated, so as to allocate video memory to the models to be deployed in multiple graphics processors.

[0017] The video memory allocation processing method, device, equipment and system provided in this specification perform hash operations on the model parameters in the intelligent learning model, and determine whether the model parameters of the model to be deployed are repeated with the model parameters that have been deployed by comparing the hash values ​​of the model parameters. If they are repeated, there is no need to allocate new physical video memory. The repeated model parameters are mapped to the corresponding physical video memory using virtual pointers to achieve sharing of the same content. The same model parameters do not need to be saved repeatedly, and data sharing of the same content is achieved, which greatly saves physical video memory space and improves the utilization rate of the video memory space. It is then possible to deploy more instances within a limited video memory space, thereby improving system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 It is a flowchart of an embodiment of a method for video memory allocation processing provided in an embodiment of this specification;

[0020] Figure 2 It is a schematic diagram of the principle of model parameter hash calculation in one embodiment of this specification;

[0021] Figure 3 This is a schematic diagram of the model loading memory allocation process in one embodiment of this specification;

[0022] Figure 4 This is a schematic diagram of the effect of video memory allocation management in a multi-process, multi-thread, and multi-container model in one embodiment of this specification;

[0023] Figure 5 It is a schematic diagram of the module structure of an embodiment of a video memory allocation processing device provided in this specification;

[0024] Figure 6 It is a schematic diagram of the principle framework of a video memory allocation processing system in one embodiment of this specification;

[0025] Figure 7 is a flowchart of physical video memory allocation management in another embodiment of the present specification;

[0026] Figure 8 It is a hardware structure block diagram of a video memory allocation processing server in one embodiment of this specification. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0028] Nowadays, there are more and more application scenarios for intelligent models. For example, deep learning models can be used in payment (face recognition), damage assessment (image recognition), interaction and customer service (speech recognition, content filtering) and other scenarios. However, the use of intelligent models generally requires strong computing power support, so most of the current tasks are run on acceleration devices such as GPUs (graphics processing units). In order to achieve higher throughput performance or higher resource utilization, multiple GPUs are often deployed on a single node, and each GPU deploys multiple instances of the model (multi-process or multi-thread or multiple containers), and each instance can independently provide service capabilities (scale-out, combined with upper-layer load balancing). For example, some scenarios may require loading multiple identical intelligent learning models on one or more GPUs to speed up data processing, etc. In this way, the model parameters of multiple intelligent learning models need to be loaded on the GPU, which occupies a lot of video memory. However, the limited GPU hardware video memory capacity (currently 16GB is used online) directly restricts the number of model instances or the deployment of larger models. Video memory can be understood as high-speed memory on the GPU card. It has a high bandwidth (up to 700GB+ / sec) but a small capacity (typically 16 / 32GB), which restricts some deep learning tasks with large models and large samples.

[0029] The embodiments of this specification provide a method for video memory allocation and processing. By calculating and comparing hashes during the model loading phase, it is possible to globally and precisely identify which model parameters are the same, and then map the same content to the same GPU video memory, so as to reasonably allocate video memory space for the deployment of intelligent models, thereby reducing video memory occupancy during the long-term operation of the model and achieving the purpose of deploying larger models or more instances.

[0030] Figure 1 It is a flowchart of an embodiment of a video memory allocation processing method provided in an embodiment of this specification. Although this specification provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure shown in the embodiments or drawings of this specification. When the method or module structure is applied in an actual device, server or terminal product, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiment or drawings (for example, a parallel processor or multi-threaded processing environment, or even a distributed processing, server cluster implementation environment).

[0031] The video memory allocation processing method provided in the embodiments of this specification can be flexibly deployed in cloud-native containers, physical bare metal environments, and the like. For example, it can be applied to the client or server, such as smartphones, tablet computers, computers, smart wearable devices, vehicle-mounted devices and other terminals. The specific method can be determined according to actual needs and is not specifically limited in the embodiments of this specification.

[0032] like Figure 1 As shown, the method may include the following steps:

[0033] Step 102: Obtain a set of model parameters of the model to be deployed.

[0034] In the specific implementation process, the model to be deployed can be understood as an intelligent learning model that needs to be deployed on the GPU, that is, the GPU needs to load the model to perform the corresponding task. Referring to the records of the above embodiments, after the model is deployed on the GPU, the model can be used to perform the corresponding task. It can be a deep learning model, a tree model, a random deep forest model, a regression model, etc. It can be determined according to actual needs, and the embodiments of this specification do not make specific limitations. There can be multiple models to be deployed in the embodiments of this specification, and they can be deployed on different GPUs. In some embodiments of this specification, the models to be deployed can be located in different threads or different processes or different containers. In other words, the embodiments of this specification can realize the allocation management of model video memory in multiple processes, multiple containers, and even multiple GPUs, and realize unified management of video memory allocation across processes and GPUs.

[0035] In general, there are multiple model parameters in the intelligent learning model, and the model parameters can also be understood as the weights of the model, that is, the features learned through model training. When the model is inferred, the weights are often all loaded onto the GPU of the inference device. For performance (such as throughput), a node often loads multiple model instances (thus occupying more video memory), and each instance can independently provide service capabilities. Therefore, it is necessary to reasonably allocate the video memory space of the GPU for the model parameters, that is, the weights. In the embodiment of this specification, the classification rules of the model parameters can be pre-set, and the model parameters of the model to be deployed can be divided into multiple model parameter sets, and each model parameter set can include multiple model parameters. Among them, the classification rules of the model parameters can be determined according to actual needs, such as: they can be divided according to the position of the parameters in the model, the size of the parameters, etc., or all the model parameters of the entire model can be directly used as a model parameter set, which is not specifically limited in the embodiment of this specification.

[0036] Step 104: Perform a hash operation on each of the acquired model parameter sets to obtain a parameter hash value of each model parameter set.

[0037] In the specific implementation process, after the model parameters of the model to be deployed are divided, hash operations can be performed on each model parameter set in turn. Specifically, hash operations can be performed on all model parameter values ​​in the model parameter set to obtain parameter hash values ​​of each model parameter set.

[0038] In some embodiments of this specification, obtaining a model parameter set of a to-be-deployed model includes:

[0039] Obtaining model parameters of each level of the model to be deployed in sequence, and taking a set of model parameters of each level as a model parameter set;

[0040] Alternatively, model parameters of consecutive preset levels of the model to be deployed are obtained in sequence, and a set of model parameters of consecutive preset levels is used as a model parameter set.

[0041] In the specific implementation process, the embodiments of this specification can divide the model parameters of the model to be deployed layer by layer, and take the set of all model parameters of each layer as a model parameter set, and then implement hash operations on the model parameters of each layer, implement fine-grained operations layer by layer, and increase the accuracy of data processing. Alternatively, in some embodiments of this specification, the model parameters of the continuous preset layers of the model to be deployed, such as: 3 consecutive layers, can also be taken as a model parameter set to avoid too fine parameter division and increase the amount of hash calculation. The hash operation of the model parameters in the specified range can be implemented, and the data processing speed can be improved while ensuring the accuracy of data processing, laying an accurate data foundation for the subsequent memory allocation of model parameters.

[0042] The level of the model can be understood as a range in the model structure. Especially for some hierarchical models, network structure models or tree models, the model can be divided into layers according to the structure of the model. For example, for a tree structure model, all tree nodes in each layer can be regarded as a level, and for a network structure model, each layer of the network can be regarded as a level.

[0043] In some embodiments of this specification, obtaining a model parameter set of a to-be-deployed model includes:

[0044] Calculate the parameter size of the model parameters within the specified range of the model to be deployed. If the parameter size is less than a preset threshold, add the adjacent model parameters in the specified range to the specified range, and calculate the parameter size of the model parameters in the new specified range. When the parameter size is greater than the preset threshold, take the model parameters in the corresponding specified range as a model parameter set.

[0045] In the specific implementation process, a preset threshold can be set in advance, and the parameter size of the model parameters within the specified range can be calculated at one time, wherein the specified range can be determined according to actual needs, such as: a level, or adjacent continuous n parameters. If the parameter size of the model parameters within the specified range is less than the preset threshold, it means that the parameters within the specified range are relatively small, and the specified range can be increased, that is, the model parameters adjacent to the specified range are added to the specified range, and the parameter size of the model parameters within the increased specified range is calculated. If the parameter size is still less than the preset threshold, the specified range is continued to be increased until the parameter size within the specified range is greater than the preset threshold, and the model parameters within the latest specified range can be used as a model parameter set. By analogy, the subsequent model parameters are divided into sets. Among them, the number of model parameters added to the specified range each time can be determined according to actual needs, such as: it can be consistent with the specified range, that is, if the parameter size within the specified range is less than the preset threshold, the model parameters within the specified range adjacent to the specified range are added to the original specified range.

[0046] For example: Figure 2 is a schematic diagram of the principle of model parameter hash calculation in one embodiment of this specification, such as Figure 2 As shown, Figure 2 In layers, the model layers are represented. The specified range is preset as one layer. If the parameter size of a layer is less than 2MB, multiple consecutive layer parameters (until > 2MB) are calculated together. Figure 2 As shown, it can be seen that because the size of the model parameters at the L1 level is less than 2M, the model parameters at the L2 level are calculated together with the model parameters at the L1 level. If the size of the model parameters at the L1 and L2 levels is greater than 2M, the model parameters at the L1 and L2 levels are taken as a model parameter set. Continue to calculate the model parameters at the L3 level, and it is found that the parameter size of the model parameters at the L3 level is greater than 2M, so the model parameters at the L3 level can be taken as a model parameter set alone. Similarly, the model parameters at the L4 to L6 levels are taken as a model parameter set, and the model parameters at the L7 level are taken as a model parameter set. In this way, for particularly small model parameters, the subsequent hash calculations, the number of queries, and the metadata overhead can be reduced.

[0047] Of course, the maximum granularity can be to calculate a hash for the model parameters of all levels of the entire model, at which time the granularity is the coarsest. Hash digest algorithms that are widely verified in the fields of security and storage can be used, such as MD5 (generating 128 bits) or SHA1 (generating 160 bits).

[0048] The embodiments of this specification can configure model parameter hash calculation methods of different granularities according to actual usage needs to meet different data processing requirements.

[0049] Step 106: Match the parameter hash value of each model parameter set with the video memory mapping table in turn to determine whether each model parameter set of the model to be deployed is the same as the deployed model parameters in the video memory mapping table; the video memory mapping table includes parameter hash values ​​of multiple deployed model parameters and physical video memory addresses corresponding to the deployed model parameters.

[0050] In the specific implementation process, a video memory mapping table can be pre-built, and the physical video memory address of each deployed model parameter and the deployed model parameter can be stored in the video memory mapping table. After calculating the parameter hash value of each model parameter set of the model to be deployed, the parameter hash value of each model parameter can be matched with the video memory mapping table to query whether the model parameters of the model to be deployed are repeated with the model parameters of the deployed model. In general, if the content of the model parameters (which can be refined to the model layer level) is the same, then the hash value calculated based on the content (for example, using MD5, SHA1) must also be the same, so as to identify the model parameters of the same content. For example: after performing a layer-by-layer parameter hash operation on the model to be deployed, and calculating the hash value of the model parameters of each layer, the hash value of the model parameters of each layer can be compared with the parameter hash value in the video memory mapping table. If there is a parameter hash value in the video memory mapping table that is the same as the parameter hash value of a certain layer of the model to be deployed, it can be considered that the model parameters of the layer are the same as the corresponding deployed model parameters in the video memory mapping table.

[0051] In addition, it should be noted that the hash values ​​of all model parameter sets in the model to be deployed can be calculated first and then matched with the video memory mapping table. Alternatively, the parameter hash value of a model parameter set can be calculated, that is, the calculated parameter hash value is matched with the parameter hash value of the deployed model parameters in the video memory mapping table to see if they are consistent, and then the parameter hash value of the next model parameter is calculated. The specific process can be determined according to actual needs, and the embodiments of this specification do not make specific limitations.

[0052] Step 108: If it is determined that the model parameter set is the same as the deployed model parameters in the memory mapping table, a virtual memory pointer is allocated to the model parameter set, and the physical memory address of the deployed model parameters that are the same as the model parameter set is mapped to the virtual memory pointer.

[0053] In the specific implementation process, if a model parameter set in the model to be deployed is found to be the same as the deployed model parameters in the video memory mapping table, a virtual video memory pointer can be returned, and the physical video memory address of the deployed model parameters that are the same as the model parameter set is obtained according to the video memory mapping table, and the obtained physical video memory address is mapped to the virtual video memory pointer corresponding to the model parameter set. Among them, the virtual video memory pointer can be understood as a pointer visible to the application, which can be used to point multiple virtual video memory pointers to the same physical memory. Figure 3 Schematic diagram of model loading memory allocation process in one embodiment of this specification, such as Figure 3 As shown, if the hash value of the model parameter set of the 1 / Nth level of model 2 is queried and is repeated with a deployed model parameter in the video memory mapping table, the corresponding physical video memory address, i.e., addr1, can be returned according to the video memory mapping table, and then addr1 can be mapped to the virtual video memory pointer, i.e., ptr2, corresponding to the model parameter set. There is no need to place the model parameter set of the 1 / Nth level of model 2 on the GPU. Figure 3 The GWM (Global Weight Hash Manager) in can be understood as global model parameter management.

[0054] Among them, a certain model parameter set in the model to be deployed is the same as the deployed model parameters in the video memory mapping table. The model to be deployed and a certain deployed model in the video memory mapping table may be the same model. At this time, all model parameters of the model to be deployed and a certain deployed model in the video memory mapping table may be the same. It may also be that the model to be deployed and a certain deployed model in the video memory mapping table are similar models, and some model parameters are the same.

[0055] In addition, the virtual video memory pointer can return a virtual video memory pointer after the parameter hash value of the model parameter set is calculated, or after the model parameters of the model to be deployed are divided into sets, a virtual video memory pointer can be allocated to each model parameter set. The virtual video memory pointer corresponding to each model parameter set is different, and the method for generating the virtual video memory pointer is not specifically limited in the embodiments of this specification. Moreover, in some embodiments of this specification, a sub-virtual video memory pointer can be generated for each model parameter in the model parameter set according to the virtual video memory pointer of the model parameter set, such as: the offset function (reference function) can be used, and the sub-virtual video memory pointer of each model parameter can be the virtual video memory pointer of the model parameter set + the offset of the model parameter. For example: the virtual video memory pointer of the model parameter set is t, and there are 5 model parameters in the model parameter set, among which the word virtual video memory pointer of model parameter 1 can be expressed as t+offset1, and so on, to obtain the word virtual video memory pointer of each model parameter. The child virtual memory pointer of each model parameter and the virtual parameter pointer of the corresponding model parameter set point to the same physical memory address. In this way, each model parameter can correspond to its own virtual memory pointer, and the corresponding physical memory address can be found using the virtual memory pointer, so that each model parameter corresponds to a certain physical memory. When it is necessary to load and use the model parameter, the corresponding physical memory address can be queried based on the virtual memory pointer, and then the corresponding model parameter can be loaded or referenced.

[0056] Figure 4 FIG. 1 is a schematic diagram of the effect of video memory allocation management in a multi-process, multi-thread, and multi-container model in an embodiment of this specification. Figure 4 As shown, if the models in different threads, processes or containers have the same model parameters, that is, the same content, then only one copy of the model parameters can be saved to the physical memory of the GPU. By using the virtual memory pointer, the physical memory address where the same model parameters are stored is mapped to the virtual memory pointer, thereby realizing the sharing of the same model parameters in different processes, threads, and containers.

[0057] In some embodiments of this specification, the method further includes:

[0058] If it is determined that the model parameter set is different from all deployed model parameters in the video memory mapping table, allocating a virtual video memory pointer and a physical video memory address to the model parameter;

[0059] The model parameters in the model parameter set are transmitted to the corresponding physical video memory address, and the physical video memory address allocated by the model parameter set is mapped to the corresponding virtual video memory.

[0060] In the specific implementation process, Figure 3As shown, the hash value of the model parameter set of the 1st / Nth level of the query model 1 is different from all the deployed model parameters in the memory mapping table, that is, the model parameter set of the 1st / Nth level of the query model 1 does not repeat the deployed model parameters. At this time, it can be returned that it does not exist, and the physical memory address addr1 and the virtual memory pointer ptr1 are allocated to the model parameter set, addr1 is mapped to ptr1, and the model parameters in the model parameter set are transferred to addr1 in the GPU.

[0061] In some embodiments of this specification, the method further includes:

[0062] The parameter hash value and the physical video memory address of the model parameter set are added to the video memory mapping table.

[0063] In the specific implementation process, after the physical video memory address is newly allocated, the parameter hash value of the model parameter set of the newly allocated video memory and the physical video memory address can be added to the video memory mapping table, and the video memory mapping table can be updated in time to facilitate subsequent model parameter queries.

[0064] Among them, the deployed model parameters in the video memory mapping table can also be a parameter set of a deployed model. The model parameter video memory allocation rules when the model is deployed can be pre-set, and the parameter hash value of the corresponding model parameter set is calculated according to the preset rules each time the model is loaded and deployed. For example: physical video memory is allocated to the model parameters of each level according to the level of the model. When loading and deploying the first model, the parameter hash values ​​of the model parameter set of each level of the first model are calculated in turn, and the calculated parameter hash values ​​are matched with the video memory mapping table. At this time, there is no data in the video memory mapping table. Therefore, it can be determined that the model parameter sets of each level of the first model will not be repeated. Physical video memory can be allocated to each level of model parameter set, and each level of model parameters and the corresponding parameter hash values ​​and physical video memory addresses are stored in the video memory mapping table.

[0065] In the embodiments of the present specification, a hash operation is performed on the model parameters in the intelligent learning model, and the hash values ​​of the model parameters are compared to determine whether the model parameters of the model to be deployed are repeated with the model parameters that have been deployed. If they are repeated, there is no need to allocate new physical video memory. The repeated model parameters are mapped to the corresponding physical video memory using virtual pointers to achieve sharing of the same content. The same model parameters do not need to be saved repeatedly, and new physical video memory addresses are allocated for non-duplicate model parameters, thereby achieving data sharing of the same content and greatly saving physical video memory space.

[0066] In some embodiments of the present specification, the video memory mapping table also includes parameter sizes of deployed model parameters, and sequentially matching the parameter hash values ​​of each model parameter set with the video memory mapping table to determine whether each model parameter set of the model to be deployed is the same as the deployed model parameters in the video memory mapping table, includes:

[0067] Match the parameter hash value of each model parameter set with the video memory mapping table in turn. If the parameter hash value of the model parameter set is the same as the parameter hash value of the target deployed model parameter in the video memory mapping table, compare the parameter size of the model parameter set with the parameter size of the target deployed model parameter. If the parameter size of the model parameter set is the same as the parameter size of the target deployed model parameter, determine that the model parameter set is the same as the target deployed model parameter.

[0068] In the specific implementation process, the video memory mapping table may also include the parameter size of the deployed model parameters. When checking the model parameter set of the model to be deployed, not only the parameter hash value of the model parameter set can be matched with the parameter hash value in the video memory mapping table, but also the parameter size can be compared. If the parameter hash value of a model parameter set of the model to be deployed is the same as the parameter hash value of the target deployed model parameter in the video memory mapping table, then compare the parameter size of the model parameter set and the target deployed model parameter. If the parameter size is also the same, it is determined that the model parameter set is the same as the target deployed model parameter. Among them, the target deployed model parameter can be understood as a deployed model parameter whose parameter hash value in the video memory mapping table is the same as the parameter hash value of a model parameter set of the model to be deployed.

[0069] Generally, if the hash values ​​are the same, the contents of the hash calculations are also the same. However, there are a few cases where the hash values ​​are the same but the contents are different. By comparing the hash values ​​and parameter sizes, when the hash values ​​and parameter sizes are the same, it is determined that the model parameter set is the same as the target deployed model parameters, which improves the accuracy of judging whether the model parameters are the same, thereby improving the accuracy of model deployment and avoiding the problem of model parameter loading errors due to misjudgment that affects system performance.

[0070] In some embodiments of this specification, the method further includes:

[0071] After determining that the parameter size of the model parameter set is the same as the parameter size of the target deployed model parameters, the model parameters in the model parameter set are compared byte by byte with the target deployed model parameters. If the model parameters in the model parameter set are the same as the bytes of the target deployed model parameters, it is determined that the model parameter set is the same as the target deployed model parameters.

[0072] In a specific implementation process, after determining that the parameter hash value and parameter size of a model parameter set of the model to be deployed are the same as the parameter hash value and parameter size of a target deployed model parameter in the video memory mapping table, the model parameters in the model parameter set of the model to be deployed can also be compared byte by byte with the target deployed model parameters. If the model parameters in the model parameter set are the same as the target deployed model parameters in terms of bytes, it can be determined that the model parameter set is the same as the target deployed model parameters.

[0073] For example: the model S to be deployed has 5 model parameter sets, among which the parameter hash value of model parameter set 1 is a. After comparing the parameter hash value a of model parameter set 1 with the parameter hash value in the video memory mapping table, it is found that the parameter hash value of a deployed model parameter P in the video memory mapping table is the same as the parameter hash value of model parameter set 1, then the deployed model parameter P can be used as the target deployed model parameter of model parameter set 1. Then compare the parameter size of model parameter set 1 with the parameter size of deployed model parameter P. If the parameter sizes of the two are also the same, compare the bytes of each model parameter in model parameter set 1 with the bytes of deployed model parameter P byte by byte. If the bytes of the two are also the same, it is determined that model parameter set 1 is the same as deployed model parameter P.

[0074] The embodiments of this specification perform a hash operation on the model parameters of the model to be deployed, compare the parameter hash value and parameter size of the model parameters with the data in the video memory mapping table, and then perform a byte-by-byte comparison of the parameters after the parameter hash value and parameter size are consistent with the data in the video memory mapping table, thereby improving the accuracy of repeated query results of the model parameters, thereby ensuring accurate model deployment and improving system performance.

[0075] In some embodiments of the present specification, the video memory mapping table also includes the video memory number of the deployed model parameter, and the matching of the parameter hash value of each model parameter set with the video memory mapping table in sequence to determine whether each model parameter set of the model to be deployed is the same as the deployed model parameter in the video memory mapping table includes:

[0076] Match the parameter hash value of each model parameter set with the video memory mapping table in turn. If the parameter hash value of the model parameter set is the same as the parameter hash value of the target deployed model parameter in the video memory mapping table, compare the video memory number corresponding to the model parameter set with the video memory number corresponding to the target deployed model parameter. If the video memory numbers are the same, it is determined that the model parameter set is the same as the target deployed model parameter.

[0077] In the specific implementation process, the video memory mapping table may also include the video memory number of the deployed model parameters, that is, the GPU number, and the video memory number may indicate on which GPU the model corresponding to the model parameter is loaded and deployed. Table 1 is the content of the video memory mapping table in a scenario example of this specification. As shown in Table 1, the video memory mapping table may include GPU# (i.e., video memory number), parameter size, parameter hash value, and physical video memory address. Among them, GPU# can be understood as the card number of the GPU. Each node, that is, each device, can have multiple cards, and each card stores a unique instance of the content. The parameter size can be understood as the size of the model parameters of the deployed model (layer by layer, multiple layers or the entire model), in bytes. The parameter hash value can be understood as the result of calculating the hash of the deployed model parameters (layer by layer, multiple layers or the entire model). For example, a typical MD5hash generates a summary of 128b, or SHA1 generates 160bit. Among them, the first three items, namely GPU#, parameter size, and parameter hash value, can be used as the key of the parameter hash value, and the value is the physical address of the deployed model parameters on GPU#.

[0078] Table 1: Video memory mapping table

[0079] GPU# Parameter size Parameter Hash Physical video memory address 0 50MB 0x**** 0xE0710000 0 24MB 0x**** 0xA4353400 1 12MB 0x**** 0x78E3B200 …… …… …… ……

[0080] When querying whether the model parameters of the model to be deployed are repeated with the parameters of the existing deployed model, not only can the parameter hash values ​​be compared to see if they are consistent, but also the video memory numbers can be compared to see if they are the same. For example: if it is determined that the parameter hash value of the model parameter set 1 of the model to be deployed is the same as the parameter hash value of the target deployed model parameter P in the video memory mapping table, then the video memory numbers of the two can be compared to see if they are the same. If the video memory numbers are also the same, then it can be considered that the models corresponding to the two model parameters are deployed on the same GPU, and it can be determined that the model parameter set 1 of the model to be deployed is the same as the target deployed model parameter P. Of course, you can also refer to the records of the above embodiments, and compare the parameter hash value, parameter size, video memory number, and parameter byte of the model parameter set of the model to be deployed with the data of the target deployed model parameter in the video memory mapping table in turn. If they are all the same, then it is determined that the two model parameters are the same.

[0081] In the embodiments of this specification, by comparing the memory labels corresponding to the model parameters, the models deployed on the same memory can be managed for memory allocation, that is, the model parameters with the same content on the same memory can share the physical memory, reducing the memory consumption of multiple models on one graphics card, and basically not affecting the performance. This avoids the problem of performance being affected due to the sharing of model parameters on different graphics cards.

[0082] Combine the following Figure 3 The allocation and management process of the physical video memory in the embodiments of this specification is specifically described. Figure 3As shown in the figure, the method in the embodiments of this specification may include:

[0083] 1. When the first model loads the model parameters, the model parameters of all models can be temporarily stored in the CPU (central processing unit), and then according to the configuration, the hash of the model parameters is calculated at different granularities such as layer by layer or continuously for multiple layers (even the entire model). For specific details, reference can be made to the description of the above embodiments, which will not be elaborated here.

[0084] 2. Duplicate check: Send <GPU#, parameter size, parameter hash value> to the GWM. The GWM queries the video memory mapping table, such as Table 1 above. Only when the GPU#, parameter size, and parameter hash value are all the same, it is considered that the model parameters already exist (when necessary, the parameter size and parameter hash value can be compared first to determine whether there is a duplication. If they are the same, then a byte-by-byte comparison can be made). If not, NULL is returned; otherwise, the physical video memory address is returned.

[0085] 3. If not, Model 1 allocates physical video memory (such as address addr1), maps it to virtual video memory such as ptr1, transfers the content of the model parameters from the CPU to addr1 of the GPU, and reports the corresponding information of GPU#, parameter size, parameter hash value, and addr1 to the GWM to update the video memory mapping table to record this corresponding information. By using the VMM (virtual machine monitor) interface, the driver layer can automatically track the reference count of addr1.

[0086] 4. After that, the process of Model 2 is similar. If the content is the same, the queried address (addr1) is returned. At this time, the model does not need to allocate physical video memory again, but maps the physical video memory addr1 to the virtual video memory corresponding to the model parameters of Model 2, such as ptr2, thus sharing the weights.

[0087] The video memory allocation processing method provided by the embodiments of this specification saves runtime video memory by sharing video memory based on the content value of the model parameters. In a practical example, it is determined that 45% can be saved, and the performance remains basically unchanged. Moreover, it can realize the sharing of model parameters of multiple models within the global scope of nodes, such as across multiple processes or multiple containers, and support multiple GPUs, thereby improving the target usage scenario. In addition, the identification and extraction of model parameters are at the layer level, with a finer granularity, so that partial identical layers in multiple models can be shared, also expanding the target scenario.

[0088] The various embodiments of the above methods in this specification are all described in a progressive manner. The same or similar parts among the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For the relevant parts, reference can be made to the partial description of the method embodiments.

[0089] Based on the above-mentioned video memory allocation processing method, one or more embodiments of this specification also provide a device for video memory allocation processing. The device may include a device (including a distributed system), software (application), module, plug-in, server, client, etc. that uses the method described in the embodiments of this specification and is combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiments of this specification is as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiments of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0090] Specifically, Figure 5 is a schematic diagram of the module structure of an embodiment of a video memory allocation processing device provided in this specification, such as Figure 5 As shown, the video memory allocation processing device provided in this specification may include:

[0091] A parameter acquisition module 51 is used to acquire a model parameter set of a model to be deployed;

[0092] A hash operation module 52 is used to perform a hash operation on each of the obtained model parameter sets to obtain a parameter hash value of each model parameter set;

[0093] The parameter duplication checking module 53 is used to match the parameter hash values ​​of each model parameter set with the video memory mapping table in turn to determine whether each model parameter set of the model to be deployed is the same as the deployed model parameters in the video memory mapping table; the video memory mapping table includes parameter hash values ​​of multiple deployed model parameters and physical video memory addresses corresponding to the deployed model parameters;

[0094] The video memory allocation module 54 is used to allocate a virtual video memory pointer to the model parameter set if it is determined that the model parameter set is the same as the deployed model parameters in the video memory mapping table, and map the physical video memory address of the deployed model parameters that are the same as the model parameter set to the virtual video memory pointer.

[0095] In some embodiments of this specification, the video memory allocation module is further used to:

[0096] If it is determined that the model parameter set is different from all deployed model parameters in the video memory mapping table, allocating a virtual video memory pointer and a physical video memory address to the model parameter;

[0097] The model parameters in the model parameter set are transmitted to the corresponding physical video memory address, and the physical video memory address allocated by the model parameter set is mapped to the corresponding virtual video memory.

[0098] The embodiments of this specification perform hash operations on the model parameters in the intelligent learning model, and determine whether the model parameters of the model to be deployed are repeated with the model parameters that have been deployed by comparing the hash values ​​of the model parameters. If they are repeated, there is no need to allocate new physical video memory. The repeated model parameters are mapped to the corresponding physical video memory using virtual pointers to achieve sharing of the same content. The same model parameters do not need to be saved repeatedly, and new physical video memory addresses are allocated for non-duplicate model parameters, thereby achieving data sharing of the same content and greatly saving physical video memory space.

[0099] It should be noted that the above-mentioned device may also include other implementations according to the description of the corresponding method embodiment. The specific implementation methods can refer to the description of the corresponding method embodiment above, and will not be described one by one here.

[0100] The embodiment of the present specification also provides a video memory allocation processing device, including: at least one processor and a memory for storing processor executable instructions, the processor executes the instructions to implement the video memory allocation processing method of the above embodiment, the method includes:

[0101] Get the model parameter set of the model to be deployed;

[0102] Performing a hash operation on each of the acquired model parameter sets to obtain a parameter hash value of each model parameter set;

[0103] Matching the parameter hash values ​​of each model parameter set with the video memory mapping table in sequence to determine whether each model parameter set of the model to be deployed is the same as the deployed model parameters in the video memory mapping table; the video memory mapping table includes parameter hash values ​​of multiple deployed model parameters and physical video memory addresses corresponding to the deployed model parameters;

[0104] If it is determined that the model parameter set is the same as the deployed model parameters in the video memory mapping table, a virtual video memory pointer is allocated to the model parameter set, and the physical video memory address of the deployed model parameters that are the same as the model parameter set is mapped to the virtual video memory pointer.

[0105] It should be noted that the above-mentioned device or system may also include other implementation modes according to the description of the method embodiment. The specific implementation modes may refer to the description of the relevant method embodiment, which will not be described one by one here.

[0106] The embodiment of this specification provides a video memory allocation processing system, Figure 6 FIG. 1 is a schematic diagram of the principle framework of a video memory allocation processing system in one embodiment of the present specification. Figure 6 As shown, the system includes: multiple graphics processors, namely GPUs, and a global model parameter management module, namely GWM (Global Weight Hash Manager). Each graphics processor includes multiple processes or containers, and each process or container includes at least one model to be deployed, wherein:

[0107] Each process or container is equipped with an inter-process communication module, such as Figure 6 As shown, the small square in the lower right corner of each process or container can represent an inter-process communication module. The global parameter management module is used to execute the method described in the above embodiment. The inter-process communication module is used to query whether the model parameters of each model to be deployed are repeated, so as to allocate video memory to the models to be deployed in multiple graphics processors.

[0108] A global parameter management module GWM can be deployed on each node, which is a separate process or container responsible for the management of all model parameter related data on the node (hash value, physical memory address, reference count, etc.) and provides IPC (Inter-Process Communication) services (such as UNIX socket). Figure 7 FIG. 1 is a flow chart of physical video memory allocation management in another embodiment of the present specification. Figure 7 As shown, the GWM can be configured and started first, and run for a long time. After the GWM is started on the node, the IPC service is provided to the outside world, and the model parameters of each model are queried for duplication while waiting for the model to be loaded. For details, please refer to the records of the above embodiments, which will not be repeated here. For example: online model deployment, loading model parameters, before allocating physical video memory for model parameters, first query whether there is duplication based on the hash of the content. If it has been loaded, the physical video memory is shared, and the virtual video memory pointer is mapped to the corresponding physical video memory address. If no duplicate content is found, new video memory is allocated, and hash and other data are sent to the GWM.

[0109] By calculating and comparing hashes during the model loading phase (which generates some additional computing overhead on the CPU, as model loading is often a one-time process and then runs for months or years), we can globally and precisely identify which models have the same parameters, and then map the same content to the same GPU memory. This reduces memory usage during the long-term operation of the model, allowing for the deployment of larger models or more instances. We also use the client-server model and introduce GWM to achieve global coordination of physical memory allocation management.

[0110] The video memory allocation processing device, equipment and system provided in this specification can also be applied to a variety of data analysis and processing systems. The system or server or terminal or device can be a separate server, or it can include a server cluster, system (including distributed system), software (application), actual operation device, logic gate circuit device, quantum computer, etc. that uses one or more of the methods or one or more embodiments of the system or server or terminal or device of this specification and a terminal device combined with necessary implementation hardware. The detection system for checking difference data may include at least one processor and a memory storing computer executable instructions, and the processor implements the steps of the method described in any one or more of the above embodiments when executing the instructions.

[0111] The method embodiments provided in the embodiments of this specification can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Taking running on a server as an example, Figure 8 1 is a hardware structure block diagram of a video memory allocation processing server in one embodiment of this specification. The computer terminal may be a video memory allocation processing server or a video memory allocation processing device in the above embodiment. Figure 8 The server 10 shown may include one or more (only one is shown in the figure) processors 100 (the processor 100 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a non-volatile memory 200 for storing data, and a transmission module 300 for communication functions. Those skilled in the art will understand that Figure 8 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 8 More or fewer plug-ins shown in, for example, may also include other processing hardware, such as a database or multi-level cache, GPU, or have Figure 8 Different configurations are shown.

[0112] The non-volatile memory 200 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the video memory allocation processing method in the embodiment of this specification. The processor 100 executes various functional applications and resource data updates by running the software programs and modules stored in the non-volatile memory 200. The non-volatile memory 200 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the non-volatile memory 200 may further include a memory remotely arranged relative to the processor 100, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0113] The transmission module 300 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission module 300 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 300 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0114] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0115] The methods or devices described in the above embodiments provided in this specification can implement business logic through computer programs and record them on storage media, and the storage media can be read and executed by a computer to achieve the effects of the solutions described in the embodiments of this specification.

[0116] The storage medium may include a physical device for storing information, which is usually a medium that digitizes the information and then stores it in an electrical, magnetic or optical manner. The storage medium may include: a device that stores information in an electrical energy manner, such as various memories, such as RAM, ROM, etc.; a device that stores information in a magnetic energy manner, such as a hard disk, a floppy disk, a magnetic tape, a magnetic core memory, a magnetic bubble memory, a USB flash drive; a device that stores information in an optical manner, such as a CD or a DVD. Of course, there are other readable storage media, such as quantum memory, graphene memory, etc.

[0117] The above-mentioned video memory allocation processing method or device provided in the embodiments of this specification can be implemented by a processor in a computer executing corresponding program instructions, such as using the C++ language of the Windows operating system to implement it on a PC, on a Linux system, or other systems such as Android and iOS system programming languages ​​to implement it on a smart terminal, as well as based on the processing logic of a quantum computer.

[0118] The embodiments of this specification are not limited to complying with industry communication standards, standard computer resource data update and data storage rules, or the situations described in one or more embodiments of this specification. Certain industry standards or slightly modified implementation plans based on the implementation described in the custom method or embodiment can also achieve the same, equivalent or similar, or predictable implementation effects after deformation of the above-mentioned embodiments. The embodiments obtained by using these modified or deformed data acquisition, storage, judgment, processing methods, etc. can still fall within the scope of the optional implementation plans of the embodiments of this specification.

[0119] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0120] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0121] For the convenience of description, the above platforms and terminals are described by dividing them into various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or plug-ins can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0122] These computer program instructions may also be loaded onto a computer or other programmable resource data updating device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0123] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. In the description of this specification, the description of the reference term "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0124] The above description is only an example of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. For those skilled in the art, one or more embodiments of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims.

Claims

1. A method for processing video memory allocation, the method comprising: Get the model parameter set of the model to be deployed; Performing a hash operation on each of the acquired model parameter sets to obtain a parameter hash value of each model parameter set; Matching the parameter hash values ​​of each model parameter set with the video memory mapping table in sequence to determine whether each model parameter set of the model to be deployed is the same as the deployed model parameters in the video memory mapping table; the video memory mapping table includes parameter hash values ​​of multiple deployed model parameters and physical video memory addresses corresponding to the deployed model parameters; If it is determined that the model parameter set is the same as the deployed model parameters in the video memory mapping table, a virtual video memory pointer is allocated to the model parameter set, and a physical video memory address of the deployed model parameters that are the same as the model parameter set is mapped to the virtual video memory pointer; Different model parameter sets are assigned different virtual display pointers; The step of obtaining a model parameter set of a model to be deployed includes: Obtaining model parameters of each level of the model to be deployed in sequence, and taking a set of model parameters of each level as a model parameter set; Alternatively, model parameters of consecutive preset levels of the model to be deployed are obtained in sequence, and a set of model parameters of consecutive preset levels is used as a model parameter set.

2. The method of claim 1, further comprising: If it is determined that the model parameter set is different from all deployed model parameters in the video memory mapping table, allocating a virtual video memory pointer and a physical video memory address to the model parameter; The model parameters in the model parameter set are transmitted to the corresponding physical video memory address, and the physical video memory address allocated by the model parameter set is mapped to the corresponding virtual video memory.

3. The method of claim 2, further comprising: The parameter hash value and the physical video memory address of the model parameter set are added to the video memory mapping table.

4. The method according to claim 1, wherein obtaining a model parameter set of a to-be-deployed model comprises: Calculate the parameter size of the model parameters within the specified range of the model to be deployed. If the parameter size is less than a preset threshold, add the adjacent model parameters in the specified range to the specified range, and calculate the parameter size of the model parameters in the new specified range. When the parameter size is greater than the preset threshold, take the model parameters in the corresponding specified range as a model parameter set.

5. The method according to claim 1, wherein the video memory mapping table further includes parameter sizes of deployed model parameters, and the step of sequentially matching the parameter hash values ​​of each model parameter set with the video memory mapping table to determine whether each model parameter set of the model to be deployed is the same as the deployed model parameters in the video memory mapping table, comprises: Match the parameter hash value of each model parameter set with the video memory mapping table in turn. If the parameter hash value of the model parameter set is the same as the parameter hash value of the target deployed model parameter in the video memory mapping table, compare the parameter size of the model parameter set with the parameter size of the target deployed model parameter. If the parameter size of the model parameter set is the same as the parameter size of the target deployed model parameter, determine that the model parameter set is the same as the target deployed model parameter.

6. The method of claim 5, further comprising: After determining that the parameter size of the model parameter set is the same as the parameter size of the target deployed model parameters, the model parameters in the model parameter set are compared byte by byte with the target deployed model parameters. If the model parameters in the model parameter set are the same as the bytes of the target deployed model parameters, it is determined that the model parameter set is the same as the target deployed model parameters.

7. The method according to claim 1, wherein the video memory mapping table further includes the video memory number of the deployed model parameters, and the step of sequentially matching the parameter hash value of each model parameter set with the video memory mapping table to determine whether each model parameter set of the model to be deployed is the same as the deployed model parameters in the video memory mapping table comprises: Match the parameter hash value of each model parameter set with the video memory mapping table in turn. If the parameter hash value of the model parameter set is the same as the parameter hash value of the target deployed model parameter in the video memory mapping table, compare the video memory number corresponding to the model parameter set with the video memory number corresponding to the target deployed model parameter. If the video memory numbers are the same, it is determined that the model parameter set is the same as the target deployed model parameter.

8. The method according to claim 1, wherein the models to be deployed are located in different threads, different processes or different containers.

9. The method of claim 1, further comprising: According to the virtual video memory pointer corresponding to the model parameter set, the sub-virtual video memory pointer of each model parameter in the model parameter set is determined by using a reference function, and the physical video memory address corresponding to the model parameter set is mapped to the sub-virtual video memory pointer of each model parameter.

10. A video memory allocation processing device, the device comprising: A parameter acquisition module is used to obtain a set of model parameters of the model to be deployed; A hash operation module, used to perform a hash operation on each of the acquired model parameter sets to obtain a parameter hash value of each model parameter set; A parameter duplication checking module is used to match the parameter hash values ​​of each model parameter set with the video memory mapping table in turn to determine whether each model parameter set of the model to be deployed is the same as the deployed model parameters in the video memory mapping table; the video memory mapping table includes parameter hash values ​​of multiple deployed model parameters and physical video memory addresses corresponding to the deployed model parameters; a video memory allocation module, configured to allocate a virtual video memory pointer to the model parameter set if it is determined that the model parameter set is the same as the deployed model parameters in the video memory mapping table, and map the physical video memory address of the deployed model parameters that are the same as the model parameter set to the virtual video memory pointer; different model parameter sets are allocated different virtual display pointers; The parameter acquisition module is specifically used to: sequentially acquire the model parameters of each level of the model to be deployed, and take the set of model parameters of each level as a model parameter set; or sequentially acquire the model parameters of consecutive preset levels of the model to be deployed, and take the set of model parameters of consecutive preset levels as a model parameter set.

11. The device according to claim 10, wherein the video memory allocation module is further configured to: If it is determined that the model parameter set is different from all deployed model parameters in the video memory mapping table, allocating a virtual video memory pointer and a physical video memory address to the model parameter; The model parameters in the model parameter set are transmitted to the corresponding physical video memory address, and the physical video memory address allocated by the model parameter set is mapped to the corresponding virtual video memory.

12. A video memory allocation processing device, comprising: At least one processor and a memory for storing processor-executable instructions, wherein when the processor executes the instructions, the method according to any one of claims 1 to 9 is implemented.

13. A video memory allocation processing system, the system comprising: Multiple graphics processors and a global model parameter management module, each graphics processor includes multiple processes or containers, each process or container includes at least one model to be deployed, wherein: An inter-process communication module is provided in each process or container, and the global model parameter management module is used to execute the method described in any one of claims 1 to 9, and query whether the model parameters of each model to be deployed are repeated through the inter-process communication module, so as to allocate video memory to the models to be deployed in multiple graphics processors.

Images