Large model memory optimization method and device, electronic equipment and storage medium
By only loading and optimizing the weights required for the target graph structure during inference model training, the problem of memory resource waste in existing technologies is solved and efficient use of memory resources is achieved.
Patent Information
- Application Number
- CN202510895935.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-30
- Publication Date
- 2025-10-17
AI Technical Summary
During the inference model training process, the existing technology loads the weights of all model layers into the memory each time, resulting in a large amount of unnecessary waste of memory resources.
Only the weights required by the target graph structure are loaded into memory and optimized based on the optimization items. The optimized weights are stored in memory and the unoptimized weights are released to disk. During the optimization process, only the weights required by the target graph structure are loaded.
It reduces the overhead of non-essential memory resources, saves the usage of memory resources, and improves the efficiency of memory resource use.
Smart Images
Figure CN120803707A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a large model memory optimization method and device, electronic equipment and storage medium. BACKGROUND
[0002] In recent years, the scale of inference models has grown exponentially. Inference models have evolved from early million parameters to today's trillion parameters, and the model weight data volume has reached terabytes (TB). When deploying inference models to actual application scenarios, reasonable use of memory resources is the key to improving the resource utilization efficiency of inference models.
[0003] In related technologies, as shown in FIG. 1, in the training process of an inference model, all weights required by each layer in the model are loaded in turn and stored in memory resources, realizing the hierarchical loading of the inference model during model inference. However, in each training process, all weights corresponding to all layers in the inference model are reloaded, and all loaded weights are stored in memory resources, thereby occupying a large amount of unnecessary memory resource overhead, and further causing waste of memory resources. Figure 1 SUMMARY
[0004] The present disclosure provides a large model memory optimization method and device, electronic equipment and storage medium to solve the problems in related technologies. In the optimization process, only the weights required by the target graph structure are loaded, and all weights are not loaded. The weights required by the target graph structure are stored in the memory resources, reducing the overhead of unnecessary memory resources, and further saving the amount of memory resources used.
[0005] According to a first aspect of the embodiments of the present disclosure, a large model memory optimization method is provided, which includes:
[0006] In response to the received optimization request information of the graph structure, determining a target graph structure according to at least one optimization item, wherein the graph structure is used to represent the connection relationship between different operator layers of a model;
[0007] Loading the weights of the target graph structure from the disk into the memory, and optimizing the target graph structure based on the loaded weights and the at least one optimization item to obtain an optimized target graph structure and optimized weights; wherein the optimized weights are stored in the memory;
[0008] After the optimization of the target graph structure is completed, the optimized weights in the memory are stored in the disk, and the unoptimized weights in the memory are released.
[0009] In some embodiments of the present disclosure, the loading of the weights of the target graph structure from the disk into the memory includes:
[0010] determine identification information corresponding to each of the operator layers in the target graph structure;
[0011] determine weights corresponding to each of the identification information in the disk; wherein the disk stores the weights according to a mapping relationship between the identification information of the operator layers and the weights;
[0012] load the weights corresponding to each of the identification information into the memory.
[0013] In some embodiments of the present disclosure, the storing the optimized weights in the memory into the disk comprises:
[0014] update the weights corresponding to the target graph structure stored in the disk based on the update content of the optimized weights.
[0015] In some embodiments of the present disclosure, the updating the weights corresponding to the target graph structure stored in the disk based on the update content of the optimized weights comprises:
[0016] update the weights corresponding to all the operator layers contained in the target graph structure in the disk.
[0017] In some embodiments of the present disclosure, the determining the target graph structure according to at least one optimization item comprises:
[0018] match a target graph structure template corresponding to each optimization item from a preset knowledge base; wherein the preset knowledge base is used to store graph structure templates corresponding to all graph structures related to the model;
[0019] determine the target graph structure according to the graph structure template.
[0020] In some embodiments of the present disclosure, after optimizing the target graph structure based on the loaded weights and the at least one optimization item, the method further comprises:
[0021] store the optimized weights in a preset target format, wherein the preset target format is a format that can be directly accessed without parsing the disk.
[0022] According to a second aspect of the present disclosure, an optimization device for a large model memory is provided, the device comprises:
[0023] a determination unit configured to determine a target graph structure according to at least one optimization item in response to received optimization request information of a graph structure, wherein the graph structure is used to represent the connection relationship between different operator layers of a model;
[0024] A loading unit, configured to load the weights of the target graph structure from a disk into a memory;
[0025] an optimization unit, configured to optimize the target graph structure based on the loaded weights and the at least one optimization item to obtain an optimized target graph structure and optimized weights; wherein the optimized weights are stored in the memory;
[0026] A first storage unit is configured to store the optimized weights in the memory in the disk after the target graph structure is optimized;
[0027] The release unit is used to release the unoptimized weights in the memory.
[0028] In some embodiments of the present disclosure, the loading unit includes:
[0029] A first determining module is used to determine identification information corresponding to all operator layers in the target graph structure;
[0030] A second determining module is configured to determine a weight corresponding to each identification information in the disk; wherein the weight is stored in the disk according to a mapping relationship between the identification information of the operator layer and the weight;
[0031] A loading module is used to load the weights corresponding to the respective identification information into the memory.
[0032] In some embodiments of the present disclosure, the first storage unit is further configured to update the weights corresponding to the target graph structure stored in the disk based on the updated content of the optimized weights.
[0033] In some embodiments of the present disclosure, the first storage unit is further used to update the weights corresponding to all operator layers included in the target graph structure in the disk.
[0034] In some embodiments of the present disclosure, the determining unit includes:
[0035] A matching module, configured to match the target graph structure template corresponding to each optimization item from a preset knowledge base; wherein the preset knowledge base is used to store graph structure templates corresponding to all graph structures related to the model;
[0036] The third determining module is configured to determine the target graph structure according to the graph structure template.
[0037] In some embodiments of the present disclosure, the apparatus further comprises:
[0038] A second storage unit is configured to store the optimized weight in a preset target format after optimizing the target graph structure based on the loaded weight and the at least one optimization item, wherein the preset target format is a format that can be directly accessed without parsing by the disk.
[0039] According to a third aspect of the present disclosure, an electronic device is provided, comprising:
[0040] at least one processor; and
[0041] a memory connected with the at least one processor; wherein
[0042] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first aspect.
[0043] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method of the first aspect.
[0044] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method of the first aspect.
[0045] In summary, the present disclosure provides a method for optimizing memory of a large model, which comprises: in response to received optimization request information of a graph structure, determining a target graph structure according to at least one optimization item, wherein the graph structure is used to represent the connection relationship between each operator between different operator layers of the model; loading the weight of the target graph structure from the disk into the memory; and optimizing the target graph structure based on the loaded weight and the at least one optimization item to obtain the optimized target graph structure and the optimized weight; wherein the optimized weight is stored in the memory, after the optimization of the target graph structure is completed, the optimized weight in the memory is stored in the disk, and the weight in the memory that is not optimized is released; the graph structure only loads the weight required by the target graph structure in the optimization process, and does not load all the weights; the weight required by the target graph structure is stored in the memory resource, which reduces the overhead of unnecessary memory resources, and further saves the usage amount of the memory resources.
[0046] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS
[0047] The accompanying drawings are used to better understand the present scheme and do not constitute a limitation on the present disclosure. Among them:
[0048] Figure 1 An optimization flowchart for a reasoning model reasoning training process;
[0049] Figure 2 A flowchart of a large model memory optimization method provided by an embodiment of the present disclosure;
[0050] Figure 3 A flowchart of another large model memory optimization method provided by an embodiment of the present disclosure;
[0051] Figure 4 A flowchart of another large model memory optimization method provided by an embodiment of the present disclosure;
[0052] Figure 5 A flowchart of another large model memory optimization method provided by an embodiment of the present disclosure;
[0053] Figure 6 A matching flowchart of graph structure optimization provided by an embodiment of the present disclosure;
[0054] Figure 7 An iterative flowchart of graph structure optimization provided by an embodiment of the present disclosure;
[0055] Figure 8 A structural schematic diagram of a large model memory optimization device provided by an embodiment of the present disclosure;
[0056] Figure 9 A structural schematic diagram of another large model memory optimization device provided by an embodiment of the present disclosure;
[0057] Figure 10 A schematic block diagram of an example electronic device 1000 provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0058] The embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numbers represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present disclosure, and cannot be understood as a limitation on the present disclosure.
[0059] In recent years, the scale of inference models has grown exponentially, and inference models have grown from millions of parameters in the early days to hundreds of billions of parameters today. The amount of model weight data has reached terabytes (TB). When deploying inference models to actual application scenarios, reasonable use of memory resources is the key to improving the resource utilization efficiency of inference models.
[0060] In the related art, in the inference model training process, all weights required by each layer in the model are loaded in turn and stored in the memory resource, realizing the hierarchical loading of the inference model during model inference. However, in each training process, all weights corresponding to all layers in the inference model are reloaded, and the loaded all weights are stored in the memory resource, thereby occupying a large amount of unnecessary memory resource overhead, and further causing waste of memory resources.
[0061] Therefore, in order to solve the problems in the related art, the present disclosure provides a large model memory optimization method, which comprises: in response to the received optimization request information of the graph structure, determining a target graph structure according to at least one optimization item, wherein the graph structure is used to represent the connection relationship between each operator between different operator layers of the model, loading the weights of the target graph structure from the disk into the memory, and optimizing the target graph structure based on the loaded weights and at least one optimization item to obtain an optimized target graph structure and optimized weights; wherein the optimized weights are stored in the memory, after the optimization of the target graph structure is completed, the optimized weights in the memory are stored in the disk, and the unoptimized weights in the memory are released, only the weights required by the target graph structure are loaded during the optimization process, all weights are not loaded, the weights required by the target graph structure are stored in the memory resource, the unnecessary memory resource overhead is reduced, and the use amount of memory resources is saved.
[0062] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing some steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily, in addition, the optional implementation manners in an embodiment can be combined arbitrarily. In addition, the embodiments can be combined arbitrarily, for example, some or all steps of different embodiments can be combined arbitrarily, an embodiment can be combined with the optional implementation manners of other embodiments.
[0063] In each embodiment of the present disclosure, the terms and / or descriptions of the embodiments are consistent and can be referred to each other if there is no special description and logical conflict, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.
[0064] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and not as a limitation on the present disclosure.
[0065] In the embodiments of the present disclosure, an element expressed in singular form, such as "a", "an", "the", "said", "the aforementioned", "the foregoing", "this", and the like, unless otherwise specified, can represent "one and only one", or can represent "one or more", "at least one", and the like. For example, in the case of using articles such as "a", "an", "the" in English, the noun after the article can be understood as a singular expression, or can be understood as a plural expression.
[0066] In some embodiments, the terms "in response to", "in response to determining", "in the case of", "when", "when", "if", "if", and the like can be replaced with each other.
[0067] In some embodiments, the terms "greater than", "greater than or equal to", "not less than", "more than", "more than or equal to", "not less than", "higher than", "higher than or equal to", "not lower than", "above", and the like can be replaced with each other, and the terms "less than", "less than or equal to", "not greater than", "less than", "less than or equal to", "not more than", "lower than", "lower than or equal to", "not higher than", "below", and the like can be replaced with each other.
[0068] The prefix words "first", "second", and the like in the embodiments of the present disclosure are only used to distinguish different description objects, and do not constitute a limitation on the position, order, priority, quantity, or content of the description objects. The description of the description objects should be referred to the description in the context of the claims or embodiments, and should not constitute an unnecessary limitation because of the use of the prefix words.
[0069] In the embodiments of the present disclosure, "a plurality of" means two or more.
[0070] In the embodiments of the present disclosure, the terms "import", "input", "read in", and the like can be replaced with each other.
[0071] In some embodiments, the apparatus and the like can be interpreted as physical or virtual, and the name thereof is not limited to the name recorded in the embodiments. The terms "apparatus", "equipment", "device", "circuit", "network element", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject", and the like can be replaced with each other.
[0072] In some embodiments, the terms "terminal", "terminal device", "user equipment (UE)", "user terminal", "mobile station (MS)", "mobile terminal (MT)", subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, and the like can be replaced with each other.
[0073] Figure 2 A flowchart of a large model memory optimization method provided by an embodiment of the present disclosure is shown in FIG. 1, which includes steps 101-103. Figure 2
[0074] Step 101, in response to the received optimization request information of the graph structure, determining a target graph structure according to at least one optimization item, wherein the graph structure is used to represent the connection relationship between different operator layers of the model.
[0075] In an embodiment of the present disclosure, the graph structure is optimized by the received optimization request information, wherein the optimization request information is instruction modification information for graph structure optimization, the graph structure is optimized according to the optimization request information through at least one optimization item, the target graph structure is determined based on the optimization result, and the graph structure is represented in the form of a directed acyclic graph structure representing the connection relationship between different operator layers of the model.
[0076] By receiving the optimization request information of the graph structure, a target graph structure is generated based on specific optimization items, including but not limited to template matching, weight binding and the like, flexible adaptation to diversified requirements is realized, optimization is performed according to at least one optimization item, and memory resource waste caused by indiscriminate optimization is avoided.
[0077] In step 102, the weights of the target graph structure are loaded from the disk into the memory, and the target graph structure is optimized based on the loaded weights and the at least one optimization item, to obtain an optimized target graph structure and optimized weights; wherein the optimized weights are stored in the memory.
[0078] In the embodiments of the present disclosure, the corresponding weights of the target graph structure are loaded from the disk, the loaded weights are stored in the memory, and the target graph structure is optimized based on the loaded weights and at least one optimization item, to obtain an optimized target graph structure and optimized weights, wherein the optimized weights are stored in the memory.
[0079] For example, the target graph structure "a->b->c" is loaded from the disk, the corresponding weights w1, w2 and w3 of the operator layers a, b and c are loaded, the loaded weights w1, w2 and w3 are stored in the memory, the target graph structure "a->b->c" is optimized based on the loaded weights w1, w2 and w3 and at least one optimization item, to obtain an optimized target graph structure "a->b->c" and weights w1, w2 and w3, and the optimized weights w1, w2 and w3 are stored in the memory. 1 ->b 1 ->c 1 ” and weights w1 1 , w2 1 , w3 1 . 1 1 1
[0080] It should be noted that the above examples are given for the purpose of understanding and are exemplary explanations, and are not intended to limit the specific content.
[0081] In step 103, after the optimization of the target graph structure is completed, the optimized weights in the memory are stored in the disk, and the unoptimized weights in the memory are released.
[0082] After the optimization of the target graph structure is completed, the optimized weights in the memory are stored in the disk, and the unoptimized weights in the memory are released.
[0083] After the optimization of the target graph structure "a->b->c" is completed, the optimized target graph structure "a->b->c" is obtained. 1 ->b 1 ->c1 ”, the optimized weights w1 1 , w2 1 , w3 1 in the memory are stored to the disk, and the unoptimized weights in the memory except the optimized weights w1 1 , w2 1 , w3 1 are released, and are read from the disk next time for target graph structure optimization.
[0084] As a possible implementation manner of the embodiment of the present disclosure, as shown in Figure 3 , the embodiment of the present disclosure provides a flowchart of a large model memory optimization method. The original Open Neural Network Exchange (ONNX) model is split into a graph structure and original weights, the graph structure is optimized, and in the graph structure optimization process, if the original weights need to be optimized, the optimized weights are stored to the graph structure, to obtain an optimized graph structure, and the unoptimized original weights are still stored to the position of the original weights. After the graph structure optimization is completed, the optimized weights in the optimized graph structure and the unoptimized original weights are combined to form new weights, and the new weights are stored to the disk.
[0085] In summary, according to the large model memory optimization method provided by the present disclosure, the method comprises: in response to received optimization request information of a graph structure, determining a target graph structure according to at least one optimization item, wherein the graph structure is used to represent the connection relationship between each operator between different operator layers of a model; loading the weights of the target graph structure from the disk to the memory, and optimizing the target graph structure based on the loaded weights and the at least one optimization item, to obtain an optimized target graph structure and optimized weights; wherein the optimized weights are stored in the memory, and after the target graph structure optimization is completed, the optimized weights in the memory are stored to the disk, and the unoptimized weights in the memory are released. In the optimization process, the graph structure only loads the weights required by the target graph structure, and does not load all the weights. The weights required by the target graph structure are stored to the memory resources, which reduces the overhead of unnecessary memory resources, and further saves the usage amount of memory resources.
[0086] Figure 4 Further, a flowchart of a large model memory optimization method is shown. Based on the embodiment shown in Figure 2 , step 102 is further explained, Figure 4 may include the following steps:
[0087] Step 201, determining the identification information corresponding to all operator layers in the target graph structure, respectively.
[0088] In the embodiments of the present disclosure, the identification information corresponding to all the operator layers in the target graph structure is determined.
[0089] For example, the identification information a1, b1 and c1 corresponding to the operator layers a, b and c in the target graph structure is determined.
[0090] It should be noted that the above examples are given for the purpose of facilitating understanding and are exemplary explanations, but not limitations on specific contents.
[0091] In step 202, the weight corresponding to each identification information in the disk is determined; wherein the weight is stored in the disk according to the mapping relationship between the identification information and the weight of the operator layer.
[0092] Based on the obtained each identification information, the weight corresponding to each identification information is determined, and the weight is stored through the mapping relationship between the identification information and the weight.
[0093] For example, the weight is stored through establishing the mapping relationship of "identification information a1->weight w1 1 ".
[0094] It should be noted that the above examples are given for the purpose of facilitating understanding and are exemplary explanations, but not limitations on specific contents.
[0095] In step 203, the weight corresponding to each identification information is loaded into the memory.
[0096] In the embodiments of the present disclosure, the weight corresponding to each identification information is loaded into the memory.
[0097] For example, the weight w1 1 corresponding to the identification information a1 is loaded into the memory.
[0098] It should be noted that the above examples are given for the purpose of facilitating understanding and are exemplary explanations, but not limitations on specific contents.
[0099] In order to further optimize the graph structure, when step 103 is executed, it further includes: updating the weight corresponding to the target graph structure stored in the disk based on the update content of the optimized weight.
[0100] When the graph structure is optimized, the operator layer is looped and traversed, in each round of looped and traversed, the graph structure is optimized by using at least one optimization item, and the weight corresponding to the target graph structure stored in the disk is updated according to the update content of the weight of the optimized graph structure.
[0101] In order to further optimize the graph structure, when updating the weight corresponding to the target graph structure stored in the disk based on the update content of the optimized weight, the weight corresponding to all operator layers contained in the target graph structure in the disk is also updated.
[0102] The weight corresponding to all operator layers contained in the target graph structure is loaded from the disk, optimized (application memory space, the preset ONNX tool supports searching for the required weight parameter from the ONNX weight file), and the weight corresponding to all operator layers is updated based on the weight corresponding to all operator layers.
[0103] Figure 5 Further, a flowchart of the optimization method of the large model memory according to the embodiments of the present disclosure is shown. Based on the embodiments shown in Figure 2 or Figure 4 The step 101 is further explained, Figure 5 may include the following steps:
[0104] Step 301: matching the target graph structure template corresponding to each optimization item from the preset knowledge base; wherein the preset knowledge base is used to store the graph structure templates corresponding to all graph structures related to the model.
[0105] Matching the target graph structure template corresponding to each optimization item from the preset knowledge base (graph level template matching), wherein the preset knowledge base is used to store the target graph structure templates related to the model, and the optimization items include but are not limited to matching templates, weight binding, etc.
[0106] The efficient reuse of the graph structure template is realized, the optimization efficiency is improved, the optimization time is shortened, the technical threshold and the labor cost are reduced, and reliable target graph structures can be quickly generated without professional algorithm knowledge. The standardization processing of different models in similar scenarios is supported, and the optimization ability can be formed through continuous sedimentation of new experience to form a virtuous cycle, invalid calculation is avoided to improve resource utilization, and strong support is provided for the rapid iteration and efficient deployment of the model.
[0107] Step 302: determining the target graph structure according to the graph structure template.
[0108] According to the graph structure template matched from the preset knowledge base, the target graph structure is determined.
[0109] In order to facilitate the matching process of the graph structure, as Figure 6As shown in the matching flowchart of the graph structure optimization, a template of a target graph structure is matched from a preset knowledge base. If the matching fails, graph structure template matching is continued. If the matching succeeds, weights are loaded from a disk to a memory. All operator layers in the memory are optimized. It is judged whether the weights corresponding to all operator layers need to be modified. If the weights corresponding to all operator layers need to be optimized, the optimized weights are stored in the memory until the optimization of the weights corresponding to all operator layers is completed. The optimized weights are stored in the optimized target graph structure. If the weights corresponding to all operator layers do not need to be optimized, the memory space is released. Next time, the unoptimized weights in the disk are directly read.
[0110] To further optimize the graph structure, after the step 102 is executed, the optimized weights are stored in a preset target format. The preset target format is a format that can be directly accessed without parsing the disk.
[0111] The optimized weights are stored in a preset target format. The target format after storage can be directly accessed without parsing.
[0112] To facilitate the storage mode of the target format, Figure 7 The iteration flowchart of the graph structure optimization is shown. Figure 7 For example, Conv and BatchNorm represent each operator layer of the model. Conv and BatchNorm include but are not limited to weights weight, bias, scale, running_mean, running_vard and the like. If the Conv operator layer is not connected to BatchNorm, the present scheme does not read the weight and bias of Conv to the memory. When the Conv+BatchNorm template is matched, the weights of Conv and BatchNorm are stored in the memory from the disk. The storage mode is 101. The storage mode of the related art is 102. The format of the read-in protobuf cannot be directly used for operation. It needs to be converted into the numpy format first. Protobuf and numpy represent different data formats. When the Conv+BatchNorm operator layer is optimized, the BatchNorm operator layer is deleted. The weight weight and bias of the Conv operator layer are updated. The updated weight weight and bias are stored in the memory in the numpy format. In the related art, the weight weight and bias are serialized to the protobuf format and stored in the mode 102. After the optimization of the target graph structure is completed, the unoptimized weights are uniformly loaded to the memory.
[0113] It should be noted that, Figure 7The exemplary description is given for the purpose of facilitating understanding without limiting the specific content.
[0114] The embodiments of the present disclosure can achieve the following beneficial effects:
[0115] 1. Take the whisper framework leading in the voice field as an example, by using the embodiments of the present disclosure, the optimization time is shortened from 21.15s to 13.6s, with a performance improvement of 30%.
[0116] 2. The memory after graph structure optimization decreases from 442.21M to 238.30M, with a 50% improvement, saving memory resources.
[0117] Figure 8 The structure diagram of the large model memory optimization device provided by the embodiments of the present disclosure is shown in Figure 8 As shown, it comprises a determination unit 81, a loading unit 82, an optimization unit 83, a first storage unit 84 and a release unit 85.
[0118] The determination unit 81 is configured to determine a target graph structure according to at least one optimization item in response to the received optimization request information of the graph structure, wherein the graph structure is used to represent the connection relationship between different operator layers of the model.
[0119] The loading unit 82 is configured to load the weight of the target graph structure from the disk to the memory.
[0120] The optimization unit 83 is configured to optimize the target graph structure based on the loaded weight and the at least one optimization item, to obtain an optimized target graph structure and an optimized weight; wherein the optimized weight is stored in the memory.
[0121] The first storage unit 84 is configured to store the optimized weight in the memory in the disk after the optimization of the target graph structure is completed.
[0122] The release unit 85 is configured to release the unoptimized weight in the memory.
[0123] In summary, the present disclosure provides an optimization device for large model memory, which comprises: in response to received optimization request information of a graph structure, determining a target graph structure according to at least one optimization item, wherein the graph structure is used to represent the connection relationship between each operator between different operator layers of the model, loading the weight of the target graph structure from the disk into the memory, and optimizing the target graph structure based on the loaded weight and the at least one optimization item to obtain an optimized target graph structure and an optimized weight; wherein the optimized weight is stored in the memory, after the optimization of the target graph structure is completed, the optimized weight in the memory is stored in the disk, and the unoptimized weight in the memory is released, the graph structure only loads the weight required by the target graph structure in the optimization process, and does not load all the weights, the weight required by the target graph structure is stored in the memory resource, the overhead of unnecessary memory resource is reduced, and the use amount of the memory resource is saved.
[0124] Further, in a possible implementation manner of an embodiment of the present disclosure, as shown in Figure 9 the loading unit 82 comprises:
[0125] The first determination module 821 is configured to determine the identification information corresponding to each operator layer in the target graph structure.
[0126] The second determination module 822 is configured to determine the weight corresponding to each identification information in the disk; wherein the weight is stored in the disk according to the mapping relationship between the identification information of the operator layer and the weight.
[0127] The loading module 823 is configured to load the weight corresponding to each identification information into the memory.
[0128] Further, in a possible implementation manner of an embodiment of the present disclosure, as shown in Figure 9 the first storage unit 84 is further configured to update the weight corresponding to the target graph structure stored in the disk based on the update content of the optimized weight.
[0129] Further, in a possible implementation manner of an embodiment of the present disclosure, as shown in Figure 9 the first storage unit 84 is further configured to update the weight corresponding to all operator layers contained in the target graph structure in the disk.
[0130] Further, in a possible implementation manner of an embodiment of the present disclosure, as shown in Figure 9 the determination unit 81 comprises:
[0131] The matching module 811 is configured to match a target graph structure template corresponding to each optimization item from a preset knowledge base. The preset knowledge base is configured to store graph structure templates corresponding to all graph structures related to the model.
[0132] The third determining module 812 is configured to determine the target graph structure according to the graph structure template.
[0133] Further, in a possible implementation of the embodiments of the present disclosure, as shown in Figure 9 The apparatus further includes:
[0134] The second storage unit 86 is configured to store the optimized weight in a preset target format after optimizing the target graph structure based on the loaded weight and the at least one optimization item, where the preset target format is a format that can be directly accessed without parsing by the disk.
[0135] It should be noted that the foregoing explanation and description of the method embodiments are also applicable to the apparatus of the embodiments of the present disclosure, and the principles are the same, which are not limited in the embodiments of the present disclosure.
[0136] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.
[0137] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.
[0138] As Figure 10As shown, the electronic device 1000 includes a computing unit 1001 that can perform various appropriate actions and processes in accordance with a computer program stored in a ROM (Read-Only Memory) 1002 or a computer program loaded into a RAM (Random Access Memory) 1003 from the storage unit 1008. Various programs and data required for the operation of the electronic device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An I / O (Input / Output) interface 1005 is also connected to the bus 1004.
[0139] A plurality of components in the electronic device 1000 are connected to the I / O interface 1005, including an input unit 1006 such as a keyboard, a mouse, and the like, an output unit 1007 such as various types of displays, a speaker, and the like, a storage unit 1008 such as a magnetic disk, an optical disk, and the like, and a communication unit 1009 such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0140] The computing unit 1001 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs various methods and processes described above, such as the optimization method of large model memory. For example, in some embodiments, the optimization method of large model memory can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded onto the RAM 1003 and executed by the computing unit 1001, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the aforementioned optimization method of large model memory by any other appropriate means, such as by means of firmware.
[0141] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a System on Chip (SOC), a Complex Programmable Logic Device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
[0142] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.
[0143] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include but are not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include one or more lines of electrical wire, portable computer diskette, hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory), or flash memory, fiber optics, CD-ROM (Compact Disc Read-Only Memory), optical storage device, magnetic storage device, or any suitable combination of the foregoing.
[0144] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.
[0145] The systems and techniques described here can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or a combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, and a blockchain network.
[0146] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is generally established using computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service ("Virtual Private Server", or simply "VPS"). The server can also be a server of a distributed system, or a server combined with a blockchain.
[0147] It should be noted that artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors of humans (such as learning, reasoning, thinking, planning, etc.), both hardware and software technologies. Artificial intelligence hardware technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc. several major directions.
[0148] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which is not limited herein.
[0149] The above detailed description does not limit the scope of the disclosure. Various modifications, combinations, sub-combinations and alternatives can be made to the detailed description. Any modification, equivalent replacement and improvement etc. made within the spirit and principle of the disclosure shall be included in the scope of the disclosure.
Claims
1. A method for optimizing large model memory, characterized in that: The method comprises: In response to the received graph structure optimization request information, determining a target graph structure according to at least one optimization item, wherein the graph structure is used to represent a connection relationship between different operator layers of the model; Loading the weights of the target graph structure from the disk into the memory, and optimizing the target graph structure based on the loaded weights and the at least one optimization item to obtain an optimized target graph structure and optimized weights; wherein the optimized weights are stored in the memory; After the target graph structure is optimized, the optimized weights in the memory are stored in the disk, and the unoptimized weights in the memory are released.
2. The method according to claim 1, characterized in that The step of loading the weights of the target graph structure from the disk into the memory includes: Determining identification information corresponding to all operator layers in the target graph structure; Determining a weight corresponding to each identification information in the disk; wherein the weight is stored in the disk according to a mapping relationship between the identification information of the operator layer and the weight; The weights corresponding to the respective identification information are loaded into the memory.
3. The method according to claim 2, characterized in that Storing the optimized weights in the memory in the disk includes: Based on the updated content of the optimized weights, the weights corresponding to the target graph structure stored in the disk are updated.
4. The method according to claim 3, characterized in that The updating of the weights corresponding to the target graph structure stored in the disk based on the updated content of the optimized weights includes: The weights corresponding to all operator layers contained in the target graph structure on the disk are updated.
5. The method according to any one of claims 1 to 4, characterized in that Determining the target graph structure according to at least one optimization item includes: Matching the target graph structure template corresponding to each optimization item from a preset knowledge base; wherein the preset knowledge base is used to store graph structure templates corresponding to all graph structures related to the model; The target graph structure is determined according to the graph structure template.
6. The method according to claim 1, characterized in that After optimizing the target graph structure based on the loaded weights and the at least one optimization item, the method further includes: The optimized weights are stored in a preset target format, wherein the preset target format is a format that can be directly accessed without the need for parsing by the disk.
7. A large model memory optimization device, characterized in that: include: a determining unit, configured to determine, in response to the received graph structure optimization request information, a target graph structure according to at least one optimization item, wherein the graph structure is used to represent a connection relationship between operators in different operator layers of the model; A loading unit, configured to load the weights of the target graph structure from a disk into a memory; an optimization unit, configured to optimize the target graph structure based on the loaded weights and the at least one optimization item to obtain an optimized target graph structure and optimized weights; wherein the optimized weights are stored in the memory; A first storage unit is configured to store the optimized weights in the memory in the disk after the target graph structure is optimized; The release unit is used to release the unoptimized weights in the memory.
8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.
10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.
Citation Information
Cited By
Model processing method, device and system and electronic equipment
CN121349368A