Model check point determination method and device
By analyzing the module storage occupancy information and association relationships of the large model, checkpoints are automatically determined, which solves the problem of excessive storage resource occupancy in large model training, and realizes efficient storage resource management.
Patent Information
- Application Number
- CN202411067224.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-12-06
AI Technical Summary
In big model training, there is a lack of systematization and automation when setting checkpoints, and relying on the experience of professionals, resulting in excessive storage resources occupied.
By analyzing the module storage occupancy information of the target model and the association relationship between the modules, candidate checkpoints are automatically determined, and a set of checkpoints that can save maximum storage space is selected.
It realizes automatic setting of checkpoints in large-scale model training, significantly saving storage resources, improving the accuracy and efficiency of checkpoint settings, and reducing the dependence on professional experience.
Smart Images

Figure CN119088541A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of information processing technology, and more specifically to a method and device for determining a model checkpoint. Background Art
[0002] With the rapid development of artificial intelligence, large models have a wider range of applications. Large models usually have more parameters and deeper hierarchical structures, and a large amount of storage and computing resources are required to train large models. Checkpointing is an optimization technique that can be used to train large models with limited storage resources. The method of setting a checkpoint is to only retain part of the calculation results in the forward propagation, such as retaining only the output information of the module as a checkpoint, and release other intermediate calculation results. When backpropagation occurs, re-run the forward propagation of the checkpoint part to restore the released intermediate calculation results, and use computing time in exchange for storage space. At present, when setting checkpoints in large models, it is mainly based on the experience of professionals. Summary of the invention
[0003] In view of this, this application provides the following technical solutions:
[0004] A method for determining a model checkpoint, comprising:
[0005] Obtaining a first parameter of storage occupancy information of at least one module constituting a target model, wherein the target model is used in the field of image, speech or natural language processing;
[0006] Based on the first parameter of each module and the association relationship between the modules, obtaining the second parameter of the storage occupancy information corresponding to the module if the module is set as a candidate checkpoint;
[0007] Based on the second parameter corresponding to the top-level module of the target model, each module in the target model is screened to determine a target module set, each module in the target module set is used to set a checkpoint, and the top-level module does not have an upper-level module in the target model.
[0008] Optionally, the first parameter represents the storage occupancy of the processing process corresponding to the module; the second parameter includes at least one parameter data group, each of the parameter data group includes the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module, the sum of the storage release amount of each candidate checkpoint in the candidate checkpoint scheme, and identification information representing whether the current module is a candidate checkpoint in the candidate checkpoint scheme; if the module is set as a candidate checkpoint, the storage release amount of the module is the storage occupancy of the processing process corresponding to the module.
[0009] Optionally, the obtaining, based on the first parameter of each module and the association relationship between the modules, a second parameter of the storage occupancy information corresponding to the module if the module is set as a candidate checkpoint includes:
[0010] Based on a first parameter corresponding to the current first module, determining first data corresponding to the first module if the first module is set as a candidate checkpoint, the first data including a first maximum storage occupancy of the first module and a first storage release amount of the first module;
[0011] Based on the first parameter corresponding to the second module, determining the second data corresponding to the second module if the second module is set as the candidate checkpoint; wherein the second module is a module of a next level corresponding to the current first module determined based on the association relationship between the modules; the second data includes the second maximum storage occupancy of the second module and the second storage release amount of the second module;
[0012] If the second module does not have a next-level module in the target model, determining a second parameter of the storage occupancy information corresponding to the current first module according to the first data and the second data;
[0013] If the second module has a next-level module in the target model, the second parameter of the storage occupancy information corresponding to the current first module is determined according to the first data and the second parameter corresponding to the second module.
[0014] Optionally, determining, according to the first data and the second data, a second parameter of storage occupancy information corresponding to the current first module includes:
[0015] Determine, based on the first maximum storage occupancy and the second maximum storage occupancy, an additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current first module;
[0016] Based on the first storage release amount and / or the second storage release amount, determine a parameter data group corresponding to a target candidate checkpoint solution having a maximum sum of storage release amounts of each candidate checkpoint in the candidate checkpoint solutions corresponding to the current first value;
[0017] Among them, the first value belongs to the selectable values of the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme; the candidate checkpoint scheme includes setting the first module as a candidate checkpoint, or if the second module is one, setting the second module as a candidate checkpoint, or if the second module of the next level corresponding to the first module includes at least one parallel sub-module, setting at least one combination of the parallel sub-modules as at least one of the candidate checkpoints.
[0018] Optionally, determining the second parameter of the storage occupancy information corresponding to the current first module according to the first data and the second parameter corresponding to the second module includes:
[0019] If the second module of the next level corresponding to the first module includes at least one parallel submodule, based on the target data corresponding to each of the parallel submodules and the first data, determine the second parameter of the storage occupancy information corresponding to the current first module, the target information corresponding to the parallel submodule includes third data or the second parameter corresponding to the parallel submodule, and the third data includes the third maximum storage occupancy of the parallel submodule and the third storage release amount of the parallel submodule;
[0020] If there is one second module at the next level corresponding to the first module, a second parameter of the storage occupancy information corresponding to the current first module is determined based on the first data and the second parameter corresponding to the second module.
[0021] Optionally, the method further comprises:
[0022] Constructing a target structure tree, wherein each node in the target structure tree corresponds to a module in the target model, and the hierarchical structure between the nodes in the target structure tree corresponds to the association relationship between the modules;
[0023] Determine a first node corresponding to the first module in the target structure tree;
[0024] If the first node is a non-leaf node of the target structure tree, determining the next-level node of the first node as the second module of the next level corresponding to the first module;
[0025] If the first node is a leaf node of the target structure tree, the first module does not have a corresponding second module at the next level.
[0026] Optionally, the obtaining, based on the first parameter of each module and the association relationship between the modules, a second parameter of the storage occupancy information corresponding to the module if the module is set as a candidate checkpoint includes:
[0027] Based on first parameters of each module of a first level of the target model, determining that if each module of the first level is set as a second parameter of each module of the first level of a candidate checkpoint, the module of the first level does not have a next level module in the target model;
[0028] Based on the second parameters of the modules of the first level and the first parameters of the modules of the second level, determining that if the modules of the second level are set as the second parameters of the modules of the second level of the candidate checkpoint, the second level is an upper level of the first level;
[0029] Based on the second parameters of each module of the second level and the first parameters of the module of the previous level corresponding to the second level, continue to execute the processing of determining the second parameters of the module of the previous level corresponding to the second level if the module of the previous level corresponding to the second level is set as a candidate checkpoint, until the second parameters of the top-level module of the topmost level of the target model are obtained if the top-level module is set as a candidate checkpoint.
[0030] Optionally, the filtering of the modules in the target model based on the second parameter corresponding to the top-level module of the target model to determine the target module set includes:
[0031] Constructing a target structure tree, wherein each node in the target structure tree corresponds to a module in the target model, and the hierarchical structure between the nodes in the target structure tree corresponds to the association relationship between the modules;
[0032] Determine a root node corresponding to the top-level module in the target structure tree, the root node being a node in the target structure tree for which there is no upper-level node;
[0033] Based on each parameter data group of the second parameter of the root node, determining the difference between the sum of the storage release amount and the corresponding additional storage occupation amount in each parameter data group;
[0034] The modules in the target model corresponding to each node in the node set corresponding to the parameter data group with the largest difference are determined as the target module set.
[0035] Optionally, the additional storage occupation introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module includes at least one of the following:
[0036] If the module releases storage space that is no longer used during the back propagation of the target model, the additional storage occupation introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module is determined according to the first parameter of the module corresponding to each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module and the sum of the storage release amounts of the module set having a target relationship with the module;
[0037] If the module does not release storage space during the back propagation of the target model, the additional storage occupation introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module is determined according to the first parameter of the module corresponding to each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module. A device for determining a model checkpoint includes:
[0038] A first acquisition unit, used to obtain a first parameter of storage occupancy information of at least one module constituting a target model, wherein the target model is used in the field of image, speech or natural language processing;
[0039] A second acquisition unit, configured to obtain, based on the first parameter of each module and the association relationship between the modules, a second parameter of the storage occupancy information corresponding to the module if the module is set as a candidate checkpoint;
[0040] A determination unit is used to screen each module in the target model based on a second parameter corresponding to the top-level module of the target model to determine a target module set, wherein each module in the target module set is used to set a checkpoint, and the top-level module does not have an upper-level module in the target model. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0042] Figure 1 A flowchart of a method for determining a model checkpoint provided in an embodiment of the present application;
[0043] Figure 2 A schematic diagram of the hierarchy of modules in a target model provided in an embodiment of the present application;
[0044] Figure 3 A schematic diagram of a structure tree of another target model provided in an embodiment of the present application;
[0045] Figure 4A schematic diagram of the structure of a device for determining a model checkpoint provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0047] The terms "first" and "second" in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.
[0048] A method for determining a model checkpoint provided in an embodiment of the present application can be applied to application scenarios where a large model processes image data, voice data, or text data, etc., and can save storage resources by setting checkpoints for the corresponding models. The present application can automatically determine the module for setting the checkpoint by analyzing the storage occupancy information of each module of the model and the association relationship between the modules, without relying on the experience of professionals for setting, and the selected module for setting the checkpoint can save storage space to the maximum extent, solving the problem of excessive storage resource occupation.
[0049] See also Figure 1 , is a flow chart of a method for determining a model checkpoint provided in an embodiment of the present application, the method may include the following steps:
[0050] S101. Obtain a first parameter of storage occupancy information of at least one module constituting a target model.
[0051] The target model may be a large model for image, speech or natural language processing that requires checkpoints. Large models usually have large-scale parameters and complex computational structures, such as large language models, diffusion models or multimodal models. Based on the model structure of the target model, at least one module may be determined, for example, it may include a feature extraction module, a feature mapping module, a prediction module, etc.
[0052] When obtaining the storage occupancy information of at least one module of the target model, the storage occupancy information of each module in the target model can be obtained. Further, in order to save processing resources, the storage occupancy information of a specific type of module in the target model can also be obtained. Storage occupancy information refers to the storage space information occupied by each module in the large model during the actual training process, which can be the occupancy of memory information, the occupancy of video memory information, or the occupancy of a specific storage space, such as the occupancy of the storage space corresponding to the neural network processor (Neural-Network Process Units, NPU). Specifically, the storage occupancy of the large model includes the model parameters and training data of the machine model. In addition, the processing process during model training also occupies a large storage space. Among them, the processing process mainly includes intermediate calculation results. Taking video memory occupancy as an example, during model training, the intermediate calculation results of each module of the model in the forward propagation need to be saved in the video memory so that these intermediate calculation results can be used to calculate the gradient during back propagation. As the structure of the model becomes more and more complex, the video memory occupied by the intermediate calculation results will be larger. Therefore, it is necessary to solve this problem by setting checkpoints.
[0053] Usually, the large model in the actual application scenario is improved on the basis of the existing open source deep learning architecture, for example, the large model created by the PyTorch architecture, PyTorch is an open source deep learning framework for machine learning and deep learning. For the PyTorch architecture's own basic operators (such as Linear, Conv2d), no intermediate data will be generated during calculation, that is, the modules corresponding to these built-in basic operators under the architecture will not have the storage occupancy of the processing process corresponding to the module, and setting the checkpoint will not generate any benefits, that is, no corresponding storage release will be generated. Therefore, when setting the checkpoint in the embodiment of the present application, only the customized modules in the target model are considered.
[0054] For the convenience of description, in the embodiment of the present application, the storage occupancy of the processing process corresponding to the module is determined as the first parameter of the storage occupancy information. The storage occupancy of the processing process corresponding to the module can also be called the storage occupancy of the intermediate calculation results during module training, or the storage occupancy of the intermediate activation value, or the storage occupancy of information other than the output information (i.e., the output tensor tensor) in the module. The present application does not impose any restrictions on this.
[0055] Specifically, the forward propagation process of the target model without setting a checkpoint can be used to record the storage usage of the corresponding module in the target model during calculation. i , you can get its total storage usage For example, the total memory usage of a module can be obtained through the memory_allocated() function in the deep learning framework PyTorch, and the memory usage of the module output information can be calculated. Thus, the storage usage of the processing process corresponding to the module can be obtained, that is, the first parameter Because if the module is set as a checkpoint, it may release the storage occupancy corresponding to the first parameter, therefore, in some embodiments of the present application, the first parameter is also referred to as potential benefit, where: By determining the first parameter of the storage occupancy information of at least one module of the target model, the amount of storage release that may be brought about when the module is set as a checkpoint can be obtained, providing reference data for subsequent checkpoint determination plans, so that the process of automatically determining the checkpoint can be evaluated with real quantitative parameters, which is more accurate than manual settings.
[0056] S102: Based on the first parameter of each module and the association relationship between the modules, obtain the second parameter of the storage occupancy information corresponding to setting the module as the candidate checkpoint module.
[0057] The association relationship between the modules of the target model can represent the hierarchical relationship between the modules and the parallel relationship between the modules at the same level. For example, the modules at the upper level of module A of the target model include module B, the modules at the lower level of module A include module C, and the modules at the same level as module A include module D, etc. Therefore, according to the association relationship between the modules of the target model, the modules with upper and lower hierarchical relationships can be determined, that is, the inclusion relationship between the modules can be obtained. In this way, when determining the candidate checkpoint scheme, the modules with inclusion relationships can be excluded based on the association relationship between the modules, so that the setting of the checkpoint is more in line with the current model structure.
[0058] Specifically, setting a module as a candidate checkpoint means that it is expected to set the current module as a checkpoint, but when actually setting the module as a candidate checkpoint, in addition to setting the module itself as a candidate checkpoint, the next-level module corresponding to the module may also be set as a candidate checkpoint, thereby realizing the transfer of the state of the next-level module to the record information of the current module, so that the relevant data of the storage occupancy information of the next-level module can be obtained according to the record information of the current module, and the parameter data group finally presented in the second parameter is the relevant data corresponding to the candidate checkpoint scheme with the largest storage release of the current module and the corresponding next-level module.
[0059] In the embodiment of the present application, when determining whether the current module is set as a candidate checkpoint, it is necessary to consider the scheme in which the next-level module of the current module is set as a candidate checkpoint, because the process of calculating the second parameter of the current module (i.e., the scheme that can characterize the candidate checkpoint corresponding to the current module) depends on the second parameter of the next-level module corresponding to the current module. In order to make it possible for the modules included in the checkpoint setting scheme to not have an inclusion relationship, in the calculation process of the current module, the sum of the storage release amount corresponding to the current module is not calculated as the sum of the storage release amount of the current module and the storage release amount of each module in the next-level module corresponding to the candidate checkpoint scheme; instead, the larger of the current module storage release amount or the sum of the storage release amounts of each node of the current next-level module is taken as the value corresponding to the sum of the storage release amounts in the second parameter corresponding to the current module.
[0060] In order to quantify the video memory occupancy information when the module in the target model is set as a candidate checkpoint, in the embodiment of the present application, based on the first parameter of each module and the association between each module, a second parameter of the storage occupancy information corresponding to the module when the module is set as a candidate checkpoint is obtained. The second parameter can characterize the additional storage occupancy and storage release information introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the module being set as a candidate checkpoint, wherein the storage release is the storage resource that can be saved if the module is set as a candidate checkpoint. The additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme is based on the storage occupancy introduced by each module corresponding to these candidate checkpoints during the back propagation of the target model by recalculating the intermediate process, that is, it is determined according to the recalculation cost. The additional storage occupancy corresponds to the entire current candidate checkpoint scheme, and the additional storage occupancy corresponding to the candidate checkpoint scheme obtained by different module combinations is different. The first case is that for the additional storage occupancy introduced by each candidate checkpoint in a candidate checkpoint scheme, the storage occupancy brought by the module with the largest recalculation cost in the candidate checkpoint scheme is mainly considered. For the module set as a candidate checkpoint in the back propagation, the case where the recalculation cost is the highest is to re-store its corresponding processing process, that is, to re-store its intermediate calculation process. Therefore, the storage occupancy of the processing process corresponding to this module can be Determine the additional storage occupancy introduced for each candidate checkpoint in the current candidate checkpoint scheme. For example, the current candidate checkpoint scheme includes modules 1, 2, and 3, the storage occupancy of the process processed by module 1 is 3, the storage occupancy of the process processed by module 2 is 5, and the storage occupancy of the process processed by module 3 is 1. The additional storage occupancy introduced by this scheme during back propagation recalculation is, in the worst case, the maximum recalculation occupancy introduced by module 2 due to recalculation, that is, module 2 stores its corresponding processing process again. Therefore, the additional storage occupancy introduced during back propagation recalculation in this example is the storage occupancy of the processing process corresponding to module 2, which is 5.
[0061] The above is a worst-case scenario, but in the intermediate process of back-propagation recalculation set as candidate checkpoints, there are other cases where the module calculated after the current module calculates releases its storage space. The storage space released by each module can be or That is, the additional storage usage introduced by each candidate checkpoint in the current candidate checkpoint scheme is based on the storage usage of the processing process corresponding to each module. Subtract the sum of the total storage space released by each module calculated later. It should be noted that "after the current module is calculated" in the above description refers to the module before in the reverse propagation, that is, after in the forward propagation, after the current module is calculated. For example, in the forward propagation, after calculating module A and A's lower module A-1, module B and B's lower modules B-1 and B-2 will be calculated. After calculating module B and lower modules B-1 and B-2, module C will be calculated, and so on. In the reverse propagation, after recalculating module C, module B and lower modules will be recalculated, and then module A and lower modules will be recalculated. If the current module is module A-1, then the modules after the current module is calculated are module B and module C. If the current module is module B-1, then the modules after the current module is calculated are module B-2 and module C.
[0062] Specifically, the second parameter includes at least one parameter data group, each parameter data group includes the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module, the sum of the storage release amounts of each candidate checkpoint in the candidate checkpoint scheme, and identification information representing whether the current module is a candidate checkpoint in the candidate checkpoint scheme.
[0063] Among them, the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module has been described in the above embodiment. The sum of the storage release amounts of each candidate checkpoint in the candidate checkpoint scheme is obtained by adding the storage release amounts of each candidate checkpoint in the current candidate checkpoint scheme. The storage release amount of each candidate checkpoint is usually the storage occupancy of the processing process corresponding to the module corresponding to the checkpoint. For example, the current candidate checkpoint scheme sets modules E, F and G as candidate checkpoints. If the storage release amount of module E is e, the storage release amount of module F is f and the storage release amount of module G is g, then the sum of the storage release amounts of each candidate checkpoint in the current candidate checkpoint scheme is (e+f+g). The second parameter includes the identification information in the parameter data group that indicates whether the current module is a candidate checkpoint in the candidate checkpoint scheme, including the first identification information that the current module is a candidate checkpoint in the candidate checkpoint scheme, such as the first identification information can be set to 1; and can also include the second identification information that the current module is not a candidate checkpoint in the candidate checkpoint scheme, such as the second identification information can be set to 0. For example, when determining the candidate checkpoint scheme corresponding to module A, if the modules set as candidate checkpoints in the candidate checkpoint scheme include module A, then the identification information in the candidate checkpoint scheme indicating whether the current module is a candidate checkpoint in the candidate checkpoint scheme is 1, otherwise it is 0.
[0064] If the next-level module corresponding to module A is module B, and there is no next-level module corresponding to module B. At this time, if module A is set as a candidate checkpoint, in addition to considering the storage occupancy information of module A, the storage occupancy information of module B also needs to be considered, so that the second parameter of the storage occupancy information corresponding to module A includes the parameter data group corresponding to module A and the parameter data group corresponding to module B. Among them, if a module is set as a candidate checkpoint, the storage release amount corresponding to the module is the storage occupancy of the processing process corresponding to the module. For example, if module A is set as a candidate checkpoint, it may save the storage occupancy of the processing process corresponding to it. Among them, the storage occupancy of the processing process corresponding to the module can be characterized as the storage occupancy of the intermediate calculation results during module training, and can also be called the storage occupancy of the intermediate activation value, as shown in step S101. That is, the potential benefits brought by the module. Through step S102, the second parameter of the storage occupancy information corresponding to the module in the target model being set as the candidate checkpoint module can be obtained. Since the second parameter can not only characterize the storage occupancy information of the current module being set as the candidate checkpoint, but also include a scheme for setting the corresponding next-level module as the candidate checkpoint, the association relationship between modules can be fully considered, so that the model structure of the current target model is more in line with the checkpoint setting process, thereby improving the accuracy of the checkpoint setting.
[0065] S103 . Based on the second parameter corresponding to the top-level module of the target model, filter each module in the target model to determine a target module set.
[0066] Wherein, each module in the target module set is used to set a checkpoint, and the top-level module does not exist in the target model. When the second parameter of the storage occupancy information corresponding to the module set as the candidate checkpoint module is obtained in step S102, each parameter data group included in the second parameter has covered the storage occupancy related data corresponding to the module itself as a candidate checkpoint, as well as the second parameter corresponding to the next-level module corresponding to the module. Wherein, taking the target structure tree corresponding to the target model as an example, the second parameter corresponding to the next-level module is to take the node corresponding to the next-level module as the current tree root, and the modules contained therein as all candidate checkpoint solutions for a submodel.
[0067] The second parameter corresponding to the top module of the target model already includes the node corresponding to the next level module corresponding to the top module as the current tree root, and the modules contained in it are all candidate checkpoint solutions as a submodel. Therefore, it is only necessary to screen the modules in the target model according to the second parameter of the top module to determine the target module set. Specifically, the second parameter of the top module includes at least one parameter data group, and the optimal target parameter data group can be obtained according to the data in each parameter data group. The module set corresponding to the target parameter data group is the target module set. The parameter data group includes the sum of the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint solution corresponding to the current module and the storage release amount of each candidate checkpoint in the candidate checkpoint solution, so that the difference between the sum of the storage release amount and the additional storage occupancy introduced by each candidate checkpoint can be calculated, and the difference is determined as the actual benefit corresponding to the current candidate checkpoint solution, that is, the actual storage release amount (or the actual amount of storage resources saved), so that the module set corresponding to the parameter data group with the largest difference is selected as the target module set, so that when the checkpoint is set in each module in the target module set, the amount of storage resources saved is the largest.
[0068] It can be seen that in the embodiment of the present application, based on the first parameter of at least one module included in the target model to store occupancy information and the association relationship between the modules, a second parameter of the storage occupancy information corresponding to the module being set as a candidate checkpoint module can be obtained, and the second parameter is used as a quantitative indicator for screening checkpoint setting schemes, so that the checkpoint determination process can be automatically screened based on the second parameter, without relying on professional debugging, and the screening efficiency is higher. In addition, the association relationship between the modules included in the model is taken into account, and the corresponding checkpoint setting scheme can be obtained for any model, with a wider range of applications and higher accuracy.
[0069] The following describes the relevant implementation methods of the embodiments of the present application in conjunction with an exemplary target model.
[0070] For the sake of convenience, we will combine the first case mentioned above, that is, the storage usage of the processing process corresponding to the module The additional storage usage introduced by each candidate checkpoint in the current candidate checkpoint scheme is used as an example to illustrate. And for the module in the target model that may be set as a checkpoint (such as a custom module in the target model), a data representing the storage usage and storage release amount if it is set as a candidate checkpoint is determined, which facilitates the subsequent description of related information screening and processing. Among them, the additional storage usage introduced by each candidate checkpoint in the candidate checkpoint scheme, that is, the recalculated storage usage introduced by the recalculation process of each candidate checkpoint in the candidate checkpoint scheme in the back propagation can be called potential cost, and correspondingly, if the module is set as a candidate checkpoint, its corresponding storage release amount can be called potential benefit.
[0071] Specifically, in one implementation of the embodiment of the present application, based on the first parameter of each module and the association relationship between the modules, obtaining the second parameter of the storage occupancy information corresponding to the module if the module is set as the candidate checkpoint module includes:
[0072] S201. Based on a first parameter corresponding to the current first module, determine first data corresponding to the first module if the first module is set as a candidate checkpoint first module.
[0073] S202. Based on the first parameter corresponding to the second module, determine the second data corresponding to the second module if the second module is set as the candidate checkpoint.
[0074] S203: If the second module does not have a next-level module in the target model, determine a second parameter of the storage occupancy information corresponding to the current first module according to the first data and the second data.
[0075] Alternatively, S204, if the second module has a next-level module in the target model, determine the second parameter of the storage occupancy information corresponding to the current first module according to the first data and the second parameter corresponding to the second module.
[0076] The second module is a module of the next level corresponding to the current first module, determined based on the association relationship between the modules. The first data includes the first maximum storage occupancy of the first module and the first storage release of the first module; the second data includes the second maximum storage occupancy of the second module and the second storage release of the second module.
[0077] In the embodiment of the present application, the data corresponding to each module can be determined based on the first parameter of each module, and the data represents the storage release amount when the module is set as a candidate checkpoint, and the maximum storage occupancy when the module is set as a candidate checkpoint. The storage release amount when the module is set as a candidate checkpoint is the storage occupancy of the processing process corresponding to the module. One case where the maximum storage occupancy when the module is set as a candidate checkpoint is that when the module is set as a candidate checkpoint, the processing process corresponding to the module needs to be recalculated in the back propagation recalculation, and the relevant data needs to be stored during the recalculation process. If the processing process corresponding to the module is stored again, then the maximum storage occupancy when the module is set as a candidate checkpoint is In other cases, during back propagation, the module that is calculated after this module releases its storage space. The storage space released by each module can be or At this time, when the module is set as a candidate checkpoint, the maximum storage occupancy is the storage occupancy of the processing process corresponding to the module. Subtract the sum of the total storage space released by each module calculated later. Therefore, the second parameter of the storage occupancy information corresponding to the module can be determined based on this data of the module. It should be noted that in the embodiment of the present application, this data is determined for the module in the target model (such as the first data of the first module in the embodiment or the second data of the second module) for the convenience of calculation and description. The data can characterize the storage occupancy that the corresponding module itself can release during forward propagation, and the corresponding maximum storage occupancy during reverse propagation recalculation. When calculating the second parameter, it can be calculated based on this data corresponding to the corresponding module, that is, the second parameter is generated for the candidate checkpoint scheme corresponding to the current module set as the candidate checkpoint.
[0078] For the convenience of description, in this embodiment of the present application, the maximum storage occupancy is represented as m c , denote the storage release amount as m p. The data corresponding to each module includes its corresponding maximum storage occupancy and storage release amount. For example, the first module corresponds to the first data, and the first data includes the first maximum storage occupancy and the first storage release amount. The second module corresponds to the second data, and the second data includes the second maximum storage occupancy and the second storage release amount. According to the data corresponding to each module (the data includes the maximum storage occupancy of the module and the storage release amount of the module), the second parameter corresponding to the module can be determined, and the second parameter includes at least one parameter data group, and each parameter data group represents a candidate checkpoint setting scheme with the largest storage release amount for the current module under different additional storage occupancy. For example, the first module is module A, and the corresponding second module of the next level includes module B and module C. If module A is set as a checkpoint alone, its corresponding storage release amount m p is 12, and its corresponding maximum storage occupancy is m c is 12; if module B is set as a checkpoint alone, its corresponding storage release amount m p is 5, and its maximum storage capacity is m c is 5; if module C is set as a checkpoint alone, its corresponding storage release amount m p is 6, and its maximum storage capacity is m cis 6. At this time, regardless of whether there is an inclusion relationship between modules in the candidate checkpoint scheme, if these three modules are considered to be set as candidate checkpoints, the selectable value of the additional storage occupancy introduced by each candidate checkpoint is the corresponding maximum storage occupancy set for each module separately, that is, 5, 6, and 12 can be selected. If 5 is selected as the additional storage occupancy introduced by each candidate checkpoint in the current candidate checkpoint scheme, only module B can be selected as a candidate checkpoint separately, because the maximum storage occupancy of other modules exceeds 5 (that is, exceeds the additional storage occupancy introduced by each candidate checkpoint in the current candidate checkpoint scheme), which does not meet the setting rules. Therefore, when the additional storage occupancy introduced by each candidate checkpoint in the current candidate checkpoint scheme is 5, a corresponding parameter data group in the second parameter is (5, 5, 0). If 6 is selected as the additional storage occupancy introduced by each candidate checkpoint in the current candidate checkpoint scheme, module B can be a candidate checkpoint alone, module C can be a candidate checkpoint alone, or modules B and C can be set as candidate checkpoints at the same time. In these three cases, the maximum storage occupancy of the module does not exceed the additional storage occupancy introduced by each candidate checkpoint in the current candidate checkpoint scheme. When modules B and C are set as candidate checkpoints at the same time, the sum of the storage release amount is the largest. At this time, module A is not selected. When the additional storage occupancy introduced by each candidate checkpoint in the current candidate checkpoint scheme is 6, a corresponding parameter data group in the second parameter is (6, 11, 0); if 12 is selected, module B can be a candidate checkpoint alone, module C can be set as a candidate checkpoint alone, modules B and C can be set as candidate checkpoints at the same time, or module A can be set as a candidate checkpoint alone. In these cases, the maximum storage occupancy of the module does not exceed the additional storage occupancy introduced by each candidate checkpoint in the current candidate checkpoint scheme, which meets the setting rules. When module A is used as a candidate checkpoint, the maximum storage release amount among these is 12. When the additional storage occupation introduced by each candidate checkpoint in the current candidate checkpoint scheme is 12, the corresponding parameter data group in the second parameter is (12, 12, 1). Therefore, the second parameter corresponding to module A includes the parameter data groups (5, 5, 0), (6, 11, 0), and (12, 12, 1).
[0079] If module A is the top-level module of the current model, that is, module A does not have a corresponding upper-level module, the final checkpoint setting plan can be determined according to each parameter data group in the second parameter corresponding to module A, that is, the target module set is determined so that the checkpoints are set for each module in the target module set.
[0080] It should be noted that the above is for the first case, that is, the storage usage of the processing process corresponding to the module This example uses the additional storage usage introduced by each candidate checkpoint in the current candidate checkpoint scheme as an example, that is, the maximum storage usage corresponding to each module For other cases, the additional storage usage introduced by each candidate checkpoint in the current candidate checkpoint scheme is based on the storage usage of the processing process corresponding to each module. Subtract the total storage space released by each module calculated later, that is, the maximum storage space occupied by each module m c Should be -The sum of the total storage space released by each module calculated thereafter. In this case, it is also possible to determine the target candidate checkpoint scheme corresponding to each possible value of the additional storage occupation when the sum of the storage release amounts of each candidate checkpoint of the candidate checkpoint scheme corresponding to each possible value of the additional storage occupation is the largest, and the parameter data group corresponding to the target candidate checkpoint scheme corresponding to each selectable value is used as the parameter data group in the second parameter. Thus, the corresponding second parameter is obtained when the module is set as a candidate checkpoint. The processing process can refer to the processing process in the above-mentioned example description for the first case, which will not be described in detail here.
[0081] When determining the second parameter of the storage occupancy information corresponding to the current first module if the current first module is set as a candidate checkpoint, in addition to considering setting the first module itself as a candidate checkpoint, it is also necessary to consider the solution when the second module is set as a candidate checkpoint. Therefore, if there is no next-level module for the second module, the second parameter of the storage occupancy information corresponding to the first module can be determined based on the first data of the first module and the second data of the second module. Of course, if the second module has already calculated its corresponding second parameter in advance, the first data and the second parameter of the second module can also be used to determine the second parameter of the storage occupancy information corresponding to the first module. If there is also a next-level module for the second module, the second parameter corresponding to the first module can be determined based on the second parameter corresponding to the second module, and there is no need to determine it based on the second data of the second module itself, because the second parameter of the second module already includes the parameter data group corresponding to setting the second module itself as a candidate checkpoint, and the parameter data group corresponding to setting the next-level module of the second module as a candidate checkpoint.
[0082] It should be noted that the above process of determining the second parameter of the storage occupancy information corresponding to the current first module set as the first module of the candidate checkpoint is only an example. In the case of a relatively simple model structure, if there are only two layers of model structure, when determining the second parameter of the top-most first module, the second parameter of the second module of the next layer can be calculated without calculating the second parameter of the second module, but the second data of the second module can be used for calculation, which can save computing resources. Correspondingly, it is also possible to calculate the second parameter of the bottom-most module of the target model first, and then calculate the second parameter of the corresponding module layer by layer.
[0083] See also Figure 2 , is a hierarchical diagram of each module in a target model provided in an embodiment of the present application. Figure 2 The model module represents the target model Model whose checkpoints are to be determined, that is, it is the top-level module of the target model. The modules at the next level of the model module include modules module1 and module2. The modules at the next level of module module1 include modules module1-1, module1-2, and module1-3. The modules at the next level of module module2 include module module2-1. The modules at the next level of module module1-1 include modules module1-1-1. There is no module at the next level of module module1-2. The modules at the next level of module1-3 include modules module1-3-1 and module1-3-2.
[0084] If the current first module is module1-3, and the corresponding second modules are module1-3-1 and module1-3-2, and in this case, there is no corresponding next-level module for the second module, then the first data of module1-3, the second data of module1-3-1, and the second data of module1-3-2 can be used to determine the second parameter of the storage occupancy information of module1-3 when module1-3 is set as a candidate checkpoint. If the current first module is module1, and the corresponding second modules are module1-1, module1-2, and module1-3, then the first data of module1, the second parameter of module1-1, the second data of module1-2 (which can also be the second parameter of module1-2), and the second parameter of module1-3 are used to determine the second parameter of the storage occupancy information of module1 when module1 is set as a candidate checkpoint.
[0085] Further, in an implementation manner of the embodiment of the present application, determining the second parameter of the storage occupancy information corresponding to the current first module according to the first data and the second data in step S203 in the above embodiment may include the following steps:
[0086] S2031. Based on the first maximum storage occupancy and the second maximum storage occupancy, determine the selectable values of the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current first module.
[0087] S2032. Based on the first storage release amount and / or the second storage release amount, determine a parameter data group corresponding to a target candidate checkpoint scheme having the largest sum of storage release amounts of each candidate checkpoint in the candidate checkpoint schemes corresponding to the current first value.
[0088] Among them, when a module is set as a candidate checkpoint, the information of the processing process corresponding to the module may no longer be stored, that is, the intermediate activation values or intermediate calculation processes within the module will not be stored. However, when the target model is back-propagated, since it is necessary to infer the processing process corresponding to the module set as the checkpoint based on the existing data, recalculation of storage occupancy may be introduced in this process. In one case, when the module is set as a candidate checkpoint, the processing process corresponding to the module needs to be recalculated in the back propagation process. In this recalculation process, the relevant data needs to be stored. If the processing process corresponding to the module is stored again, then the maximum storage occupancy when the module is set as a candidate checkpoint is In other cases, during back propagation, the module that is calculated after this module releases its storage space. The storage space released by each module can be or At this time, when the module is set as a candidate checkpoint, the maximum storage occupancy is the storage occupancy of the processing process corresponding to the module. Subtract the sum of the total storage space released by each module calculated later. In this scheme, the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme can be determined according to the maximum storage occupancy corresponding to each module set as a candidate checkpoint in the current candidate checkpoint scheme. The maximum storage occupancy mainly considers the storage occupancy of the processing process corresponding to the module, or the storage occupancy of the processing process corresponding to the module and the storage space released by the module calculated after the module during the model back propagation. This is because regardless of whether the module is set as a checkpoint, its corresponding output information will be retained, and the storage occupancy of the output part will not affect the setting of the checkpoint scheme, so the storage occupancy of the module output part can be ignored first. The first value belongs to the selectable values of the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme, wherein the selectable value of the additional storage occupancy may be related to the maximum storage occupancy of each module included in the current candidate checkpoint scheme during back propagation recalculation. If three modules are included in the current candidate checkpoint scheme, there are three selectable values of the corresponding additional storage occupancy, which are the maximum storage occupancy corresponding to each module. However, the corresponding storage release amount under each maximum storage occupancy may be different. Therefore, it is necessary to calculate the scheme with the largest sum of storage release amounts under each selectable value as the parameter data group in the second parameter; the candidate checkpoint scheme includes setting the first module as a candidate checkpoint, or setting the second module as a candidate checkpoint if there is only one second module, or if the second module of the next level corresponding to the first module includes at least one parallel sub-module, setting at least one combination of the parallel sub-modules as at least one of the candidate checkpoints.
[0089] In this embodiment, among the schemes corresponding to the selectable values of the additional storage usage introduced by each candidate checkpoint in each candidate checkpoint scheme, only the parameter data group corresponding to the scheme with the largest sum of corresponding storage release amounts is retained, so that when the checkpoint is determined later, the storage release amount of the corresponding scheme is maximized. Figure 3 The corresponding target model illustrates this process. Figure 3 The corresponding target model includes module 1, module 1-1, module 1-2, module 1-1-1, and module 1-1-2. Among them, the modules of the next level corresponding to module 1 are module 1-1 and module 1-2; the modules of the next level corresponding to module 1-1 include module 1-1-1 and module 1-1-2, and modules 1-1-1 and module 1-1-2 belong to the parallel sub-modules of the next level corresponding to module 1-1.
[0090] Based on the maximum storage capacity of each module (in m c ) and storage release (in m pRepresented), determine the relevant data of each module, Figure 3 For example, module 1's m c =18,m p =18; m of module 1-1 c =10,m p =10; m of module 1-2 c =6,m p =6; m of module 1-1-1 c =3,m p =3; m of module 1-1-1 c =4,m p =4.
[0091] If the current first module is module 1-1, the second modules are modules 1-1-1 and 1-1-2. The first data of module 1-1 is m c =10,m p =10; the second data of module 1-1-1 is m c =3,m p =3; the second data of module 1-1-2 is m c =4,m p =4. Module 1-1-1 and module 1-1-2 do not have modules at the next level. Then according to the m corresponding to these three modules c , it can be obtained that the selectable values of the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current first module, i.e., module 1-1, can be 3, 4, or 10. The corresponding checkpoint scheme includes selecting module 1-1-1 as a candidate checkpoint, selecting module 1-1-2 as a candidate checkpoint, selecting module 1-1 as a candidate checkpoint, and selecting module 1-1-1 and module 1-1-2 as candidate checkpoints at the same time. When the first value is 3, only the scheme in which the maximum storage occupancy corresponding to the module cannot exceed 3 can be selected, that is, only module 1-1-1 can be selected as a candidate checkpoint, and module 1-1 is not selected (the identification information indicating that module 1-1 is not a candidate checkpoint in the candidate checkpoint scheme can be set to 0). At this time, the corresponding parameter data group is (3, 3, 0). When the first value is 4, only the scheme whose corresponding maximum storage occupancy cannot exceed 4 can be selected, that is, module 1-1-1 can be selected as a candidate checkpoint, module 1-1-2 can be selected as a candidate checkpoint, or module 1-1-1 and module 1-1-1 can be selected as candidate checkpoints at the same time. The sum of the storage release amounts corresponding to the candidate checkpoint schemes of selecting modules 1-1-1 and module 1-1-1 is 7, which is the largest compared with other candidate checkpoint schemes. At this time, module 1-1 is not selected (the identification information indicating that module 1-1 is not a candidate checkpoint in the candidate checkpoint scheme can be set to 0), m c=4, the corresponding parameter data set is (4,7,0). Similarly, when m c =10, when any module 1-1-1 and module 1-1-2 are selected, the corresponding storage release amount is small. If the sum of the storage release amounts of modules 1-1-1 and 1-1-2 is 7, and the storage release amount when module 1-1 itself is selected is 10, then module 1-1 is selected (the identification information indicating that module 1-1 is a candidate checkpoint in the candidate checkpoint scheme can be set to 1), then m c = 4, the corresponding parameter data set is (10, 10, 1). Therefore, the parameter data sets in the second parameter corresponding to module 1-1 are (3, 3, 0), (4, 7, 0) and (10, 10, 1).
[0092] Correspondingly, if the second module has a next-level module in the target model, in one implementation of the embodiment of the present application, determining the second parameter of the storage occupancy information corresponding to the current first module according to the first data and the second parameter corresponding to the second module in the above step S204 may include the following steps:
[0093] S2041: If the second module of the next level corresponding to the first module includes at least one parallel submodule, determine a second parameter of the storage occupancy information corresponding to the current first module based on the target information corresponding to each of the parallel submodules and the first data.
[0094] Alternatively, S2042: if the first module corresponds to one second module of the next layer, determine the second parameter of the storage occupancy information corresponding to the current first module based on the first data and the second parameter corresponding to the second module.
[0095] The target information corresponding to the parallel submodule includes third data or a second parameter corresponding to the parallel submodule, and the third data includes a third maximum storage occupancy of the parallel submodule and a third storage release amount of the parallel submodule.
[0096] See also Figure 3 , if the first module is module 1, the parallel submodules of the second module include module 1-1 and module 1-2, both of which belong to the next level of modules of the first module, that is, module 1-1 and module 1-2 belong to the same level of modules. When determining the second parameter of module 1, the second parameter of module 1-1, the first data of module 1, and the target data of module 1-2 can be used. Figure 3 The module 1-2 shown does not include the module at the next level, that is, the target data corresponding to the module 1-2 is the third data (m c =6,m p=6). If module 1-2 includes a module at the next level, the corresponding second parameter is used for processing. Similarly, if the module at the next level corresponding to module 1 only includes module 1-1, the second parameter of module 1-1 and the first data of module 1 (m c =18,m p =18), and determine the second parameter of module 1.
[0097] Referring to the process of determining the second parameter of module 1-1 in the above embodiment, the process of determining the second parameter of module 1 is similar to the process of determining the second parameter of module 1-1, which will not be described in detail here. Figure 3 The second parameters of module 1 include parameter data groups of (3, 3, 0), (4, 7, 0), (6, 13, 0), (10, 16, 0) and (18, 18, 1) respectively.
[0098] In one implementation of the embodiment of the present application, the additional storage occupation introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module includes at least one of the following:
[0099] If the module releases storage space that is no longer used during the back propagation of the target model, the additional storage occupation introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module is determined according to the first parameter of the module corresponding to each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module and the sum of the storage release amounts of the module set having a target relationship with the module;
[0100] If the module does not release storage space during back propagation of the target model, the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module is determined according to the first parameter of the module corresponding to each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module.
[0101] Specifically, the additional storage usage introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module is related to the storage space released by the module calculated after the module is calculated during the back propagation of the model.
[0102] Taking the neural network model as an example, the forward propagation in the neural network model refers to the process of transferring the input sample from the input layer to the output layer after the weighted summation of neurons in each layer and the activation function processing, which is used to obtain the prediction result of the model; back propagation refers to the process of calculating the gradient through the back propagation of the loss function according to the error between the model prediction result and the true value, and updating the model parameters according to the gradient, which is used to train the model to improve the prediction accuracy. For example, in the target model based on the pytorch deep learning architecture, the model will automatically release the intermediate variables that are no longer used during the back propagation, that is, the module calculated after the module calculates releases its storage space. The storage space released by each module can be or At this time, when the module is set as a candidate checkpoint, the maximum storage occupancy is the storage occupancy of the processing process corresponding to the module. Subtract the total storage space released by each module calculated later. The sum of the memory usage of the modules calculated after this module in the representation model. If the modules in the combination of these modules have an inclusion relationship, only the memory release of the outermost module is calculated. At this time, the memory usage of the module in the target model is Indicates that when the module is set as a candidate checkpoint, the maximum storage occupancy is the storage occupancy of the processing process corresponding to the module (i.e., the first parameter in the above embodiment) Then the additional storage usage introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module can be expressed as:
[0103]
[0104] If you need to keep the intermediate variables set outside the candidate checkpoints, the module in the target model does not release the storage space during the back propagation. Specifically, the module calculated after the current module will no longer release the storage space. At this time, the storage space occupied by the processing process corresponding to this module can be (i.e., the first parameter in the above embodiment) is determined as the additional storage occupancy introduced by each candidate checkpoint in the current candidate checkpoint scheme, which is expressed as:
[0105]
[0106] In order to obtain more detailed information about the modules in the target model, an attribute information set can be set for each module. The attribute information set may include the module number i, the total storage usage of the module, The potential benefit of this module being set as a candidate checkpoint (i.e., the first parameter of the module), the potential cost of the module if it is set as a candidate checkpoint (i.e. the maximum storage capacity of the module) and the list of next-level modules corresponding to the module, etc.
[0107] In order to clearly obtain the association relationship between each module in the target model, each module in the target model can be represented by constructing a structure tree.
[0108] First, construct a target structure tree. Each node in the target structure tree corresponds to a module in the target model. The hierarchical structure between the nodes in the target structure tree corresponds to the association relationship between the modules. Then determine the first node in the target structure tree that corresponds to the first module; if the first node is a non-leaf node in the target structure tree, determine the next-level node of the first node as the second module of the next level corresponding to the first module. If the first node is a leaf node in the target structure tree, the first module does not have a corresponding second module of the next level. Figure 2 Taking the corresponding target structure tree as an example, module1-1-1, module1-3-1 and module1-3-2 are leaf nodes of the target structure tree, and the others are non-leaf nodes of the target structure tree. It can be seen that the leaf nodes do not have nodes at the next level.
[0109] Specifically, if the first node is a leaf node of the target structure tree, the second parameter of the storage occupancy information corresponding to the first module can be determined based on the first data of the first node. Figure 3 , Figure 3 The leaf nodes of the corresponding target structure tree are module 1-1-1 and module 1-1-2. Therefore, the modules can be classified according to the m corresponding to each module. c and m p The values of determine the corresponding second parameters, so that the second parameters of module 1-1-1 are (3,3,1), and the second parameters of module 1-1-2 are (4,4,1). The specific processing process will not be described in detail here. Please refer to the above Figure 3 The processing process of obtaining the second parameter of module 1-1.
[0110] If the first node is a non-leaf node of the target structure tree, and the node at the next level of the first node is a leaf node of the target structure tree, based on the first data of the first node and the second data of the node at the next level of the first node, the second parameter of the storage occupancy information corresponding to the current first module is determined. This process can be referred to Figure 3 The processing process of determining the second parameter of module 1-1 will not be described in detail here.
[0111] If the first node is a non-leaf node of the target structure tree, and the node of the next level of the first node is a non-leaf node of the target structure tree, the second parameter of the storage occupancy information corresponding to the current first module is determined based on the first data of the first node and the second parameter of the node of the next level of the first node. Figure 3 The processing process of determining the second parameter of module 1 will not be described in detail here.
[0112] In order to save computing resources in the embodiment of the present application, a post-order traversal method is used to determine the second parameter of the video memory occupancy information corresponding to each level of the target model when the module is set as a candidate checkpoint. In this way, the relevant information of the next level module can be obtained based on the second parameter corresponding to the current module, without having to search for the basic data of the next level every time. Specifically, in one embodiment, based on the first parameter of each module and the association relationship between the modules, the second parameter of the storage occupancy information corresponding to the module when the module is set as a candidate checkpoint module is obtained, including:
[0113] Based on the first parameters of each module of the first level of the target model and the second parameters of each module of the first level of the candidate checkpoint, it is determined that if the modules of the first level are set as the second parameters of the modules of the first level of the candidate checkpoint, the modules of the first level do not exist in the target model as the modules of the next level; based on the second parameters of each module of the first level and the first parameters of each module of the second level, it is determined that if the modules of the second level are set as the second parameters of the modules of the second level of the candidate checkpoint, the second level is the previous level of the first level; based on the second parameters of each module of the second level and the first parameters of the modules of the previous level corresponding to the second level, the process of determining the second parameters of the modules of the previous level corresponding to the second level if the modules of the previous level corresponding to the second level are set as candidate checkpoints is continued until the second parameters of the top-level module of the top-most level of the target model is set as the top-level module of the candidate checkpoint are obtained.
[0114] Still Figure 3 For example, if neither module 1-1-1 nor module 1-1-2 has a module at the next level, the first data can be determined according to the first parameter of the module 1-1-1, that is, m c =3,m p =3; m of module 1-1-1 c =4,m p =4, we can calculate that the second parameter of module 1-1-1 is (3,3,1), and the second parameter of module 1-1-2 is (4,4,1). Figure 3 There is no next-level module in module 1-2. The data determined by its own first parameter, that is, the m of module 1-2 c =6,m p=6, it can be determined that if it is set as a candidate checkpoint, the second parameter of module 1-2 is (6,6,1). At the same time, module 1-2 is a peer module of module 1-1, and it will also be used when calculating the corresponding second parameter when module 1 is a candidate checkpoint. It should be noted that for the convenience of calculation, it will be calculated from the bottom layer to the top layer, so the calculation process of the second parameter of module 1-2 can also be put into the calculation of the second layer where module 1-1 is located.
[0115] After obtaining that the second parameter of module 1-1-2 is (3,3,1) and the second parameter of module 1-1-2 is (4,4,1), it is possible to determine the second parameter of the video memory occupancy corresponding to module 1-1 when module 1-1 is set as a candidate checkpoint, that is, the data determined according to the first parameter of module 1-1, that is, the m of module 1-1 c =10,m p =10, and the second parameter of module 1-1-2 is (3,3,1) and the second parameter of module 1-1-2 is (4,4,1), the second parameter of module 1-1 is determined, namely including (3,3,0), (4,7,0) and (10,10,1).
[0116] Similarly, continue to calculate the second parameter of the top-level module when the top-level module is set as the candidate checkpoint. Figure 3 There are only three levels in it, so the top-level module is module 1, and its corresponding second parameters are (3,3,0), (4,7,0), (6,13,0), (10,16,0) and (18,18,1).
[0117] After obtaining the second parameter of the storage occupancy information of the corresponding module when each level of the module is set as a candidate checkpoint, the target module set to be set as the checkpoint can be determined based on the second parameter of the top-level module. Since the second parameter of the module includes the relevant storage occupancy information of the candidate checkpoint scheme corresponding to the module of the next level, when determining the target module set, it is only necessary to select the modules represented by the parameter data groups corresponding to the second parameter of the top-level module (that is, the top-level module does not have a module of the previous level) as the candidate checkpoint scheme, and the module set corresponding to the optimal parameter data group is the target module set.
[0118] Taking the target structure tree to represent each module in the target model as an example, in one implementation of the embodiment of the present application, based on the second parameter corresponding to the top-level module of the target model, each module in the target model is screened to determine the target module set, including:
[0119] S301, construct a target structure tree.
[0120] Each node in the target structure tree corresponds to each node in the target model corresponds to a module in the target model, and the hierarchical structure between the nodes in the target structure tree corresponds to the association relationship between the modules;
[0121] S302: Determine the root node corresponding to the top-level module in the target structure tree.
[0122] The root node is a node in the target structure tree for which there is no upper-level node.
[0123] S303 . Based on each parameter data group of the second parameter of the root node, determine the difference between the sum of the storage release amount and the corresponding additional storage occupation amount in each parameter data group.
[0124] S304: Determine the modules in the target model corresponding to each node in the node set corresponding to the parameter data group with the largest difference as the target module set.
[0125] Still Figure 3 For example, the top-level module is module 1, and the corresponding second parameter includes parameter data groups (3, 3, 0), (4, 7, 0), (6, 13, 0), (10, 16, 0) and (18, 18, 1), respectively. The first number corresponding to each parameter group represents the additional storage occupation introduced by each candidate checkpoint in the candidate checkpoint scheme, the second number represents the sum of the storage release amounts of each candidate checkpoint corresponding to the candidate checkpoint scheme, and the third number represents whether module 1 is set as a candidate checkpoint. If module 1 is set as a candidate checkpoint, the third number is 1, otherwise it is 0. The difference between the sum of the storage release amounts and the corresponding additional storage occupation in each parameter data group is determined as the actual benefit corresponding to the candidate checkpoint scheme. The actual benefit corresponding to the parameter data group (3,3,0) is 3-3=0; the actual benefit corresponding to the parameter data group (4,7,0) is 7-4=3; the actual benefit corresponding to the parameter data group (6,13,0) is 13-6=7, and the actual benefit corresponding to the parameter data group (10,16,0) is 16-10=6; the actual benefit corresponding to the parameter data group (18,18,1) is 18-18=0. Therefore, it can be concluded that the parameter data group with the largest actual benefit (i.e., the largest corresponding difference) is (6,13,0), and its corresponding target module set includes module 1-1-1, module 1-1-2, and module 1-2, i.e., these modules are used to set checkpoints.
[0126] The method for determining a model checkpoint provided in an embodiment of the present application can determine a second parameter of the storage occupancy information of a module set as a candidate checkpoint based on a first parameter of the storage occupancy information of at least one module in the target model and the association relationship between the modules, thereby realizing the ability to quantify the storage occupancy information of a module set as a candidate checkpoint with a specific parameter indicator, and then realizing the process of automatically searching and determining a target module set based on the second parameter, so that the checkpoint setting process is no longer dependent on professionals, and the second parameter corresponding to the module set as a candidate checkpoint covers the relevant information of the next-level module of the module, so that the process of searching and determining a target module set is more efficient and applicable to any model and has a wider range of applications.
[0127] In another embodiment of the present application, a device for determining a model checkpoint is provided. Figure 4 , the device comprises:
[0128] A first acquisition unit 401 is used to obtain a first parameter of storage occupancy information of at least one module constituting a target model, wherein the target model is used in the field of image, speech or natural language processing;
[0129] A second acquisition unit 402 is used to obtain, based on the first parameter of each module and the association relationship between the modules, a second parameter of the storage occupancy information corresponding to the module if the module is set as a candidate checkpoint;
[0130] The determination unit 403 is used to screen each module in the target model based on the second parameter corresponding to the top-level module of the target model to determine a target module set, wherein each module in the target module set is used to set a checkpoint, and the top-level module does not have an upper-level module in the target model.
[0131] Optionally, the first parameter represents the storage occupancy of the processing process corresponding to the module; the second parameter includes at least one parameter data group, each of the parameter data group includes the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module, the sum of the storage release amount of each candidate checkpoint in the candidate checkpoint scheme, and identification information representing whether the current module is a candidate checkpoint in the candidate checkpoint scheme; if the module is set as a candidate checkpoint, the storage release amount of the module is the storage occupancy of the processing process corresponding to the module.
[0132] Optionally, the second acquiring unit includes:
[0133] A first determining subunit is configured to determine, based on a first parameter corresponding to a current first module, first data corresponding to the first module if the first module is set as a candidate checkpoint, the first data including a first maximum storage occupancy of the first module and a first storage release amount of the first module;
[0134] A second determination subunit is used to determine, based on the first parameter corresponding to the second module, second data corresponding to the second module if the second module is set as a candidate checkpoint; wherein the second module is a module of a next level corresponding to the current first module determined based on the association relationship between the modules; the second data includes a second maximum storage occupancy of the second module and a second storage release amount of the second module;
[0135] a third determining subunit, configured to determine, if the second module does not have a next-level module in the target model, a second parameter of the storage occupancy information corresponding to the current first module according to the first data and the second data;
[0136] The fourth determining subunit is used to determine the second parameter of the storage occupancy information corresponding to the current first module according to the first data and the second parameter corresponding to the second module if there is a module of the next level in the target model for the second module.
[0137] Optionally, the third determining subunit is configured to:
[0138] Determine, based on the first maximum storage occupancy and the second maximum storage occupancy, selectable values of additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current first module;
[0139] Based on the first storage release amount and / or the second storage release amount, determine a parameter data group corresponding to a target candidate checkpoint solution having a maximum sum of storage release amounts of each candidate checkpoint in the candidate checkpoint solutions corresponding to the current first value;
[0140] Among them, the first value belongs to the selectable values of the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme; the candidate checkpoint scheme includes setting the first module as a candidate checkpoint, or if the second module is one, setting the second module as a candidate checkpoint, or if the second module of the next level corresponding to the first module includes at least one parallel sub-module, setting at least one combination of the parallel sub-modules as at least one of the candidate checkpoints.
[0141] Optionally, the fourth determining subunit is configured to:
[0142] If the second module of the next level corresponding to the first module includes at least one parallel submodule, based on the target information corresponding to each of the parallel submodules and the first data, determine the second parameter of the storage occupancy information corresponding to the current first module, the target information corresponding to the parallel submodule includes third data or the second parameter corresponding to the parallel submodule, and the third data includes the third maximum storage occupancy of the parallel submodule and the third storage release amount of the parallel submodule;
[0143] If there is one second module at the next level corresponding to the first module, a second parameter of the storage occupancy information corresponding to the current first module is determined based on the first data and the second parameter corresponding to the second module.
[0144] Optionally, the device further comprises:
[0145] A structure tree construction unit, used to construct a target structure tree, each node in the target structure tree corresponds to a module in the target model, and the hierarchical structure between the nodes in the target structure tree corresponds to the association relationship between the modules;
[0146] A first node determining unit, configured to determine a first node corresponding to the first module in the target structure tree;
[0147] A second module determining unit, if the first node is a non-leaf node of the target structure tree, determining a node at a next level of the first node as a second module at a next level corresponding to the first module;
[0148] If the first node is a leaf node of the target structure tree, the first module does not have a corresponding second module at the next level.
[0149] Optionally, the second acquiring unit includes:
[0150] a fifth determining subunit, configured to determine, based on first parameters of each module of the first level of the target model, that if each module of the first level is set as a second parameter of each module of the first level of the candidate checkpoint, the module of the first level does not exist in the target model as a next-level module;
[0151] a sixth determining subunit, configured to determine, based on the second parameters of the modules of the first level and the first parameters of the modules of the second level, if the modules of the second level are set as candidate checkpoints, the second level being an upper level of the first level;
[0152] An execution subunit is used to continue the processing of determining the second parameters of the modules of the previous level corresponding to the second level if the modules of the previous level corresponding to the second level are set as candidate checkpoints based on the second parameters of the modules of the second level and the first parameters of the modules of the previous level corresponding to the second level, until obtaining the second parameters of the top-level module if the top-level module of the topmost level of the target model is set as the candidate checkpoint.
[0153] Optionally, the determining unit includes:
[0154] A construction subunit is used to construct a target structure tree, each node in the target structure tree corresponds to a module in the target model, and the hierarchical structure between the nodes in the target structure tree corresponds to the association relationship between the modules;
[0155] A seventh determining subunit is used to determine a root node corresponding to the top-level module in the target structure tree, wherein the root node is a node in the target structure tree for which there is no upper-level node;
[0156] an eighth determination subunit, configured to determine, based on each parameter data group of the second parameter of the tree root node, a difference between a sum of storage release amounts and a corresponding additional storage occupation amount in each parameter data group;
[0157] The ninth determination subunit is used to determine the modules in the target model corresponding to each node in the node set corresponding to the parameter data group with the largest difference as the target module set.
[0158] Optionally,
[0159] The additional storage occupation introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module includes at least one of the following:
[0160] If the module releases storage space that is no longer used during the back propagation of the target model, the additional storage occupation introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module is determined according to the first parameter of the module corresponding to each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module and the sum of the storage release amounts of the module set having a target relationship with the module;
[0161] If the module does not release storage space during back propagation of the target model, the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module is determined according to the first parameter of the module corresponding to each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module.
[0162] In another embodiment of the present application, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for determining a model checkpoint as described in any one of the above items is implemented.
[0163] In another embodiment of the present application, an electronic device is further provided, and the electronic device may include:
[0164] A memory, used to store applications and data generated by the operation of the applications;
[0165] A processor is used to execute the application program to implement the method for determining a model checkpoint as described in any one of the above items.
[0166] It should be noted that the specific implementation of the processor in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.
[0167] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0168] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0169] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0170] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for determining a model checkpoint, comprising: Obtaining a first parameter of storage occupancy information of at least one module constituting a target model, wherein the target model is used in the field of image, speech or natural language processing; Based on the first parameter of each module and the association relationship between the modules, obtaining the second parameter of the storage occupancy information corresponding to the module if the module is set as a candidate checkpoint; Based on the second parameter corresponding to the top-level module of the target model, each module in the target model is screened to determine a target module set, each module in the target module set is used to set a checkpoint, and the top-level module does not have an upper-level module in the target model.
2. According to the method described in claim 1, the first parameter represents the storage occupancy of the processing process corresponding to the module; the second parameter includes at least one parameter data group, each of the parameter data group includes the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module, the sum of the storage release amount of each candidate checkpoint in the candidate checkpoint scheme, and identification information representing whether the current module is a candidate checkpoint in the candidate checkpoint scheme; if the module is set as a candidate checkpoint, the storage release amount of the module is the storage occupancy of the processing process corresponding to the module.
3. The method according to claim 1 or 2, wherein the obtaining, based on the first parameter of each module and the association relationship between the modules, the second parameter of the storage occupancy information corresponding to the module if the module is set as a candidate checkpoint comprises: Based on a first parameter corresponding to the current first module, determining first data corresponding to the first module if the first module is set as a candidate checkpoint, the first data including a first maximum storage occupancy of the first module and a first storage release amount of the first module; Based on the first parameter corresponding to the second module, determining the second data corresponding to the second module if the second module is set as the candidate checkpoint; wherein the second module is a module of a next level corresponding to the current first module determined based on the association relationship between the modules; the second data includes the second maximum storage occupancy of the second module and the second storage release amount of the second module; If the second module does not have a next-level module in the target model, determining a second parameter of the storage occupancy information corresponding to the current first module according to the first data and the second data; If the second module has a next-level module in the target model, the second parameter of the storage occupancy information corresponding to the current first module is determined according to the first data and the second parameter corresponding to the second module.
4. The method according to claim 3, wherein determining the second parameter of the storage occupancy information corresponding to the current first module according to the first data and the second data comprises: Determine, based on the first maximum storage occupancy and the second maximum storage occupancy, selectable values of additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current first module; Based on the first storage release amount and / or the second storage release amount, determine a parameter data group corresponding to a target candidate checkpoint solution having a maximum sum of storage release amounts of each candidate checkpoint in the candidate checkpoint solutions corresponding to the current first value; Among them, the first value belongs to the selectable values of the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme; the candidate checkpoint scheme includes setting the first module as a candidate checkpoint, or if the second module is one, setting the second module as a candidate checkpoint, or if the second module of the next level corresponding to the first module includes at least one parallel sub-module, setting at least one combination of the parallel sub-modules as at least one of the candidate checkpoints.
5. The method according to claim 3, wherein determining the second parameter of the storage occupancy information corresponding to the current first module according to the first data and the second parameter corresponding to the second module comprises: If the second module of the next level corresponding to the first module includes at least one parallel submodule, based on the target information corresponding to each of the parallel submodules and the first data, determine the second parameter of the storage occupancy information corresponding to the current first module, the target information corresponding to the parallel submodule includes third data or the second parameter corresponding to the parallel submodule, and the third data includes the third maximum storage occupancy of the parallel submodule and the third storage release amount of the parallel submodule; If there is one second module at the next level corresponding to the first module, a second parameter of the storage occupancy information corresponding to the current first module is determined based on the first data and the second parameter corresponding to the second module.
6. The method according to claim 2, further comprising: Constructing a target structure tree, wherein each node in the target structure tree corresponds to a module in the target model, and the hierarchical structure between the nodes in the target structure tree corresponds to the association relationship between the modules; Determine a first node corresponding to the first module in the target structure tree; If the first node is a non-leaf node of the target structure tree, determining the next-level node of the first node as the second module of the next level corresponding to the first module; If the first node is a leaf node of the target structure tree, the first module does not have a corresponding second module at the next level.
7. The method according to claim 1, wherein the obtaining, based on the first parameter of each module and the association relationship between the modules, the second parameter of the storage occupancy information corresponding to the module if the module is set as a candidate checkpoint comprises: Based on first parameters of each module of a first level of the target model, determining that if each module of the first level is set as a second parameter of each module of the first level of a candidate checkpoint, the module of the first level does not have a next level module in the target model; Based on the second parameters of the modules of the first level and the first parameters of the modules of the second level, determining that if the modules of the second level are set as the second parameters of the modules of the second level of the candidate checkpoint, the second level is an upper level of the first level; Based on the second parameters of each module of the second level and the first parameters of the module of the previous level corresponding to the second level, continue to execute the processing of determining the second parameters of the module of the previous level corresponding to the second level if the module of the previous level corresponding to the second level is set as a candidate checkpoint, until the second parameters of the top-level module of the topmost level of the target model are obtained if the top-level module is set as a candidate checkpoint.
8. The method according to claim 2, wherein the step of screening each module in the target model based on the second parameter corresponding to the top-level module of the target model to determine the target module set comprises: Constructing a target structure tree, wherein each node in the target structure tree corresponds to a module in the target model, and the hierarchical structure between the nodes in the target structure tree corresponds to the association relationship between the modules; Determine a root node corresponding to the top-level module in the target structure tree, the root node being a node in the target structure tree for which there is no upper-level node; Based on each parameter data group of the second parameter of the root node, determining the difference between the sum of the storage release amount and the corresponding additional storage occupation amount in each parameter data group; The modules in the target model corresponding to each node in the node set corresponding to the parameter data group with the largest difference are determined as the target module set.
9. The method according to claim 2, wherein the additional storage usage introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module comprises at least one of the following: If the module releases storage space that is no longer used during the back propagation of the target model, the additional storage occupation introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module is determined according to the first parameter of the module corresponding to each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module and the sum of the storage release amounts of the module set having a target relationship with the module; If the module does not release storage space during back propagation of the target model, the additional storage occupancy introduced by each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module is determined according to the first parameter of the module corresponding to each candidate checkpoint in the candidate checkpoint scheme corresponding to the current module.
10. A device for determining a model checkpoint, comprising: A first acquisition unit, used to obtain a first parameter of storage occupancy information of at least one module constituting a target model, wherein the target model is used in the field of image, speech or natural language processing; A second acquisition unit, configured to obtain, based on the first parameter of each module and the association relationship between the modules, a second parameter of the storage occupancy information corresponding to the module if the module is set as a candidate checkpoint; A determination unit is used to screen each module in the target model based on a second parameter corresponding to the top-level module of the target model to determine a target module set, wherein each module in the target module set is used to set a checkpoint, and the top-level module does not have an upper-level module in the target model.
Citation Information
Cited By
Model inspection number determination method, device and equipment
CN121365055A
A model checking quantity determination method, device and equipment
CN121365055B
