Memory optimization method for distributed deep learning

Abstract

The invention provides a memory optimization method for distributed deep learning, and relates to the technical field of deep learning. The method comprises the following steps: constructing a double-cost index containing memory exchange strategy extra time overhead and re-calculation strategy extra time overhead and an optimized value index based on a re-calculation value index according to a time sequence relationship between a data object generation node and a calling node; determining an upstream or downstream device label of each computing device according to the computing duration of forward propagation and reverse propagation of the device; performing priority calculation on the input gradient of the downstream label equipment, and determining an overlapping calculation time window for triggering back propagation of the upstream label equipment; and differential memory optimization processing aiming at different equipment tags is implemented by utilizing an overlapping calculation time window and a two-dimensional cost index, a corresponding memory management scheme is determined, and collaborative improvement of the memory utilization rate and the calculation efficiency is realized.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, specifically a memory optimization method for distributed deep learning. Background Technology

[0002] With the rapid expansion of deep learning model parameter scale (such as GPT-3, SwingTransformer, etc.), the memory capacity of a single computing device is far from sufficient to support the training task of the entire model; therefore, distributed training architectures using technologies such as model parallelism and pipeline parallelism have become an inevitable choice for large-scale deep learning training.

[0003] However, distributed training of deep learning faces two core challenges: pipeline bubbles and memory bottlenecks. Pipeline bubbles are caused by strict data dependencies between devices, resulting in idle computing resources due to synchronization waiting, which significantly reduces training efficiency. Memory bottlenecks are caused by the surge in model size, which makes the memory required to store intermediate activation values ​​far exceed the device capacity. Although traditional "time-for-space" optimization methods can alleviate memory pressure, the additional overhead they introduce often exacerbates pipeline bubbles and creates performance conflicts.

[0004] Therefore, existing technologies for distributed deep learning training are prone to problems such as uneven memory utilization among devices, long communication latency, and idle computing resources, failing to fully leverage the powerful computing capabilities of distributed clusters. Developing a memory optimization method for distributed deep learning training, and systematically analyzing and resolving pipeline bubble and memory bottleneck issues, is therefore particularly urgent. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a memory optimization method for distributed deep learning.

[0006] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a memory optimization method for distributed deep learning, comprising: Based on the temporal relationship between data object generation nodes, calling nodes, and recovery nodes during the training process of deep learning models, a dual-cost index and an optimized value index are constructed. Based on the time relationship between forward and backward propagation of each computing device during the training of the deep learning model, the device label corresponding to each computing device is determined; the device label includes: upstream label and downstream label; The input gradient of the downstream tag device is preferentially calculated to determine the overlapping calculation time window used to trigger the back propagation of the upstream tag device; Based on the overlapping computation time window, the dual-cost index, and the optimized value index, differentiated memory optimization is performed for each tag device to determine the corresponding memory management scheme for each tag device.

[0007] Furthermore, the temporal relationship between the data object generation node, the calling node, and the recovery node includes: Based on the order of operations of nodes during the training of deep learning models, the data dependencies between nodes are analyzed to determine the node topology. Based on the node topology, the time statistics of the node calculation process are performed to obtain node calculation duration data. Data analysis is performed based on the computational operations executed by the nodes to identify the data objects related to the node computations. The data object This includes: parameter memory, intermediate feature maps, gradient data, and optimizer state; With data objects Based on the computational generation process during forward propagation, the computational generation node for each data object is determined. ; With data objects Based on the process of calculating gradients during backpropagation, the gradient generation nodes for each data object are determined. ; With data objects Based on the process of data recovery using memory swapping in backpropagation, the memory swapping data recovery node for each data object is determined. .

[0008] Furthermore, the construction process of the dual-cost index includes: Calculate the time when memory swapping for each data object begins and computation begins. Exchange the time when the computation is completed with the memory of this node. The time difference between them determines the additional time overhead of the memory swapping strategy for data objects. ; Calculate the time when the computation begins for each data object. The time when the calculation is completed at this node. The time difference between them determines the additional time overhead of the recalculation strategy. ; Additional time overhead based on the memory swapping strategy Additional time overhead of recalculation strategy This yields the double-cost index for each data object; Accordingly, the construction process of the optimized value index includes: Based on the memory usage of the data object The additional computation time required to recalculate this data object Calculate the ratio between memory usage and recompile time to determine the recompile value metric that characterizes the memory capacity released per unit of recompile time. ; The recalculated value index This serves as the index for the optimized value.

[0009] Furthermore, the method for determining the device tag corresponding to each computing device includes: Based on the forward propagation order of each computing device during the training of the deep learning model, the computing device at the beginning of the forward propagation order is determined as the initial upstream label device; For the target computing device in the forward propagation sequence, based on the forward propagation duration of the target computing device. Compared with backpropagation duration Summation is performed to determine the total usage time of the target computing device. ; Based on the backpropagation time of the upstream tag device located before the target computing device. Perform cumulative calculations to determine the backpropagation time threshold of the target computing device. ; Based on the total occupied time With the backpropagation duration threshold Based on the size relationship, the label of the target computing device is determined as either an upstream label or a downstream label; if the total occupancy time... Greater than the backpropagation time threshold If it is true, it is determined to be an upstream label; otherwise, it is determined to be a downstream label.

[0010] Furthermore, the method for determining the overlapping calculation time window includes: Based on the requirement of downstream tag devices to calculate their own input gradients during backpropagation, the backpropagation intermediate feature maps generated by each downstream tag device are filtered to obtain the corresponding set of backpropagation intermediate feature maps necessary for calculating the input gradients. Based on the backpropagation intermediate feature map set, perform swapping, removing or recalculating and releasing operations on non-essential feature maps to determine the available memory resources for priority computation of input gradients for each downstream label device; Based on the available memory resources, the input gradient corresponding to each downstream tag device is calculated first, and the input gradient calculation result used to trigger the backpropagation of the corresponding upstream tag device is determined. According to the non-blocking communication mechanism, the input gradient calculation result is sent to the corresponding upstream tag device, triggering the backpropagation calculation of the corresponding upstream tag device; Based on the backpropagation calculation process of the upstream tag device, the parameter gradient of the corresponding downstream tag device is calculated in parallel, and an overlapping calculation time window is determined to mask the calculation overhead of the parameter gradient of the corresponding downstream tag device.

[0011] Furthermore, the differentiated memory optimization process includes: performing a three-axis parallel memory optimization process for the upstream tag device; and performing a computationally decoupled memory optimization process for the downstream tag device.

[0012] Furthermore, the three-axis parallel memory optimization process is as follows: Based on the optimized value index, obtain the recalculated value index of each data object. ; The data object is recalculated according to the recalculated value index. Sort in descending order to obtain the recalculated value sequence; An optimized memory block sequence is constructed based on the recomputation value sequence, and data objects are added to the optimized memory block sequence in sequence according to the sorting results. The total memory usage of the hierarchy after the addition is calculated. Data objects are continuously added to the optimized memory block sequence until the total memory usage of the level is less than or equal to a preset memory optimization threshold, and the final optimized memory block sequence is determined. For the target data object in the optimized memory block sequence, calculate the space allocation time required to adopt the memory swapping strategy, and calculate the expected total memory usage at the expected level at the space allocation time point; The expected total memory usage at each level is less than or equal to the preset memory optimization threshold as the condition for implementing the memory swapping strategy. Based on the implementation conditions of the memory swapping strategy and the relationship between the additional time overhead of the memory swapping strategy and the recalculation strategy, optimization processing is performed on the target data object; When the conditions for implementing the memory swapping strategy are met, and the additional time overhead of the memory swapping strategy is less than the time overhead of the recalculation strategy, the memory swapping process is executed. When the conditions for implementing the memory swapping strategy are not met, or when the additional time overhead of the memory swapping strategy is greater than or equal to the time overhead of the recalculation strategy, recalculation is performed. The memory space occupied by the target data object is released through the memory swapping process or the recalculation process.

[0013] Furthermore, the computationally decoupled memory optimization process is as follows: During the input gradient calculation stage of the downstream tag device, the parameters necessary for input gradient calculation are retained, and the unnecessary forward propagation intermediate feature maps are released. In the parameter gradient calculation stage of the downstream tag device, the calculation time axis is established and initialized. With the communication timeline ; For the data object recovery process at each computational level of the deep learning model, a memory optimization strategy is determined layer by layer; the process of determining the memory optimization strategy layer by layer includes: Regarding the first Data objects at each computational level, based on the accumulated duration of the current computation timeline. Current communication timeline cumulative duration Memory swapping time Recalculation time and the time required for parameter gradient calculation The system performs conditional judgments and determines the corresponding memory optimization strategy, which includes: The memory swapping strategy should be selected when the following conditions are met: ; Update the timeline as follows: ; Further determine the time overhead of memory swapping and recalculation strategies when the following conditions are met: ; Furthermore, a memory swapping strategy is selected when the following conditions are met: ; Update the timeline as follows: ; Furthermore, the recalculation strategy is selected when the following conditions are met: ; Update the timeline as follows: ; The process of determining the memory optimization strategy layer by layer is repeated until the data object recovery strategy for all layers is determined.

[0014] Furthermore, the method also includes: Based on the memory management scheme corresponding to each device label, the parameter optimization process time of each computing device is statistically processed to determine the parameter optimization completion time of each computing device. Based on the parameter optimization completion time, the upstream and downstream tag devices are sorted to determine the upstream tag device with the longest and shortest parameter optimization completion time, as well as the downstream tag device with the longest parameter optimization completion time. The maximum value of the parameter optimization completion time of all calculated devices is then determined as the total completion time of the scheme. Based on the total completion time of the above scheme, the target device for tag reversal is determined. When the longest completion time for parameter optimization of the upstream tag device is greater than the longest completion time for parameter optimization of the downstream tag device, the upstream tag device with the shortest completion time for parameter optimization is determined as the target device; otherwise, the downstream tag device with the longest completion time for parameter optimization is determined as the target device. The device tag of the target device is reversed to determine the updated device tag of the target device. Based on the updated device labels, the differentiated memory optimization process is re-executed, and the memory management scheme corresponding to each device label is updated according to the preset termination judgment conditions.

[0015] Furthermore, the preset termination determination conditions include: If the total completion time of the updated solution is less than the total completion time of the solution before reversal and the maximum number of iterations has not been reached, the current device label is maintained and the label reversal process continues. If the total completion time of the updated solution is greater than or equal to the total completion time of the solution before reversal or the maximum number of iterations is reached, restore the device label before reversal and terminate the label reversal optimization process.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs an optimization value index for data objects, quantifies and analyzes the data recomputation value index, and effectively evaluates the memory optimization potential of data objects. At the same time, it constructs a dual-cost index, which uses the additional time overhead of memory swapping strategy and the additional time overhead of recomputation strategy as the decision basis for memory swapping or recomputation strategy, thereby improving the utilization efficiency of memory resources in distributed deep learning training.

[0017] This invention sets upstream and downstream labels for computing devices and adopts a differentiated memory optimization strategy. The downstream labeled device calculates the input gradient first to trigger the upstream device to backpropagate in advance, thereby effectively compressing pipeline bubbles, realizing computational overlap between devices, and improving the overall throughput and computing resource utilization of the distributed training process.

[0018] This invention solves the problem of unbalanced load between devices by using a dynamic device tag reversal mechanism to iteratively adjust and optimize the completion time based on the actual parameters of the devices. This effectively avoids performance bottlenecks of individual or partial devices and ensures that the overall memory management scheme and computing efficiency of the distributed system reach a stable and balanced optimal state. Attached Figure Description

[0019] Figure 1 This is a flowchart of a memory optimization method for distributed deep learning, as shown in Example 1.

[0020] Figure 2 This is a flowchart of a memory optimization method for distributed deep learning, as shown in Example 2.

[0021] Figure 3 This is an architecture diagram of a distributed deep learning memory optimization system as shown in Example 3.

[0022] Figure 4 This is an architecture diagram of a distributed deep learning memory optimization system as shown in Example 4.

[0023] Figure 5 This is a flowchart of the device tag allocation process for Example 1.

[0024] Figure 6 This is a comparison diagram of bubbles in the production line of Example 1.

[0025] Figure 7 This is a schematic diagram of the downstream equipment optimization stage in Example 1.

[0026] Figure 8 This is a flowchart of the downstream dual-time-axis parallel optimization process in Example 1.

[0027] Figure 9 This is a flowchart of the upstream optimized memory block selection process in Example 1.

[0028] Figure 10 The flowchart is for the upstream three-axis parallel strategy of Example 1.

[0029] Figure 11 This is a flowchart for determining the feasibility of memory swapping in Example 1.

[0030] Figure 12 This is a flowchart of the memory axis update process in Example 1.

[0031] Figure 13 This is a schematic diagram comparing memory peak values ​​in Example 1.

[0032] Figure 14 This is a flowchart of the device tag reversal process for Example 1. Detailed Implementation

[0033] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Example 1 Please see Figure 1 This invention provides a memory optimization method for distributed deep learning, comprising: Based on the temporal relationship between data object generation nodes, calling nodes, and recovery nodes during the training process of deep learning models, a dual-cost index and an optimized value index are constructed. Based on the time relationship between forward and backward propagation of each computing device during the training of the deep learning model, the device label corresponding to each computing device is determined; the device label includes: upstream label and downstream label; The input gradient of the downstream tag device is preferentially calculated to determine the overlapping calculation time window used to trigger the back propagation of the upstream tag device; Based on the overlapping computation time window, the dual-cost index, and the value index optimized based on the recomputation value index, differentiated memory optimization is performed for each tag device to determine the memory management scheme corresponding to each tag device.

[0035] It should be noted that in this embodiment, the additional time overhead of the memory swapping strategy in the dual-cost index refers to the additional time overhead introduced when the memory swapping optimization strategy is applied to the data object that needs to be optimized; the additional time overhead of the recompile strategy refers to the additional time overhead introduced when the recompile optimization strategy is applied to the data object that needs to be optimized; this embodiment uses these two explicit indicators as important bases for memory optimization decisions.

[0036] The recomputation value index refers to the amount of memory that can be released per unit of recomputation time (i.e., the recovery efficiency of data objects). This embodiment quantifies this index and uses it as an important basis for determining whether to include the corresponding data objects in the scope of memory optimization.

[0037] In order to construct the aforementioned dual-cost index and the value index optimized by recalculating the value index, this embodiment determines the temporal relationship between the data object generation node, the calling node, and the recovery node.

[0038] Specifically, the temporal relationship between the data object generation node, the calling node, and the recovery node includes: Based on the order of operations of nodes during the training of deep learning models, the data dependencies between nodes are analyzed to determine the node topology. Based on the node topology, the time statistics of the node calculation process are performed to obtain node calculation duration data. Data analysis is performed based on the computational operations executed by the nodes to identify the data objects related to the node computations. The data object This includes: parameter memory, intermediate feature maps, gradient data, and optimizer state; With data objects Based on the computational generation process during forward propagation, the computational generation node for each data object is determined. ; With data objects Based on the process of calculating gradients during backpropagation, the gradient generation nodes for each data object are determined. ; With data objects Based on the process of data recovery using memory swapping in backpropagation, the memory swapping data recovery node for each data object is determined. .

[0039] It is understood that, in this embodiment, the training process of the deep learning model can be represented as a directed acyclic graph consisting of a set of operation nodes and their data transmission relationships; where nodes represent specific deep learning operators (such as convolution, matrix multiplication, activation operations, etc.); edges represent the data tensor flow relationships between nodes; by analyzing the operation order and dependencies of each node, the node topology structure corresponding to model training can be constructed.

[0040] After determining the node topology, this embodiment statistically analyzes the actual computation time of each computing node during the forward and backward propagation processes; specifically, existing performance analysis tools (such as CUDA Profiler, TensorBoard, etc.) can be used to directly obtain the computation time data of the nodes.

[0041] It should be noted that in this embodiment, the parameter memory refers to the weight matrix and bias vector that need to be iteratively updated during the training of the deep learning model; the intermediate feature map refers to the output activation values ​​generated by each operation node during the forward propagation of the model; the gradient data refers to the parameter gradient and input gradient calculated during the backpropagation of the model; and the optimizer state refers to the intermediate state variables maintained by the optimization algorithm (e.g., the Adam optimizer) during operation, including momentum and mean square gradient.

[0042] To achieve accurate calculation of the temporal relationships between nodes of a data object, this embodiment specifically adopts topological sorting and time accumulation calculation methods, as detailed below: First, taking forward propagation as an example, the nodes are topologically sorted using the Kahn algorithm, resulting in a node topological sort sequence, specifically represented as follows: ; in, This represents the total number of nodes.

[0043] Subsequently, by combining the node computation time data, the actual execution time of the node is calculated for the first node in the topological sort sequence. The calculation method is as follows: ; For any other node in the sequence The calculation method is as follows: ; ; In the formula, Indicates the first 1 node Represents a node The start time of execution, Represents a node The end execution time, Represents a node The calculation time, Represents a node The set of all direct predecessor nodes, Represent any one A node Previously, and directly connected to the node. The One predecessor node.

[0044] Thus, the temporal relationship between nodes during the forward propagation of the data object is obtained.

[0045] In one embodiment, a dual-cost index for memory optimization decisions and an optimization value index based on recomputation value metrics are further constructed based on this temporal relationship.

[0046] Specifically, the construction process of the dual-cost index includes: Based on the timing relationship between the data object generation node and the calling node, calculate the time when memory swapping for each data object begins and computation starts. Exchange the time when the computation is completed with the memory of this node. The time difference between them determines the additional time overhead of the memory swapping strategy for data objects. ; Since the process of recompiling data objects is similar to forward propagation, the time when computation begins for each data object is calculated. The time when the calculation is completed at this node. The time difference between them determines the additional time overhead of the recalculation strategy. ; Additional time overhead based on the memory swapping strategy Additional time overhead of recalculation strategy This yields the double-cost index for each data object.

[0047] Accordingly, the construction process of the optimized value index includes: Based on the memory usage of the data object The additional computation time required to recalculate this data object Calculate the ratio between memory usage and recompile time to determine the recompile value metric that characterizes the memory capacity released per unit of recompile time. ; The recalculated value index This serves as the index for the optimized value.

[0048] In practice, the difference between the completion time of memory swapping and the start time of memory swapping is used as the additional time overhead of the memory swapping strategy for this data object. The specific calculation formula is as follows: ; In the formula, Indicates the first A data object, Represents data objects The additional time overhead of memory swapping strategies Represents data objects The start time of memory swapping. Represents data objects The end time of memory swapping execution.

[0049] Similarly, in specific implementation, the difference between the time when the data object completes its computation during forward propagation and the time when it begins computation during forward propagation is used as the additional time overhead for the recomputation strategy of the data object. The specific calculation formula is as follows: ; In the formula, Indicates the first A data object, Represents data objects The additional time overhead of the recomputation strategy Represents data objects The start time of the recalculation. Represents data objects The end execution time of the recalculation.

[0050] Furthermore, the recomputation value metric is determined by calculating the ratio between the memory footprint of a data object and the additional computation time required for its recomputation. The specific calculation formula is as follows: ; In the formula, Represents data objects The recalculation value index, Represents data objects The memory capacity occupied by it, Represents data objects The recomputation strategy incurs additional time overhead.

[0051] Finally, the additional time overhead of the memory swapping strategy will be considered. Additional time overhead of recalculation strategy Together, they serve as a dual-cost index to reflect the potential value of each data object in memory optimization. This dual-cost index can be represented as: ; In the formula, Represents data objects The dual-cost index. In practical application scenarios, in order to further realize optimized collaborative control between distributed devices, this embodiment classifies the roles of each device to achieve efficient computing collaboration and memory management strategies between distributed devices; specifically, by analyzing the relationship between the computing time required for forward and backward propagation of each computing device, the optimized role label of the device is determined, that is, the device label is divided into upstream label and downstream label.

[0052] Specifically, the method for determining the device tag corresponding to each computing device includes: Based on the forward propagation order of each computing device during the training of the deep learning model, the computing device at the beginning of the forward propagation order is determined as the initial upstream label device; For the target computing device in the forward propagation sequence, based on the forward propagation duration of the target computing device. Compared with backpropagation duration Summation is performed to determine the total usage time of the target computing device. ; Based on the backpropagation time of the upstream tag device located before the target computing device. Perform cumulative calculations to determine the backpropagation time threshold of the target computing device. ; Based on the total occupied time With the backpropagation duration threshold Based on the size relationship, with the goal of balancing the pipeline load, the label of the target computing device is determined as either an upstream label or a downstream label; if the total occupied time Greater than the backpropagation time threshold If it is true, it is determined to be an upstream label; otherwise, it is determined to be a downstream label.

[0053] In specific implementation, such as Figure 5 As shown, the first computing device to perform forward propagation computation during the training process (i.e., the first device in the forward propagation sequence) is pre-labeled as the initial upstream label device, serving as the initial reference benchmark for subsequent device label configuration.

[0054] The total time occupied by the device is calculated by summing the forward propagation computation time and the backward propagation computation time of the target computing device. The specific calculation formula is as follows: ; In the formula, This indicates the number of tags to be configured in the forward propagation sequence. One target computing device, Indicates the target computing device The total time the device is occupied. Indicates the target computing device Forward propagation computation time, Indicates the target computing device The backpropagation computation time.

[0055] It should be noted that the total device usage time reflects the computing resource usage time of the target computing device within a complete training iteration cycle.

[0056] Furthermore, the backpropagation duration threshold represents the cumulative value of the backpropagation computation duration of all upstream tag devices previously identified as the current target computing device, reflecting the allowable time window when the target computing device is configured as an upstream device. The specific calculation formula is as follows: ; In the formula, Indicates the target computing device The corresponding backpropagation duration threshold, Indicates the first One has been identified as an upstream labeling device. The backpropagation computation time, Indicates the location of the target computing device The set of all previously identified upstream tagged devices.

[0057] The total device occupancy time is compared with the backpropagation time threshold to determine the device tag of the target computing device, specifically represented as follows: ; In the formula, Indicates the target computing device The identified device label.

[0058] In practical applications, this embodiment effectively reduces pipeline bubbles between devices by decoupling the input gradient from the parameter gradient; for example... Figure 6 As shown, where Figure 6 (a) shows the large pipeline bubble generated by strict temporal dependence in traditional deep learning training. Figure 6 (b) demonstrates the effect of employing the decoupling mechanism of this invention. By prioritizing the calculation of the input gradient by the downstream device and triggering the upstream device, pipeline bubbles are significantly reduced, achieving effective overlap in computation time. Specifically, the downstream tag device triggers the upstream tag device to start backpropagation computation in advance by prioritizing the calculation and transmission of the input gradient, thereby achieving computational overlap and forming an overlapping computation time window.

[0059] In implementation, the method for determining the overlapping calculation time window includes: Based on the requirement of downstream tag devices to calculate their own input gradients during backpropagation, the backpropagation intermediate feature maps generated by each downstream tag device are filtered to obtain the corresponding set of backpropagation intermediate feature maps necessary for calculating the input gradients. Based on the backpropagation intermediate feature map set, perform swapping, removing or recalculating and releasing operations on non-essential feature maps to determine the available memory resources for priority computation of input gradients for each downstream label device; Based on the available memory resources, the input gradient corresponding to each downstream tag device is calculated first, and the input gradient calculation result used to trigger the backpropagation of the corresponding upstream tag device is determined. According to the non-blocking communication mechanism, the input gradient calculation result is sent to the corresponding upstream tag device, triggering the backpropagation calculation of the corresponding upstream tag device; Based on the backpropagation calculation process of the upstream tag device, the parameter gradient of the corresponding downstream tag device is calculated in parallel, and an overlapping calculation time window is determined to mask the calculation overhead of the parameter gradient of the corresponding downstream tag device.

[0060] It should be noted that when deep learning models calculate the input gradient during the backpropagation stage, they usually only need a portion of the feature maps generated during forward propagation, not all of the intermediate feature maps. Therefore, in this embodiment, a data dependency analysis is performed on the backpropagation intermediate feature maps generated by the downstream tag device during the backpropagation process to select the set of backpropagation intermediate feature maps that are necessary for calculating the input gradient.

[0061] It is understood that feature maps outside the aforementioned set of intermediate feature maps in the backpropagation are considered non-essential feature maps. For these non-essential feature maps, this embodiment adopts two memory release strategies: one is a swap-based removal process, which swaps the feature maps to external storage; the other is a recalculation-based release process, which releases the feature maps and restores them through recalculation when needed later.

[0062] Through the above memory release operations, this embodiment obtains the amount of available memory resources for each downstream label device to preferentially calculate the input gradient.

[0063] Furthermore, the downstream tag device immediately prioritizes calculating its own corresponding input gradient in order to obtain the input gradient calculation result as early as possible. The input gradient calculation result is used to trigger the backpropagation calculation of the corresponding upstream tag device.

[0064] In the implementation process, this embodiment adopts a non-blocking communication mechanism to realize the asynchronous transmission of input gradients. That is, after each downstream tag device obtains its own corresponding input gradient calculation result, it immediately sends the above input gradient calculation result to the corresponding upstream tag device in a non-blocking communication manner. As a result, the upstream tag device can immediately start the back propagation process without waiting for the downstream device to complete the parameter gradient calculation, thereby effectively reducing the waiting time between devices.

[0065] After the upstream tag device initiates backpropagation computation, the corresponding downstream tag device calculates its own parameter gradient in parallel during this period. Through parallel execution, the parameter gradient computation overhead of the downstream tag device can be masked by the backpropagation computation process of the upstream tag device, thus forming an overlapping computation time window and significantly reducing the resource idle time in the pipeline parallel process.

[0066] After completing the allocation of the aforementioned device tags, this embodiment performs differentiated memory optimization for each tag device based on the overlapping calculation time window, the dual-cost index, and the value index optimized based on the recalculation value index, and determines the memory management scheme corresponding to each tag device.

[0067] In practical applications, the upstream and downstream labeling devices play different roles and have different bottlenecks in distributed training, making it difficult to balance global efficiency with a single optimization strategy. The bottleneck of the upstream device is mainly the memory peak during backpropagation, while the bottleneck of the downstream device is how to quickly release gradients to reduce pipeline bubbles. Therefore, this embodiment adopts a differentiated memory optimization method.

[0068] Specifically, the differentiated memory optimization process includes: performing a three-axis parallel memory optimization process for the upstream tag device; and performing a computationally decoupled memory optimization process for the downstream tag device.

[0069] It should be understood that the optimization goal of the upstream tag device is to release memory while minimizing time overhead. To this end, this embodiment introduces a three-axis parallel optimization strategy, that is, on the basis of the dual axes of the calculation time axis and the communication time axis, a memory axis is added as a third decision dimension to finely control the selection of memory swapping and recalculation.

[0070] In one embodiment, the triaxial parallel memory optimization process is as follows: Based on the optimized value index, obtain the recalculated value index of each data object. ; The data object is recalculated according to the recalculated value index. Sort in descending order to obtain the recalculated value sequence; An optimized memory block sequence is constructed based on the recomputation value sequence, and data objects are added to the optimized memory block sequence in sequence according to the sorting results. The total memory usage of the hierarchy after the addition is calculated. Data objects are continuously added to the optimized memory block sequence until the total memory usage of the level is less than or equal to a preset memory optimization threshold, and the final optimized memory block sequence is determined. For the target data object in the optimized memory block sequence, calculate the space allocation time required to adopt the memory swapping strategy, and calculate the expected total memory usage at the expected level at the space allocation time point; The expected total memory usage at each level is less than or equal to the preset memory optimization threshold as the condition for implementing the memory swapping strategy. Based on the implementation conditions of the memory swapping strategy and the relationship between the additional time overhead of the memory swapping strategy and the recalculation strategy, optimization processing is performed on the target data object; When the conditions for implementing the memory swapping strategy are met, and the additional time overhead of the memory swapping strategy is less than the time overhead of the recalculation strategy, the memory swapping process is executed. When the conditions for implementing the memory swapping strategy are not met, or when the additional time overhead of the memory swapping strategy is greater than or equal to the time overhead of the recalculation strategy, recalculation is performed. The memory space occupied by the target data object is released through the memory swapping process or the recalculation process.

[0071] In the specific implementation process, such as Figure 9 As shown, this embodiment first extracts the recalculated value index of each layer of data objects (such as intermediate feature maps) based on the aforementioned constructed optimized value index. Recalculation of value indicators The larger the value, the more memory space the data object can gain per unit of recalculation time, or the higher the cost-effectiveness of recalculation. By sorting in descending order, the system prioritizes operating on data objects with high cost-effectiveness.

[0072] Subsequently, in order to accurately quantify the memory changes during model training, this embodiment defines the total memory usage for each computational level, specifically as follows: ; In the formula, Indicates the first Total memory usage of each computational level under the current optimization strategy Indicates the first The set of all data objects in the current optimized memory block sequence at each computation level.

[0073] Continue adding data objects to the optimized memory block sequence until the total memory usage of the computation level meets the following termination condition: ; In the formula, This indicates the preset memory optimization threshold.

[0074] It should be noted that the memory optimization threshold is... The threshold is determined based on the actual memory capacity and performance requirements of the device. In this embodiment, 80% to 90% of the memory capacity of a single device is selected as the threshold. The specific threshold can be determined based on actual testing.

[0075] like Figure 11 As shown, this is for the target data object in the optimized memory block sequence. Calculate the space allocation time using a memory swapping strategy. And at that moment, calculate the expected total memory usage for the hierarchy.

[0076] Specifically, the formula for calculating the space opening time is as follows: ; In the formula, Represents data objects The start time of space allocation for implementing the memory swapping strategy. Represents data objects The time required for memory swapping Indicates the current computing level The completion time of the computation at the previous computational level. Indicates the current computing level The communication completion time.

[0077] Based on the aforementioned space allocation time, the expected total memory usage after the target data object is added is further calculated to determine the specific implementation conditions of the memory swapping strategy.

[0078] The specific conditions for implementing the memory swapping strategy are expressed as follows: ; In the formula, This represents the computational level index corresponding to the memory space allocation time. Represents the target data object Generate a computational hierarchical index. Indicates the first The original memory usage of each computing level Represents the target data object The amount of memory that needs to be pre-occupied for memory swapping.

[0079] It is understood that the implementation conditions ensure that the model memory usage will never exceed a preset threshold when memory swapping is performed.

[0080] like Figure 10 As shown, after the implementation conditions of the memory swapping strategy are determined, the specific execution method of the optimization strategy is determined based on the additional time overhead of the memory swapping strategy and the recalculation strategy: If the conditions for implementing the memory swapping strategy are met and the additional time overhead of the memory swapping strategy is less than the additional time overhead of the recalculation strategy, then the memory swapping process will be executed. If the conditions for implementing the memory swapping strategy are not met, or if the additional time overhead of the memory swapping strategy is greater than or equal to the additional time overhead of the recalculation strategy, then the recalculation process will be performed.

[0081] For example, memory swapping is performed if and only if the following conditions are met simultaneously: the conditions for implementing the memory swapping strategy are met, and the additional time overhead of the memory swapping strategy is met. Less than the additional time overhead of the recalculation strategy ; A recalculation is performed when either of the following conditions is met: the conditions for implementing the memory swapping strategy are not met, or the memory swapping strategy incurs additional time overhead. Greater than or equal to the additional time cost of recompiling the strategy .

[0082] Release the target data object using the memory swapping or recalculation methods described above. The memory space occupied is controlled to ensure that the peak memory usage of the model is always kept within the preset threshold.

[0083] In the specific implementation process, such as Figure 12 As shown, this embodiment proposes a computationally decoupled memory optimization process for downstream tagging devices; specifically, as... Figure 7 As shown, the downstream tag device achieves efficient utilization of memory resources by releasing unnecessary intermediate feature maps during the input gradient calculation stage and by using a dual-time-axis dynamic programming memory swapping and recalculation strategy during the parameter gradient calculation stage.

[0084] In another embodiment, the computationally decoupled memory optimization process is as follows: During the input gradient calculation stage of the downstream tag device, the parameters necessary for input gradient calculation are retained, and the unnecessary forward propagation intermediate feature maps are released. like Figure 8 As shown, in the parameter gradient calculation stage of the downstream tag device, the calculation time axis is established and initialized. With the communication timeline ; For the data object recovery process of each layer of the deep learning model, a memory optimization strategy is determined layer by layer; the process of determining the memory optimization strategy layer by layer includes: Regarding the first The data objects in the layer accumulate time based on the current calculation timeline. Current communication timeline cumulative duration Memory swapping time Recalculation time and the time required for parameter gradient calculation Perform conditional checks and determine the corresponding memory optimization strategy: The memory swapping strategy should be selected when the following conditions are met: ; Update the timeline as follows: ; Further determine the time overhead of memory swapping and recalculation strategies when the following conditions are met: ; Furthermore, a memory swapping strategy is selected when the following conditions are met: ; Update the timeline as follows: ; Furthermore, the recalculation strategy is selected when the following conditions are met: ; Update the timeline as follows: ; The process of determining the memory optimization strategy layer by layer is repeated until the data object recovery strategy for all layers is determined.

[0085] In specific implementation, during the input gradient calculation stage of the downstream tag device, this embodiment only retains the parameter data necessary for calculating the input gradient and releases the unnecessary forward propagation intermediate feature maps to minimize the peak memory usage.

[0086] It is understandable that the parameter data required for the input gradient calculation stage refers to the data objects that are essential for the input gradient calculation of the current calculation layer, such as the convolution kernel parameters of the convolutional layer or certain key feature maps; the non-essential forward propagation intermediate feature maps refer to other feature map data that do not participate in the input gradient calculation.

[0087] In the parameter gradient calculation stage of the downstream tag device, this embodiment establishes and initializes the calculation time axis. With the communication timeline Specifically, these two timelines are initialized as follows: ; Among them, variables This represents the computational level number of the deep learning model. To calculate the cumulative completion time of the task, This represents the cumulative completion time of communication (including memory swapping) tasks.

[0088] For the data object recovery process at each computational level of the deep learning model, the optimal memory optimization strategy is determined layer by layer.

[0089] It should be understood that, such as Figure 13 As shown, through the above-described layer-by-layer dynamic programming process, this embodiment can automatically determine the optimal memory recovery strategy for each computing level during the computing stage of the downstream tag device, so as to effectively reduce the peak memory usage and reduce the idle time of computing resources, thereby significantly improving the overall utilization efficiency of device resources.

[0090] Example 2 like Figure 2As shown, based on Embodiment 1 above, this embodiment has been further improved. To address the problem that the fixed device tag configuration in Embodiment 1 may lead to unbalanced loads on upstream and downstream devices and insufficient optimization effects, this embodiment will focus on the dynamic adjustment and iterative optimization of device tags. This embodiment discloses a memory optimization method for distributed deep learning, which further includes: Based on the memory management scheme corresponding to each device label, the parameter optimization process time of each computing device is statistically processed to determine the parameter optimization completion time of each computing device. Based on the parameter optimization completion time, the upstream and downstream tag devices are sorted to determine the upstream tag device with the longest and shortest parameter optimization completion time, as well as the downstream tag device with the longest parameter optimization completion time. The maximum value of the parameter optimization completion time of all calculated devices is then determined as the total completion time of the scheme. Based on the total completion time of the above scheme, the target device for tag reversal is determined. When the longest completion time for parameter optimization of the upstream tag device is greater than the longest completion time for parameter optimization of the downstream tag device, the upstream tag device with the shortest completion time for parameter optimization is determined as the target device; otherwise, the downstream tag device with the longest completion time for parameter optimization is determined as the target device. The device tag of the target device is reversed to determine the updated device tag of the target device. Based on the updated device labels, the differentiated memory optimization process is re-executed, and the memory management scheme corresponding to each device label is updated according to the preset termination judgment conditions.

[0091] It should be noted that the parameter optimization process duration refers to the actual time spent by each computing device during its own parameter optimization process (i.e., gradient calculation and parameter update). The time required for each computing device to complete its own parameter optimization is obtained through performance analysis tools (such as CUDA Profiler or TensorBoard).

[0092] It is understood that the sorting process is based on the parameter optimization completion time data, and the upstream and downstream tag devices are sorted in descending order respectively, so as to identify the upstream tag devices with the longest and shortest parameter optimization completion time, and the downstream tag devices with the longest parameter optimization completion time.

[0093] The maximum value of the parameter optimization completion time for all computing devices is defined as the total completion time of the scheme in this embodiment, specifically expressed as: ; In the formula, Indicates the total completion time of the plan. Indicates computing device Parameter optimization completion time, It represents the set of all computing devices.

[0094] It should be understood that the target device refers to the device that needs to optimize the overall memory management scheme through device tag inversion. Specifically, when the longest completion time of upstream tag device parameter optimization exceeds the longest completion time of downstream tag device parameter optimization, it indicates that the load in the upstream tag device is too heavy. In this case, this embodiment explicitly selects the device with the lightest load (i.e., the shortest parameter optimization completion time) among the upstream tag devices as the target device for tag inversion. Conversely, it indicates that the load in the downstream tag device is relatively heavy. In this case, the device with the heaviest load (longest parameter optimization completion time) among the downstream tag devices is explicitly selected as the target device to achieve reasonable load adjustment.

[0095] During implementation, such as Figure 14 As shown, the specific process of the label reversal is as follows: If the original device label of the target device is an upstream label, then update its device label to a downstream label; if the original device label of the target device is a downstream label, then update its device label to an upstream label.

[0096] Finally, based on the updated device tag of the target device, the aforementioned transmission masking memory optimization process for the upstream tag device and the computation decoupling memory optimization process for the downstream tag device are re-executed to generate a new memory management scheme.

[0097] In practical applications, to prevent the device tag reversal process from going on indefinitely or the optimization effect from degrading, this embodiment presets a termination judgment condition.

[0098] Specifically, the preset termination conditions include: If the total completion time of the updated solution is less than the total completion time of the solution before reversal and the maximum number of iterations has not been reached, the current device label is maintained and the label reversal process continues. If the total completion time of the updated solution is greater than or equal to the total completion time of the solution before reversal or the maximum number of iterations is reached, restore the device label before reversal and terminate the label reversal optimization process.

[0099] Understandably, after completing the device tag reversal and re-executing the differentiated memory optimization process, if the new optimization scheme shows an improvement in the total completion time (i.e., the total completion time of the new scheme is less than the total completion time of the scheme before the last reversal), and the number of iterations of the tag reversal optimization has not reached the preset maximum number of iterations, then the device's current updated tag settings will be maintained, and the next round of device tag reversal and optimization processing will be further executed to further improve the overall optimization effect of the scheme. If the differentiated memory optimization process re-executed after the device tag is reversed does not show an optimization effect (i.e., the total completion time of the new scheme is greater than or equal to the total completion time of the scheme before the last reversal), or the number of iterations of the tag reversal optimization process has reached the preset maximum number of iterations, this embodiment terminates the device tag reversal optimization process, and the device tag is restored to the state before the last reversal, ensuring that the scheme will not degrade or continue to loop ineffectively due to the tag reversal process.

[0100] It should be noted that the maximum number of iterations ranges from 1 to 10, and can be set according to actual optimization needs.

[0101] Example 3 like Figure 3 As shown in the example, the parts not detailed in this embodiment are as shown in Example 1. This embodiment discloses a memory optimization system for distributed deep learning, the system comprising: The temporal cost analysis module is used to construct a dual cost index that includes the additional time overhead of memory swapping strategy and the additional time overhead of recomputation strategy, based on the temporal relationship between data object generation nodes and calling nodes during the training of deep learning models. Device label determination module: used to determine the device label corresponding to each computing device based on the relationship between the forward propagation and backward propagation time during the deep learning model training process; the device label includes: upstream label and downstream label; Gradient window construction module: used to perform priority calculation processing on the input gradient of the downstream tag device and determine the overlapping calculation time window used to trigger backpropagation of the upstream tag device; Memory Difference Optimization Module: This module utilizes the overlapping computation time window and dual-cost index to perform differentiated memory optimization processing for each device tag, thereby determining the memory management scheme corresponding to each device tag.

[0102] Example 4 like Figure 4 As shown in the example, the parts not detailed in this embodiment are shown in Example 2. This embodiment discloses a memory optimization system for distributed deep learning, the system further comprising: Duration statistics and analysis module: It is used to perform statistical processing on the parameter optimization process duration of each computing device according to the memory management scheme corresponding to each device tag, and determine the parameter optimization completion time of each computing device; The time-series sorting module is used to sort the upstream and downstream tag devices according to the optimized completion time of the parameters, determine the upstream tag device with the longest and shortest parameter optimization completion time and the downstream tag device with the longest parameter optimization completion time, and determine the maximum value of the optimized completion time of all calculated device parameters as the total completion time of the scheme. Reversal Target Selection Module: Used to determine the target device for tag reversal based on the total completion time of the scheme. When the longest parameter optimization completion time of the upstream tag device is greater than the longest parameter optimization completion time of the downstream tag device, the device with the shortest parameter optimization completion time among the upstream tag devices is determined as the target device; otherwise, the device with the longest parameter optimization completion time among the downstream tag devices is determined as the target device. Device tag inversion module: used to invert the device tag of the target device and determine the updated device tag of the target device; Iteration Termination Determination Module: Used to re-execute differentiated memory optimization processing based on the updated device label, and update the memory management scheme corresponding to each device label according to the preset termination determination conditions.

[0103] Some of the data in the above formula are calculated by removing dimensions and taking their numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0104] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A memory optimization method for distributed deep learning, characterized in that, include: Based on the temporal relationship between data object generation nodes, calling nodes, and recovery nodes during the training process of deep learning models, a dual-cost index and an optimized value index are constructed. Based on the relationship between the forward and backward propagation times of each computing device during the deep learning model training process, the device label corresponding to each computing device is determined. The device label includes: an upstream label and a downstream label; The input gradient of the downstream tag device is preferentially calculated to determine the overlapping calculation time window used to trigger the back propagation of the upstream tag device; Based on the overlapping computation time window, the dual-cost index, and the optimized value index, differentiated memory optimization is performed for each tag device to determine the corresponding memory management scheme for each tag device.

2. The memory optimization method for distributed deep learning according to claim 1, characterized in that, The timing relationship between the data object generation node, the calling node, and the recovery node includes: Based on the order of operations of nodes during the training of deep learning models, the data dependencies between nodes are analyzed to determine the node topology. Based on the node topology, the time statistics of the node calculation process are performed to obtain node calculation duration data. Data analysis is performed based on the computational operations executed by the nodes to identify the data objects related to the node computations. The data object This includes: parameter memory, intermediate feature maps, gradient data, and optimizer state; With data objects Based on the computational generation process during forward propagation, the computational generation node for each data object is determined. ; With data objects Based on the process of calculating gradients during backpropagation, the gradient generation nodes for each data object are determined. ; With data objects Based on the process of data recovery using memory swapping in backpropagation, the memory swapping data recovery node for each data object is determined. .

3. The memory optimization method for distributed deep learning according to claim 1, characterized in that, The construction process of the dual-cost index includes: Calculate the time when memory swapping for each data object begins and computation begins. Exchange the time when the computation is completed with the memory of this node. The time difference between them determines the additional time overhead of the memory swapping strategy for data objects. ; Calculate the time when the computation begins for each data object. The time when the calculation is completed at this node. The time difference between them determines the additional time overhead of the recalculation strategy. ; Additional time overhead based on the memory swapping strategy Additional time overhead of recalculation strategy This yields the double-cost index for each data object; Accordingly, the construction process of the optimized value index includes: Based on the memory usage of the data object The additional computation time required to recalculate this data object Calculate the ratio between memory usage and recompile time to determine the recompile value metric that characterizes the memory capacity released per unit of recompile time. ; The recalculated value index This serves as the index for the optimized value.

4. The memory optimization method for distributed deep learning according to claim 1, characterized in that, The method for determining the device tag corresponding to each computing device includes: Based on the forward propagation order of each computing device during the training of the deep learning model, the computing device at the beginning of the forward propagation order is determined as the initial upstream label device; For the target computing device in the forward propagation sequence, based on the forward propagation duration of the target computing device. With backpropagation duration Summation is performed to determine the total usage time of the target computing device. ; Based on the backpropagation time of the upstream tag device located before the target computing device. Perform cumulative calculations to determine the backpropagation time threshold of the target computing device. ; Based on the total occupied time With the backpropagation duration threshold Based on the size relationship, the label of the target computing device is determined as either an upstream label or a downstream label; if the total occupancy time... Greater than the backpropagation time threshold If it is true, it is determined to be an upstream label; otherwise, it is determined to be a downstream label.

5. The memory optimization method for distributed deep learning according to claim 1, characterized in that, The method for determining the overlapping calculation time window includes: Based on the requirement of downstream tag devices to calculate their own input gradients during backpropagation, the backpropagation intermediate feature maps generated by each downstream tag device are filtered to obtain the corresponding set of backpropagation intermediate feature maps necessary for calculating the input gradients. Based on the backpropagation intermediate feature map set, perform swapping, removing or recalculating and releasing operations on non-essential feature maps to determine the available memory resources for priority computation of input gradients for each downstream label device; Based on the available memory resources, the input gradient corresponding to each downstream tag device is calculated first, and the input gradient calculation result used to trigger the backpropagation of the corresponding upstream tag device is determined. According to the non-blocking communication mechanism, the input gradient calculation result is sent to the corresponding upstream tag device, triggering the backpropagation calculation of the corresponding upstream tag device; Based on the backpropagation calculation process of the upstream tag device, the parameter gradient of the corresponding downstream tag device is calculated in parallel, and an overlapping calculation time window is determined to mask the calculation overhead of the parameter gradient of the corresponding downstream tag device.

6. The memory optimization method for distributed deep learning according to claim 1, characterized in that, The differentiated memory optimization process includes: performing a three-axis parallel memory optimization process for the upstream tag device; and performing a computationally decoupled memory optimization process for the downstream tag device.

7. The memory optimization method for distributed deep learning according to claim 6, characterized in that, The triaxial parallel memory optimization process is as follows: Based on the optimized value index, obtain the recalculated value index of each data object. ; The data object is recalculated according to the recalculated value index. Sort in descending order to obtain the recalculated value sequence; An optimized memory block sequence is constructed based on the recomputation value sequence, and data objects are added to the optimized memory block sequence in sequence according to the sorting results. The total memory usage of the hierarchy after the addition is calculated. Data objects are continuously added to the optimized memory block sequence until the total memory usage of the level is less than or equal to a preset memory optimization threshold, and the final optimized memory block sequence is determined. For the target data object in the optimized memory block sequence, calculate the space allocation time required to adopt the memory swapping strategy, and calculate the expected total memory usage at the expected level at the space allocation time point; The expected total memory usage at each level is less than or equal to the preset memory optimization threshold as the condition for implementing the memory swapping strategy. Based on the implementation conditions of the memory swapping strategy and the relationship between the additional time overhead of the memory swapping strategy and the recalculation strategy, optimization processing is performed on the target data object; When the conditions for implementing the memory swapping strategy are met, and the additional time overhead of the memory swapping strategy is less than the time overhead of the recalculation strategy, the memory swapping process is executed. When the conditions for implementing the memory swapping strategy are not met, or when the additional time overhead of the memory swapping strategy is greater than or equal to the time overhead of the recalculation strategy, recalculation is performed. The memory space occupied by the target data object is released through the memory swapping process or the recalculation process.

8. The memory optimization method for distributed deep learning according to claim 6, characterized in that, The computationally decoupled memory optimization process is as follows: During the input gradient calculation stage of the downstream tag device, the parameters necessary for input gradient calculation are retained, and the unnecessary forward propagation intermediate feature maps are released. In the parameter gradient calculation stage of the downstream tag device, the calculation time axis is established and initialized. With the communication timeline ; For the data object recovery process at each computational level of a deep learning model, memory optimization strategies are determined layer by layer. The process of determining the memory optimization strategy layer by layer includes: Regarding the first Data objects at each computational level, based on the accumulated duration of the current computation timeline. Current communication timeline cumulative duration Memory swapping time Recalculation time and the time required for parameter gradient calculation The system performs conditional judgments and determines the corresponding memory optimization strategy, which includes: The memory swapping strategy should be selected when the following conditions are met: ； Update the timeline as follows: ； Further determine the time overhead of memory swapping and recalculation strategies when the following conditions are met: ； Furthermore, a memory swapping strategy is selected when the following conditions are met: ； Update the timeline as follows: ； Furthermore, the recalculation strategy is selected when the following conditions are met: ； Update the timeline as follows: ； The process of determining the memory optimization strategy layer by layer is repeated until the data object recovery strategy for all layers is determined.

9. A memory optimization method for distributed deep learning according to claim 1, characterized in that, The method further includes: Based on the memory management scheme corresponding to each device label, the parameter optimization process time of each computing device is statistically processed to determine the parameter optimization completion time of each computing device. Based on the parameter optimization completion time, the upstream and downstream tag devices are sorted to determine the upstream tag device with the longest and shortest parameter optimization completion time, as well as the downstream tag device with the longest parameter optimization completion time. The maximum value of the parameter optimization completion time of all calculated devices is then determined as the total completion time of the scheme. Based on the total completion time of the above scheme, the target device for tag reversal is determined. When the longest completion time for parameter optimization of the upstream tag device is greater than the longest completion time for parameter optimization of the downstream tag device, the upstream tag device with the shortest completion time for parameter optimization is determined as the target device; otherwise, the downstream tag device with the longest completion time for parameter optimization is determined as the target device. The device tag of the target device is reversed to determine the updated device tag of the target device. Based on the updated device labels, the differentiated memory optimization process is re-executed, and the memory management scheme corresponding to each device label is updated according to the preset termination judgment conditions.

10. A memory optimization method for distributed deep learning according to claim 1, characterized in that, The preset termination conditions include: If the total completion time of the updated solution is less than the total completion time of the solution before reversal and the maximum number of iterations has not been reached, the current device label is maintained and the label reversal process continues. If the total completion time of the updated solution is greater than or equal to the total completion time of the solution before reversal or the maximum number of iterations is reached, restore the device label before reversal and terminate the label reversal optimization process.

Images