Training methods for neural network models for computing power scheduling optimization

By dynamically splitting the gradient calculation logic and orthogonally decomposing the weight evolution vector, the problems of training efficiency and stability caused by the fluctuation of computing power of computing nodes are solved, and efficient training of neural network models in heterogeneous clusters is realized.

CN123117239APending Publication Date: 2026-11-17FUZHOU ZHEYAN INTELLIGENT TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202611552513.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-09
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In distributed large-scale neural network training, fluctuations in computing power of computing nodes can lead to pipeline cavitation and memory wait times. Existing methods cannot adjust gradient calculation paths and operator priorities in a timely manner, affecting the parallel efficiency and convergence stability of model training.

Method used

By acquiring the real-time computation latency and memory saturation of the operator layer, the gradient calculation logic is dynamically divided into the main update branch and the delay residual accumulation branch. The quantization bit depth and accumulation period are adjusted, and orthogonal decomposition is performed using the weight evolution vector. Collinear components that are consistent with the direction of model weight change are retained, while orthogonal components are discarded, thereby realizing the dynamic adjustment and compensation of the gradient.

Benefits of technology

In the event of computing power fluctuations, maintain the continuous operation of the training pipeline, reduce pipeline pauses and memory waits caused by node computing power fluctuations, and maintain the convergence stability and communication efficiency of model training.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117239A_ABST
    Figure CN123117239A_ABST
Patent Text Reader

Abstract

This invention relates to the field of neural network training technology and discloses a neural network model training method for optimizing computing power scheduling. The method includes: obtaining the real-time computation latency and memory saturation of the operator layer to generate a dynamic splitting threshold; when the threshold is higher than a benchmark, splitting the gradient calculation logic into a main update branch that retains low-order gradients and a delayed accumulation branch that temporarily stores high-order residuals, synchronously compressing the low-order gradient components to update the weights; when the threshold drops below the benchmark, orthogonally decomposing the temporarily stored high-order residuals according to the weight evolution vector within the accumulation period, truncating divergent components and using collinear components as compensation gradients to update weights. This invention deeply couples the computation graph topology with real-time computing power, eliminating pipeline stalls caused by computing power fluctuations, and eliminating divergent disturbances caused by latency through an orthogonal truncation mechanism, ensuring convergence stability and improving training throughput.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a neural network model training method for optimizing computing power scheduling, belonging to the field of neural network training technology. Background Technology

[0002] In distributed large-scale neural network training, the computation graph is usually compiled and divided into a directed acyclic graph consisting of multiple forward and backward operators. The execution order and memory allocation of the operators are fixed according to the preset computational resource topology. This static compilation method usually assumes that each computing node in the cluster can continuously provide stable computing power. Then, by pre-arranging the dependencies and communication pipelines between operators, parallel computation of parameter gradients and cross-node synchronization are achieved.

[0003] In heterogeneous or shared clusters, the actual computing power of nodes is not always stable. Hardware thermal throttling, shared memory bandwidth contention, and instantaneous jitter in the interconnect network can all cause node computing power to change within milliseconds. Existing training mechanisms typically handle cluster resource scheduling and model gradient calculation separately. The computation graph compilation process struggles to perceive hardware resource status in a timely manner. When the computing power of some nodes decreases or memory saturation increases, the system cannot adjust gradient calculation paths and operator priorities at the operator level in a timely manner, easily leading to pipeline cavitation and increased memory wait time. For example, Chinese invention patent CN118052260B discloses a dynamic hierarchical gradient compression method for neural network models. Although it reduces communication overhead by reusing thresholds and merging gradient tensors through dynamic programming, its control logic highly depends on the presuppositions of static network stability and predictable gradient distribution. This makes it problematic when facing dynamic heterogeneous clusters or sudden network bursts. In complex real-world scenarios such as network jitter, control mechanisms based on fixed-period threshold reuse are prone to mismatches between gradient compression accuracy and timeliness due to sudden changes in latency, leading to model convergence lag under high compression ratios. Furthermore, their heuristic merging algorithm generates additional planning time overhead in extremely deep network structures, weakening the actual effect of computational and communication overlap. To address computing power fluctuations, existing methods typically employ job migration or partial pipeline suspension, but this increases cross-node data transmission and introduces global synchronization waiting. Some methods use delayed synchronization or directly discard lagging gradients; however, these methods do not consider the changes in model weights during the delay period, but instead linearly superimpose the lagging higher-order residual gradients onto the current weights, which can easily cause the superimposed gradient direction to deviate from the original descent direction, resulting in parameter update direction divergence or local oscillations, making it difficult to guarantee the convergence stability of complex neural network training processes.

[0004] Therefore, how to balance the parallel efficiency of the training pipeline with the stability of the convergence direction of model parameters under fluctuating computing power remains a problem that needs to be solved in neural network model training. Summary of the Invention

[0005] To address the problems in the background art, the technical solution of the present invention is as follows: A neural network model training method for computing power scheduling optimization, comprising: Step S1: Obtain the real-time computation latency and memory saturation of each operator layer during the forward propagation of the neural network model, and generate a dynamic segmentation threshold based on the estimated computation time, real-time computation latency and memory saturation. Step S2: When the dynamic segmentation threshold is greater than the computing power matching benchmark, the gradient calculation logic of the segmentation operator layer is the backbone update branch and the delay residual accumulation branch. The quantization bit of the backbone update branch and the accumulation period of the delay residual accumulation branch are determined according to the dynamic segmentation threshold. The backbone update branch retains the low-order gradient components used for weight update, and the delay residual accumulation branch temporarily stores the high-order residual gradient. Step S3: Compress the low-order gradient components according to the quantization bit depth, and synchronize the compressed low-order gradient components to update the current weights of the neural network model. Step S4: When the dynamic segmentation threshold drops to no more than the computing power matching benchmark value, extract the weight state change difference of the neural network model within the accumulation period as the weight evolution vector, project the high-order residual gradient temporarily stored in the delayed residual accumulation branch onto the direction of the weight evolution vector for orthogonal decomposition, truncate the orthogonal components perpendicular to the weight evolution vector, retain the collinear components parallel to the weight evolution vector as compensation gradients, and use the compensation gradients to update the weights of the neural network model.

[0006] Preferably, step S1 includes: step S11, obtaining the estimated computation time of the operator layer on the critical path, and calculating the ratio of the estimated computation time to the real-time computation latency; step S12, multiplying the ratio by a preset weighting coefficient to obtain a computation time weighting term; step S13, calculating the difference between the estimated computation time and the preset weighting coefficient, and multiplying the difference by the memory saturation to obtain a memory saturation weighting term; step S14, adding the computation time weighting term and the memory saturation weighting term to obtain a dynamic segmentation threshold.

[0007] Preferably, step S2 includes: step S21, calculating the product of the dynamic segmentation threshold and the first preset control coefficient, calculating the difference between the product and the threshold as a reduction ratio, multiplying the reduction ratio by a preset maximum allowable quantization bit depth and rounding down to obtain the quantization bit depth; step S22, calculating the product of the dynamic segmentation threshold and the second preset control coefficient, calculating the sum of the product and the threshold as an increment ratio, multiplying the increment ratio by a preset basic accumulation period and rounding up to obtain the accumulation period.

[0008] Preferably, step S3 includes: step S31, obtaining the accumulated value of memory saturation for each cycle before the synchronized compressed low-order gradient component, calculating the dynamic gradient decay factor based on the accumulated value, the dynamic gradient decay factor having a negative exponential mapping relationship with the accumulated value; step S32, multiplying the dynamic gradient decay factor by the low-order gradient component to obtain the corrected low-order gradient component, and using the corrected low-order gradient component as the compressed low-order gradient component to be synchronized to participate in cross-node gradient synchronization.

[0009] Preferably, the method further includes: step S501, after multiplying the dynamic gradient decay factor by the low-order gradient component, calculating the reciprocal of the dynamic gradient decay factor to obtain the loss function adjustment factor; step S502, in the forward propagation stage, multiplying the forward propagation loss value by the loss function adjustment factor to correct the gradient propagation amplitude of the subsequent operator layer.

[0010] Preferably, step S4 includes: step S41, extracting the weight vectors of the neural network model at the start and end nodes of the accumulation period, calculating the vector difference between the weight vector at the end node and the weight vector at the start node, and generating a weight evolution vector; step S42, dequantizing the temporarily stored high-order residual gradient to obtain a continuous-precision residual vector to be compensated; step S43, calculating the inner product of the residual vector to be compensated and the weight evolution vector, dividing by the square of the magnitude of the weight evolution vector, to obtain scalar projection coefficients; step S44, multiplying the scalar projection coefficients by the weight evolution vector to obtain collinear components and using them as compensation gradients; step S45, subtracting the collinear components from the residual vector to be compensated to obtain the corresponding orthogonal components and discarding them.

[0011] Preferably, after the gradient calculation logic of the segmentation operator layer is used as the main update branch and the delay residual accumulation branch, and before extracting the weight state change difference of the neural network model within the accumulation period as the weight evolution vector, the method further includes: step S701, when the dynamic segmentation threshold is higher than the computing power matching benchmark value, the high-order residual gradient separated from the original gradient of the operator layer is transferred to the low-speed buffer; step S702, in multiple consecutive accumulation periods, the newly separated high-order residual gradient and the historical high-order residual gradient temporarily stored in the low-speed buffer are accumulated and stored on the same tensor dimension; step S703, when the memory occupancy rate of the low-speed buffer reaches the preset warning limit, the high-order residual gradient is truncated and compressed bit by bit to control the memory occupancy rate of the low-speed buffer to be lower than the preset warning limit.

[0012] Preferably, step S3 includes: step S33, truncating the low-order gradient components into low-bit-width fixed-point tensors according to the quantization bit depth, and generating compressed gradient data packets; step S34, asynchronously sending the compressed gradient data packets to the target synchronization node through the interconnection network; step S35, dequantizing and restoring the received compressed gradient data packets at the target synchronization node to obtain the updated gradient tensor; and step S36, weighting the updated gradient tensor into the weight parameters of the neural network model in the current iteration cycle.

[0013] Preferably, the method further includes: step S901, when the real-time computation delay of the operator layer exceeds the preset timeout limit and the memory saturation reaches the preset extreme threshold, the quantization bit width is reduced to the preset minimum constraint bit width and the accumulation period is increased to the preset maximum constraint period; step S902, when the dynamic segmentation threshold is reduced to no higher than the computing power matching benchmark value and the maintenance time reaches the preset stable window period, the compensation gradient extracted from the delay residual accumulation branch is superimposed on the current weight in one go to restore the computation graph topology.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. In the training of neural network models for computing power scheduling optimization, the computing power status of computing nodes is judged by real-time calculation of latency and memory saturation. This can promptly detect situations where the computing power supply does not match the operator's computing demand. When computing power is insufficient, the gradient calculation logic is divided into a main update branch and a delay residual accumulation branch, so that low-order gradient components can still participate in weight updates on time, while high-order residual gradients are temporarily stored and processed. This reduces pipeline pauses and memory waits caused by node computing power fluctuations, and maintains the continuous operation of the distributed training process.

[0015] 2. Adjusting the quantization bit depth and accumulation period based on the dynamic splitting threshold allows the gradient compression degree and residual storage time to vary with the computing power status. When computing power is severely scarce, it can reduce the amount of data transmitted across nodes, reduce the bandwidth pressure on the interconnection network, and prevent the low-speed buffer from filling up too quickly. As a result, computing nodes can smoothly adjust gradient synchronization and residual storage methods under different load conditions, balancing communication efficiency and training process stability.

[0016] 3. After the computing power is restored, the temporary high-order residual gradient is orthogonally decomposed using the weight evolution vector. Collinear components that are consistent with the direction of model weight change are retained, while orthogonal components that are prone to causing directional shift are discarded. This can avoid the lag residual gradient being directly superimposed on the current weight, which would cause the update direction to diverge or local oscillation. This makes the compensation gradient more consistent with the current optimization direction of the model. While making up for the accumulated bias of low-precision synchronization, it maintains the convergence stability of model training. Attached Figure Description

[0017] Figure 1This is a flowchart of the neural network model training process for the dynamic segmentation threshold of this invention; Figure 2 This is a diagram showing the state transitions during the training of the neural network model of this invention.

[0018] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] A method for training a neural network model for optimizing computing power scheduling, comprising: Step S1: Obtain the real-time computation latency and memory saturation of each operator layer during the forward propagation of the neural network model, and generate a dynamic segmentation threshold based on the estimated computation time, real-time computation latency and memory saturation. Step S2: When the dynamic segmentation threshold is greater than the computing power matching benchmark, the gradient calculation logic of the segmentation operator layer is the backbone update branch and the delay residual accumulation branch. The quantization bit of the backbone update branch and the accumulation period of the delay residual accumulation branch are determined according to the dynamic segmentation threshold. The backbone update branch retains the low-order gradient components used for weight update, and the delay residual accumulation branch temporarily stores the high-order residual gradient. Step S3: Compress the low-order gradient components according to the quantization bit depth, and synchronize the compressed low-order gradient components to update the current weights of the neural network model. Step S4: When the dynamic segmentation threshold drops to no more than the computing power matching benchmark value, extract the weight state change difference of the neural network model within the accumulation period as the weight evolution vector, project the high-order residual gradient temporarily stored in the delayed residual accumulation branch onto the direction of the weight evolution vector for orthogonal decomposition, truncate the orthogonal components perpendicular to the weight evolution vector, retain the collinear components parallel to the weight evolution vector as compensation gradients, and use the compensation gradients to update the weights of the neural network model.

[0021] Preferably, step S1 includes: step S11, obtaining the estimated computation time of the operator layer on the critical path, and calculating the ratio of the estimated computation time to the real-time computation latency; step S12, multiplying the ratio by a preset weighting coefficient to obtain a computation time weighting term; step S13, calculating the difference between the estimated computation time and the preset weighting coefficient, and multiplying the difference by the memory saturation to obtain a memory saturation weighting term; step S14, adding the computation time weighting term and the memory saturation weighting term to obtain a dynamic segmentation threshold.

[0022] Preferably, step S2 includes: step S21, calculating the product of the dynamic segmentation threshold and the first preset control coefficient, calculating the difference between the product and the threshold as a reduction ratio, multiplying the reduction ratio by a preset maximum allowable quantization bit depth and rounding down to obtain the quantization bit depth; step S22, calculating the product of the dynamic segmentation threshold and the second preset control coefficient, calculating the sum of the product and the threshold as an increment ratio, multiplying the increment ratio by a preset basic accumulation period and rounding up to obtain the accumulation period.

[0023] Preferably, step S3 includes: step S31, obtaining the accumulated value of memory saturation for each cycle before the synchronized compressed low-order gradient component, calculating the dynamic gradient decay factor based on the accumulated value, the dynamic gradient decay factor having a negative exponential mapping relationship with the accumulated value; step S32, multiplying the dynamic gradient decay factor by the low-order gradient component to obtain the corrected low-order gradient component, and using the corrected low-order gradient component as the compressed low-order gradient component to be synchronized to participate in cross-node gradient synchronization.

[0024] Preferably, the method further includes: step S501, after multiplying the dynamic gradient decay factor by the low-order gradient component, calculating the reciprocal of the dynamic gradient decay factor to obtain the loss function adjustment factor; step S502, in the forward propagation stage, multiplying the forward propagation loss value by the loss function adjustment factor to correct the gradient propagation amplitude of the subsequent operator layer.

[0025] Preferably, step S4 includes: step S41, extracting the weight vectors of the neural network model at the start and end nodes of the accumulation period, calculating the vector difference between the weight vector at the end node and the weight vector at the start node, and generating a weight evolution vector; step S42, dequantizing the temporarily stored high-order residual gradient to obtain a continuous-precision residual vector to be compensated; step S43, calculating the inner product of the residual vector to be compensated and the weight evolution vector, dividing by the square of the magnitude of the weight evolution vector, to obtain scalar projection coefficients; step S44, multiplying the scalar projection coefficients by the weight evolution vector to obtain collinear components and using them as compensation gradients; step S45, subtracting the collinear components from the residual vector to be compensated to obtain the corresponding orthogonal components and discarding them.

[0026] Preferably, after the gradient calculation logic of the segmentation operator layer is used as the main update branch and the delay residual accumulation branch, and before extracting the weight state change difference of the neural network model within the accumulation period as the weight evolution vector, the method further includes: step S701, when the dynamic segmentation threshold is higher than the computing power matching benchmark value, the high-order residual gradient separated from the original gradient of the operator layer is transferred to the low-speed buffer; step S702, in multiple consecutive accumulation periods, the newly separated high-order residual gradient and the historical high-order residual gradient temporarily stored in the low-speed buffer are accumulated and stored on the same tensor dimension; step S703, when the memory occupancy rate of the low-speed buffer reaches the preset warning limit, the high-order residual gradient is truncated and compressed bit by bit to control the memory occupancy rate of the low-speed buffer to be lower than the preset warning limit.

[0027] Preferably, step S3 includes: step S33, truncating the low-order gradient components into low-bit-width fixed-point tensors according to the quantization bit depth, and generating compressed gradient data packets; step S34, asynchronously sending the compressed gradient data packets to the target synchronization node through the interconnection network; step S35, dequantizing and restoring the received compressed gradient data packets at the target synchronization node to obtain the updated gradient tensor; and step S36, weighting the updated gradient tensor into the weight parameters of the neural network model in the current iteration cycle.

[0028] Preferably, the method further includes: step S901, when the real-time computation delay of the operator layer exceeds the preset timeout limit and the memory saturation reaches the preset extreme threshold, the quantization bit width is reduced to the preset minimum constraint bit width and the accumulation period is increased to the preset maximum constraint period; step S902, when the dynamic segmentation threshold is reduced to no higher than the computing power matching benchmark value and the maintenance time reaches the preset stable window period, the compensation gradient extracted from the delay residual accumulation branch is superimposed on the current weight in one go to restore the computation graph topology.

[0029] Example 1: This example uses a distributed heterogeneous GPU cluster with 8 compute nodes to train a neural network model with billions of parameters. Hardware thermal limiting and shared memory bandwidth contention cause fluctuations in computation latency on some nodes. During the forward propagation phase, the compute nodes collect the first... Real-time computation latency of each operator layer and memory saturation Generate computing power state vector For the fourth inverse convolution operator layer, i.e. Estimated critical path computation time The latency is 9.75ms, calculated in real time. The latency is 15.0ms, and the memory saturation is [not specified]. The preset weighting coefficient is 0.80. The value is 0.65. The ratio of the estimated critical path computation time to the real-time computation latency is calculated, and the result is... Multiplying the preset weighting coefficient by this ratio yields a computation time weighting term of 0.4225; multiplying the difference between this term and the preset weighting coefficient by the memory saturation level yields a memory saturation weighting term of 0.2800; summing the two weighting terms yields the dynamic segmentation threshold. The value is 0.7025, and the calculation module also includes a small positive real number. for When the real-time calculation latency is 0, this tiny positive real number is used as the divisor to avoid performing division by zero; in this embodiment The value is 15.0 ms, which does not affect the calculation result of the above ratio.

[0030] Dynamic segmentation threshold Benchmark values ​​for matching computing power In comparison, the dynamic segmentation threshold is higher than the computing power matching benchmark. The computing node splits the gradient calculation logic of the fourth inverse convolution operator layer into a main update branch and a delay residual accumulation branch, with a preset maximum allowed quantization bit depth. The preset base accumulation period is 16. The first preset control coefficient is 1. The second preset control coefficient is 0.50. The value is 1.20, from which the quantization bit depth of the trunk update branch is obtained. The accumulation period of the delay residual accumulation branch is 10 bits. The main branch updates the low-order gradient components by quantizing them with 10 bits and sends the compressed low-order gradient components to the target synchronization node through the interconnection network to update the current weights of the neural network model; the delay residual accumulation branch writes the high-order residual gradients into the low-speed buffer for temporary storage.

[0031] When splitting the gradient calculation logic, the computing node obtains the original 32-bit single-precision floating-point gradient tensor output by the fourth inverse convolution operator layer during backpropagation. It extracts the maximum absolute value of each element in the tensor as the scaling reference and maps and truncates each gradient element into a 10-bit signed fixed-point number according to the 10-bit quantization bit. After forming the low-order gradient components, they are sent to the main update branch. Subsequently, the low-order gradient components are back-mapped to 32-bit floating-point precision according to the same scaling reference to generate an approximate gradient tensor. Then, the approximate gradient tensor is subtracted from the original gradient tensor item by item, and the resulting difference tensor is sent to the delay residual accumulation branch as the high-order residual gradient. Before quantization storage, the approximate gradient tensor and the high-order residual gradient are added item by item to equal the original gradient tensor. The delay residual accumulation branch quantizes the high-order residual gradient according to the quantization format used by the low-speed buffer and writes the quantized high-order residual gradient and its scaling reference into the low-speed buffer for subsequent dequantization processing.

[0032] The memory saturation values ​​collected over the two cumulative cycles were as follows: and Calculate the cumulative value of memory saturation in each cycle, and take the attenuation coefficient. The value is 0.10, and the dynamic gradient decay factor is obtained by mapping to the negative exponential function. The dynamic gradient decay factor is multiplied by the low-order gradient component to obtain the corrected low-order gradient component, which is then used as the compressed low-order gradient component to be synchronized across nodes. During the forward propagation phase, the loss function adjustment factor is calculated. The forward propagation loss value is multiplied by the adjustment factor of the loss function to correct the gradient propagation magnitude of the subsequent operator layer.

[0033] After two training iterations, the computing power of the computing nodes recovered, and the dynamic segmentation threshold decreased to [a lower threshold]. The delayed residual accumulation branch begins compensation processing, extracting the neural network model weight vectors at the start and end nodes of the accumulation cycle. The weight vector at the end node is subtracted from the weight vector at the start node to generate the weight evolution vector. Based on the scaling reference stored in the low-speed buffer, the quantized high-order residual gradient is dequantized to obtain a continuously accurate residual vector to be compensated. The inner product of the residual vector to be compensated and the weight evolution vector is calculated and divided by the square of the magnitude of the weight evolution vector to obtain the scalar projection coefficient. The scalar projection coefficient is multiplied by the weight evolution vector to obtain a collinear component parallel to the weight evolution vector, which is used as the compensation gradient. The collinear component is then subtracted from the residual vector to be compensated to obtain an orthogonal component perpendicular to the weight evolution vector, which is discarded. The weights of the neural network model are updated using the compensation gradient.

[0034] During periods of limited computing resources, low-order gradient components are synchronized across nodes according to the current iteration cycle, while high-order residual gradients are temporarily stored in the delayed residual accumulation branch. After computing power is restored, only the collinear components of the high-order residual gradients that are parallel to the weight evolution vector are used for weight compensation, while orthogonal components do not participate in weight updates. Thus, the neural network model adjusts the gradient calculation logic according to real-time computing latency and memory saturation, maintains the operation of the training pipeline during node computing power fluctuations, and ensures that the direction of the compensation gradient is consistent with the weight evolution direction within the accumulation cycle.

[0035] Example 2: This example constructs a distributed GPU computing cluster consisting of 8 computing nodes. Random transmission jitter of 50ms to 200ms is superimposed on the cluster interconnection network, and bandwidth contention noise of 15% to 35% is superimposed on the memory bus of each computing node to simulate the fluctuation of computing power supply caused by hardware thermal throttling and shared memory contention. The test object is a neural network model with 10 billion parameters. The test process is performed on the 4th inverse convolution operator layer. The estimated critical path calculation time of this operator layer is 9.75ms. The test group using this method sequentially performs dynamic segmentation threshold calculation, gradient calculation logic segmentation, quantization bit and cumulative period adjustment, and orthogonal decomposition and compensation gradient update based on weight evolution vector. Control group 1 uses static compilation of computation graph topology and directly linearly superimposes the delayed gradient. Control group 2 retains gradient calculation logic segmentation and low bit quantization synchronization, but cancels orthogonal decomposition based on weight evolution vector and directly fuses the high-order residual gradient after dequantization. Control groups 3a and 3b change the preset weighting coefficients to compare the training state after the coefficients deviate from the calibration range.

[0036] This embodiment will preset the weighting coefficients. Set to 0.65, the first preset control coefficient. Set to 0.50, the second preset control coefficient Set to 1.20, the benchmark value for computing power matching. The value is set to 0.5500, and the preset weighting coefficient is calibrated within the range of 0.60 to 0.80. This coefficient is used to allocate the weight of real-time computing latency and memory saturation in the judgment of computing power status. When this coefficient is below 0.60, the response to computing latency jitter is lagging; when it is above 0.80, memory usage increases and is more likely to reach saturation. The first preset adjustment coefficient is used to control the deceleration rate of quantization bit depth, and its calibration range is 0.30 to 0.70. When it is below 0.30, the quantization compression amplitude is too large; when it is above 0.70, the reduction in cross-node transmission volume is limited. In this embodiment, under the condition of low-speed cache capacity, the calibration range of the second preset adjustment coefficient is 1.00 to 1.50, which is used to control the incremental increase of the accumulation period. When it is higher than 1.50, the temporary storage time of the higher-order residual gradient is extended, and the occupancy of the low-speed cache exceeds its capacity. The computing power matching benchmark value is determined by the baseline computing power status test under the cluster without contention load, and is taken as 1.15 times the average value of the dynamic splitting threshold under the stable state of the cluster. When this value is too low, it is easy to frequently trigger the gradient calculation logic splitting. When it is too high, the response to small computing power fluctuations is weakened.

[0037] Each test group first obtained the real-time computation latency of the fourth inverse convolution operator layer. and memory saturation The ratio of the estimated critical path computation time to the real-time computation latency is calculated. A preset weighting coefficient is multiplied by this ratio to obtain the computation time weighting term. The difference between this and the preset weighting coefficient is then multiplied by the memory saturation to obtain the memory saturation weighting term. The two weighting terms are added together to obtain the dynamic segmentation threshold. The number of quantization bits is determined based on the dynamic segmentation threshold. and cumulative cycle The dynamic gradient decay factor is calculated based on the cumulative value of memory saturation in each cycle. .

[0038] In test group 1, the real-time computation latency of the fourth inverse convolution operator layer was 9.8ms, the memory saturation was 0.42, and the obtained dynamic segmentation threshold was 0.5188, which is lower than the computing power matching benchmark. The backbone update branch used 16-bit quantization, the cumulative period was 1, and the dynamic gradient decay factor was 1.0000. When training with this configuration, the single-node training throughput was 285.4 TFLOPS, the loss value after 1000 iterations was 1.245, and the convergence Top-1 accuracy was 81.2%.

[0039] In test group 2, random transmission jitter and memory bandwidth contention increased the real-time computation latency to 15.0ms and the memory saturation to 0.80. The resulting dynamic segmentation threshold was 0.7025, which was higher than the computing power matching benchmark. The computing node divided the gradient calculation logic of the fourth inverse convolution operator layer into a backbone update branch and a delay residual accumulation branch. The quantization bit depth was adjusted to 10 bits, the accumulation period was adjusted to 2, and a dynamic gradient decay factor of 0.8564 was obtained. The backbone update branch compressed the low-order gradient components and sent them to the target synchronization node, while the delay residual accumulation branch wrote the quantized high-order residual gradient and its scaling benchmark into the low-speed buffer.

[0040] During the forward propagation phase, the loss function adjustment factor is 1.1677. The computation node uses this factor to adjust the gradient propagation magnitude of subsequent operator layers and sets a local gradient amplification mark for subsequent operator layers calculated after the current operator layer. When the backpropagation passes through the fourth backward convolution operator layer, the computation node identifies the gradient calculation logic split state of the operator layer and prevents the loss function adjustment factor from acting on the backward gradient of the layer itself. It only multiplies the dynamic gradient decay factor by the low-order gradient component separated by the layer. Other operator layers that have not undergone gradient calculation logic splitting continue to adjust the gradient propagation magnitude according to the loss function adjustment factor.

[0041] After two cumulative cycles, the computing power of the computing node recovers. The computing node extracts the difference in weight state changes of the neural network model within the two iteration cycles and generates a weight evolution vector. Then, it dequantizes the high-order residual gradient using the scaling benchmark stored in the low-speed cache to obtain a continuous-precision residual vector to be compensated. It then projects the vector onto the direction of the weight evolution vector for orthogonal decomposition. The orthogonal components perpendicular to the weight evolution vector are truncated, and the collinear components parallel to the weight evolution vector are used as compensation gradients to update the current weights of the neural network model. When training according to the above process, the single-node training throughput of test group 2 is 278.2 TFLOPS, the loss value after 1000 iterations is 1.248, and the convergence Top-1 accuracy is 81.1%.

[0042] In test group 3, the real-time computation latency was further increased to 22.5ms, the memory saturation was increased to 0.92, and the dynamic segmentation threshold was 0.8038. Based on this, the computation node adjusted the quantization bit to 9 bits, set the cumulative period to 2, and the dynamic gradient decay factor to 0.7827. After completing the low-order gradient component synchronization and high-order residual gradient compensation according to this configuration, the single-node training throughput was 271.6 TFLOPS, the loss value after 1000 iterations was 1.252, and the convergence Top-1 accuracy was 81.0%.

[0043] Control group 1 uses the same operator layer number, real-time computation latency and memory saturation as test group 2, but does not calculate dynamic splitting threshold and dynamic gradient decay factor. The quantization bit depth is kept at 16 bits and the cumulative period is kept at 1. Since the static compilation of the computation graph topology requires waiting for the nodes with lower computation speed to complete gradient synchronization, the single-node training throughput is 162.5 TFLOPS, the loss value after 1000 iterations is 1.892, and the convergence Top-1 accuracy is 75.4%.

[0044] Control group 2 uses the same input conditions as test group 2, with a dynamic segmentation threshold of 0.7025, a quantization bit depth of 10 bits, a cumulative period of 2, and a dynamic gradient decay factor of 0.8564. This group reduces cross-node transmission through the quantization of low-order gradient components, achieving a single-node training throughput of 276.9 TFLOPS. After computing power is restored, the dequantized high-order residual gradient is directly superimposed on the current weights without truncating the orthogonal components perpendicular to the weight evolution vector. After 1000 iterations, the loss value is 1.534, and the convergence Top-1 accuracy is 78.3%.

[0045] Control group 3a set the preset weighting coefficient to 0.50, and other input conditions were the same as test group 2. The resulting dynamic segmentation threshold was 0.5625, the quantization bit depth was 11 bits, the cumulative period was 2, and the dynamic gradient decay factor was 0.8564. This preset weighting coefficient reduced the weight of real-time computation latency in the dynamic segmentation threshold, thus delaying the response of gradient computation logic segmentation. The single-node training throughput was 212.8 TFLOPS, the loss value after 1000 iterations was 1.385, and the convergence Top-1 accuracy was 79.2%.

[0046] The control group 3b set the preset weighting coefficient to 0.90, and other input conditions remained the same as the test group 2. The resulting dynamic segmentation threshold was 0.8425, the quantization bit depth was 9 bits, the accumulation period was 3, and the dynamic gradient decay factor was 0.7334. After the accumulation period was extended, the high-order residual gradient temporarily stored in the low-speed buffer increased. The single-node training throughput was 268.1 TFLOPS, the loss value after 1000 iterations was 1.491, and the convergence Top-1 accuracy was 78.6%.

[0047] In the above data, when the preset weighting coefficient is in the calibration range of 0.60 to 0.80, the dynamic segmentation threshold can change simultaneously with the real-time computing latency and memory saturation. The main update branch adjusts the quantization bit depth accordingly, and the delay residual accumulation branch adjusts the accumulation period. After the computing power is restored, the temporarily stored high-order residual gradient is dequantized and orthogonally decomposed, and only the collinear components are retained to participate in the weight update. This processing enables the low-order gradient components to continue to complete cross-node synchronization during computing power fluctuations and limits the impact of delay residuals on the direction of weight update of the neural network model.

[0048] Example 3: The current distributed computing node continuously collects the data. Real-time computation latency of each operator layer and memory saturation When real-time computing latency Exceeding the preset timeout limit And memory saturation Reaching the preset extreme threshold When the system triggers the computing power backoff protection mechanism, this embodiment will preset a timeout limit. Set as the estimated critical path computation time 3.0 times, or 29.25ms, will set the preset extreme threshold. Set to 0.95, the fourth inverse convolution operator layer satisfies The real-time computing latency it collects The latency is 32.0ms, and the memory saturation is [not specified]. When the value is 0.96, the extreme value constraint condition is met, and the system will update the quantization bit depth of the main branch. Reduced to the preset minimum constraint bit width That is, 4 bits, and simultaneously the accumulation period of the delay residual accumulation branch. Increase to the preset maximum constraint period That is, 8.

[0049] The main branch update branch compresses the low-order gradient components using 4-bit quantization and transmits the compressed low-order gradient components to the target synchronization node; the delay residual accumulation branch transmits the separated high-order residual gradients to the low-speed buffer for temporary storage, and the node's computing power recovers and the dynamic splitting threshold is met. At that time, the system begins monitoring the preset stable window period. The dynamic segmentation threshold within the time frame is preset in this embodiment, and a stable window period is assumed. Set to 5 consecutive training iterations, when the dynamic segmentation threshold... When the value remains below 0.5500 for five consecutive training iterations, the system extracts the higher-order residual gradient from the delay residual accumulation branch, obtains the compensation gradient through inverse quantization and orthogonal decomposition, and then superimposes the compensation gradient onto the current weights of the neural network model in one go, restoring the computation graph topology to the undecoupled state.

[0050] Over multiple consecutive accumulation periods, newly separated high-order residual gradients and historical high-order residual gradients temporarily stored in the low-speed cache are accumulated and stored with the same tensor dimension, while the system synchronously monitors the memory usage of the low-speed cache. When memory usage Reaching the preset warning limit At a value of 0.85, the high-order residual gradients stored in the low-speed cache are truncated and compressed bit-by-bit, converting them from 32-bit floating-point format to 16-bit fixed-point format to reduce cache usage. Continue to rise until the preset limit is reached. When the value is 0.95, the variance of each channel of the temporarily stored higher-order residual gradient tensor in the discrete dimension is calculated, and the variance value is set below the preset variance threshold. ,Right now By discarding the gradient dimension, the memory usage of the slow cache area is reduced to the preset warning limit. the following.

[0051] During bitwise truncation compression, the higher-order residual gradient tensor to be compressed is first scanned, and the maximum absolute value of all elements in the tensor is taken as the dynamic quantization scaling factor. Then, each 32-bit floating-point element is divided by the dynamic quantization scaling factor for interval normalization, and the normalized value is multiplied by... The obtained fixed-point representation is truncated bitwise, retaining a 16-bit signed fixed-point number. The dynamic quantization scaling factor is appended and stored in the header of the data packet as a 32-bit floating-point number. During subsequent dequantization, the system reads the dynamic quantization scaling factor and restores the 16-bit signed fixed-point number to a high-order residual gradient with continuous precision to maintain its original numerical magnitude.

[0052] When generating the compensation gradient, the system calculates the cumulative period. The weight evolution vector is obtained by comparing the weight vectors of the neural network model at the start and end nodes. To avoid the weight evolution vector magnitude Numerical calculation divergence occurs when the scalar projection coefficients approach zero; a numerical smoothing term is added to the denominator of the scalar projection coefficients. The residual vector to be compensated and the weight evolution vector are then compared. The inner product, divided by The scalar projection coefficients are obtained, and in this embodiment, the numerical smoothing term is used. Set as This value is calibrated based on the smallest change that the current model weight parameters can be stably resolved in 32-bit single-precision floating-point operations, and is taken as 100 times the corresponding basic smoothing amount.

[0053] When the magnitude of the weight evolution vector Less than the preset evolution threshold ,Right now At that time, the system multiplies the residual vector to be compensated by the dynamic gradient decay factor. The weights are then incorporated into the weights of the neural network model, with a preset evolution threshold. Based on the product of the overall parameter magnitude of the neural network model and the current learning rate, 1% of the expected minimum update magnitude of the weights within a single iteration period is taken. When the magnitude of the weight evolution vector is lower than this threshold, the weight change magnitude within the cumulative period is small. The system adopts a weighted fusion method to process the residual vector to be compensated, thereby reducing the projection deviation caused by numerical noise affecting the direction of the weight evolution vector.

[0054] Example 4: This example combines Figures 1 to 2 The training method for neural network models oriented towards computing power scheduling optimization is explained, such as... Figure 1As shown, in step S1, the real-time computation latency and memory saturation of each operator layer of the neural network model are obtained. A dynamic segmentation threshold is generated based on the estimated computation time, real-time computation latency, and memory saturation. In step S2, when the dynamic segmentation threshold is greater than the computing power matching benchmark, the gradient calculation logic of the segmentation operator layer is used as the main update branch and the delay residual accumulation branch. The quantization bit and accumulation period are determined, the low-order gradient components are retained, and the high-order residual gradient is temporarily stored. In step S3, the low-order gradient components are compressed according to the quantization bit, and the compressed low-order gradient components are synchronously used to update the current weights of the neural network model. In step S4, when the threshold drops to no greater than the benchmark, the weight evolution vector is extracted, and the temporarily stored high-order residual gradient is projected onto the vector direction for orthogonal decomposition. The orthogonal components are truncated, and the collinear components are retained as compensation gradients to update the weights.

[0055] like Figure 2 As shown, the system enters the undecoupled computation graph state from the training start point. In this state, forward and backward propagation are performed, neural network model weights are updated, and the baseline computational power is maintained. When the dynamic segmentation threshold is greater than the baseline, the system switches to the gradient computation logic segmentation state. In this state, the system performs trunk update branch synchronization, delay residual accumulation and temporary storage, and writes the data to the low-speed cache. When a delay timeout occurs and the video memory reaches its extreme value, the system enters the computational power backoff protection mechanism. This mechanism lowers the constraint bit width to the minimum, raises it to the maximum constraint period, and controls the upper limit of memory usage. When the threshold is lowered to no greater than the baseline value in either the gradient computation logic segmentation state or the computational power backoff protection mechanism, the system transitions to the residual compensation processing stage. In this stage, the weight evolution vector is extracted, dequantized, orthogonally decomposed, and the current weights are compensated with collinear components. When the preset stable window period is maintained, the system returns to the undecoupled computation graph state.

[0056] Example 5: Before the neural network model is formally deployed and trained, the computing nodes perform a pre-warm-up sampling step to initialize the calibration. Estimated critical path computation time for each operator layer To address the heterogeneous hardware characteristics of computing nodes, 10 single-batch forward and backward propagation runs were conducted in an environment without external load contention, collecting the first... The computation delay of each operator layer in steady state is used, and the median of the measurement results is taken as the estimated critical path computation time for that operator layer. When a new operator layer is added to the computation graph, the system re-executes the pre-warm-up sampling step and updates the time. This allows the calculation of the dynamic segmentation threshold to use the computational delay benchmark corresponding to the computational node where the operator layer is located.

[0057] During distributed training, the system continuously monitors the forward propagation data input scheduler. When the training batch size or input tensor dimension is adjusted, and the change in the training batch size exceeds 20%, the dynamic splitting threshold update within the current iteration cycle is frozen. Subsequently, within three consecutive iteration cycles, the real-time computation latency of the operator layer under conditions of no external load contention is re-acquired, and the arithmetic mean of the three acquisition results is calculated. This result is used as the new estimated critical path computation time. Thus, after the training batch size or input tensor dimension changes, the estimated critical path computation time is re-determined according to the actual computational load of the operator layer.

[0058] During cluster-distributed training operations, the system uses a sliding window calibration method to update the computing power matching benchmark value. The sliding window contains 100 iteration cycles. The system continuously collects the dynamic segmentation threshold when the computing node is in a stable working condition and calculates the arithmetic mean of the dynamic segmentation threshold within the window. When the absolute value of the relative deviation between the arithmetic mean and the current computing power matching benchmark value is greater than 15%, the arithmetic mean is assigned to the computing power matching benchmark value and updated to the gradient calculation logic segmentation control process. As the computing performance changes due to hardware heat accumulation, the main update branch and the delay residual accumulation branch perform segmentation and fusion judgment based on the updated computing power matching benchmark value.

[0059] Example 6: When the dynamic partitioning threshold of multiple operator layers in a distributed cluster fluctuates repeatedly around the computing power matching benchmark, the system introduces a hysteresis interval parameter in the partitioning determination of the gradient calculation logic. And set it to 0.0500, when the first Dynamic segmentation threshold of each operator layer satisfy ,Right now At that time, the first segment was cut. The gradient calculation logic of the inverse operator layer; after computing power recovery, when the dynamic segmentation threshold is satisfied... ,Right now When the delay residual accumulation branch is merged with the main update branch, the gradient calculation logic is restored. Each operator layer is divided and determined in sequence according to the backpropagation topology. The actual number of operator layers divided in a single iteration cycle does not exceed 20% of the total number of operator layers, so as to limit the cumulative deflection of gradient direction caused by multiple operator layers temporarily storing high-order residual gradients at the same time.

[0060] When transient packet loss occurs in a heterogeneous interconnected network, the system collects the network packet loss rate in real time through the network interface. And adjust the first preset control coefficient accordingly. Second preset control coefficient ,when hour, Keep the base setting at 0.50. Keep the base setting at 1.20; when At that time, according to and Update the two preset control coefficients and set them respectively. and The upper limit constraint, where, Used to constrain the linear correction result; before the result is included in the cumulative period calculation, the part exceeding 1.50 is taken as 1.50, so that the second preset control coefficient actually involved in the calculation is kept within the effective value boundary of 1.00 to 1.50.

[0061] The system recalculates the quantization bit depth of the main branch update based on the updated first preset control coefficient. The cumulative period of the delay residual cumulative branch is recalculated based on the second preset control coefficient actually involved in the calculation. When the network packet loss rate increases, the quantization bits of the backbone update branch are reduced accordingly, and the accumulation period of the delay residual accumulation branch is extended accordingly, so as to reduce the amount of compressed gradient data transmitted across nodes in the current iteration cycle.

[0062] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for training a neural network model oriented to computing power scheduling optimization, characterized in that, include: Step S1: Obtain the real-time computation latency and memory saturation of each operator layer during the forward propagation of the neural network model, and generate a dynamic segmentation threshold based on the estimated computation time, real-time computation latency and memory saturation. Step S2: When the dynamic segmentation threshold is greater than the computing power matching benchmark, the gradient calculation logic of the segmentation operator layer is the backbone update branch and the delay residual accumulation branch. The quantization bit of the backbone update branch and the accumulation period of the delay residual accumulation branch are determined according to the dynamic segmentation threshold. The backbone update branch retains the low-order gradient components used for weight update, and the delay residual accumulation branch temporarily stores the high-order residual gradient. Step S3: Compress the low-order gradient components according to the quantization bit depth, and synchronize the compressed low-order gradient components to update the current weights of the neural network model. Step S4: When the dynamic segmentation threshold drops to no more than the computing power matching benchmark value, extract the weight state change difference of the neural network model within the accumulation period as the weight evolution vector, project the high-order residual gradient temporarily stored in the delayed residual accumulation branch onto the direction of the weight evolution vector for orthogonal decomposition, truncate the orthogonal components perpendicular to the weight evolution vector, retain the collinear components parallel to the weight evolution vector as compensation gradients, and use the compensation gradients to update the weights of the neural network model.

2. The computing power scheduling optimization-oriented neural network model training method according to claim 1, characterized in that, Step S1 includes: Step S11, obtaining the estimated computation time of the operator layer on the critical path, and calculating the ratio of the estimated computation time to the real-time computation latency; Step S12, multiplying the ratio by a preset weighting coefficient to obtain a computation time weighting term; Step S13, calculating the difference between the estimated computation time and the preset weighting coefficient, and multiplying the difference by the memory saturation to obtain a memory saturation weighting term; Step S14, adding the computation time weighting term and the memory saturation weighting term to obtain a dynamic segmentation threshold.

3. The computing power scheduling optimization-oriented neural network model training method according to claim 1, characterized in that, Step S2 includes: Step S21, calculating the product of the dynamic segmentation threshold and the first preset control coefficient, calculating the difference between the product and the threshold as the reduction ratio, multiplying the reduction ratio by the preset maximum allowable quantization bits and rounding down to obtain the quantization bits; Step S22, calculating the product of the dynamic segmentation threshold and the second preset control coefficient, calculating the sum of the product and the threshold as the increment ratio, multiplying the increment ratio by the preset basic accumulation period and rounding up to obtain the accumulation period.

4. The neural network model training method for computing power scheduling optimization according to claim 1, characterized in that, Step S3 includes: Step S31, before the synchronized compressed low-order gradient component, obtain the accumulated value of the memory saturation of each cycle, calculate the dynamic gradient decay factor based on the accumulated value, and the dynamic gradient decay factor has a negative exponential mapping relationship with the accumulated value; Step S32, multiply the dynamic gradient decay factor by the low-order gradient component to obtain the corrected low-order gradient component, and use the corrected low-order gradient component as the compressed low-order gradient component to be synchronized to participate in cross-node gradient synchronization.

5. The neural network model training method for computing power scheduling optimization according to claim 4, characterized in that, The method further includes: step S501, after multiplying the dynamic gradient decay factor by the low-order gradient component, calculating the reciprocal of the dynamic gradient decay factor to obtain the loss function adjustment factor; step S502, in the forward propagation stage, multiplying the forward propagation loss value by the loss function adjustment factor to correct the gradient propagation amplitude of the subsequent operator layer.

6. The neural network model training method for computing power scheduling optimization according to claim 1, characterized in that, Step S4 includes: Step S41, extracting the weight vectors of the neural network model at the start and end nodes of the accumulation period, calculating the vector difference between the weight vector at the end node and the weight vector at the start node, and generating a weight evolution vector; Step S42, dequantizing the temporarily stored high-order residual gradient to obtain a continuous-precision residual vector to be compensated; Step S43, calculating the inner product of the residual vector to be compensated and the weight evolution vector, dividing by the square of the magnitude of the weight evolution vector, to obtain scalar projection coefficients; Step S44, multiplying the scalar projection coefficients by the weight evolution vector to obtain collinear components and using them as compensation gradients; Step S45, subtracting the collinear components from the residual vector to be compensated to obtain the corresponding orthogonal components and discarding them.

7. The neural network model training method for computing power scheduling optimization according to claim 1, characterized in that, After the gradient calculation logic of the segmentation operator layer becomes the main update branch and the delayed residual accumulation branch, and before extracting the weight state change difference of the neural network model within the accumulation period as the weight evolution vector, the method further includes: step S701, when the dynamic segmentation threshold is higher than the computing power matching benchmark value, the high-order residual gradient separated from the original gradient of the operator layer is transferred to the low-speed buffer; step S702, within multiple consecutive accumulation periods, the newly separated high-order residual gradient and the historical high-order residual gradient temporarily stored in the low-speed buffer are accumulated and stored on the same tensor dimension; step S703, when the memory occupancy rate of the low-speed buffer reaches the preset warning limit, the high-order residual gradient is truncated and compressed bit by bit to control the memory occupancy rate of the low-speed buffer to be lower than the preset warning limit.

8. The neural network model training method for computing power scheduling optimization according to claim 1, characterized in that, Step S3 includes: Step S33, truncating the low-order gradient components into low-bit-width fixed-point tensors according to the quantization bit depth to generate compressed gradient data packets; Step S34, asynchronously sending the compressed gradient data packets to the target synchronization node through the interconnection network; Step S35, dequantizing and restoring the received compressed gradient data packets at the target synchronization node to obtain the updated gradient tensor; Step S36, weighting the updated gradient tensor into the weight parameters of the neural network model in the current iteration cycle.

9. The neural network model training method for computing power scheduling optimization according to claim 1, characterized in that, The method also includes: step S901, when the real-time computation delay of the operator layer exceeds the preset timeout limit and the memory saturation reaches the preset extreme threshold, the quantization bit width is reduced to the preset minimum constraint bit width and the accumulation period is increased to the preset maximum constraint period; step S902, when the dynamic segmentation threshold is reduced to no higher than the computing power matching benchmark value and the maintenance time reaches the preset stable window period, the compensation gradient extracted from the delay residual accumulation branch is superimposed on the current weight in one go to restore the computation graph topology.

Citation Information

Patent Citations

  • A dynamic layered gradient compression method for neural network models

    CN118052260B