A split learning system and bandwidth-aware neural subspace compression method, device and equipment and medium thereof

Abstract

The application provides a cut learning system and a bandwidth-aware neural subspace compression method, device and equipment and medium thereof, and relates to the technical field of cut learning. The bandwidth-aware neural subspace compression method comprises: obtaining to-be-compressed data. According to the to-be-compressed data, a tensor singular value spectrum is estimated by an adaptive rank selection module based on a random subspace method, and is trimmed in combination with an energy coverage threshold, a bandwidth budget and a rank upper limit to obtain a compression rank. According to the to-be-compressed data, the compression rank and an error feedback item of a previous round of iteration, left and right factor matrices are alternately updated by an alternating orthogonal approximation module, and subspaces in row and column directions are orthogonalized to obtain row and column subspace orthogonal bases and a low-rank approximation. According to the to-be-compressed data and the low-rank approximation result, an error feedback item of the current round of iteration is obtained. The row and column subspace orthogonal bases are compressed data suitable for transmission.

Description

Technical Field

[0001] This invention relates to the field of segmentation learning technology, and more specifically, to a segmentation learning system and its bandwidth-aware neural subspace compression method, apparatus, device, and medium. Background Technology

[0002] In the field of distributed machine learning, partitioned learning, as an effective paradigm, can significantly reduce the computational and memory pressure on clients. However, the resulting communication overhead is becoming increasingly prominent. Each training batch requires multiple transmissions of activation values ​​and gradients, causing communication to become a bottleneck in system performance, especially in low-bandwidth or high-latency network environments. There is an urgent need for efficient communication compression techniques in real-world applications to balance model training efficiency with resource constraints.

[0003] Existing technologies have proposed various communication compression methods to alleviate the transmission burden in distributed training. These methods include gradient sparsity techniques, such as Top-k sparsity and randomized sparsity, which reduce dimensionality by selectively retaining or discarding gradient elements; gradient quantization methods, such as SignSGD and QSGD, which map continuous values ​​to finite-width representations; low-rank decomposition methods, such as PowerSGD, which reduce the number of parameters using matrix approximation; and differential coding techniques, such as EF-SGD, which transmit differential information based on temporal correlation. These schemes reduce the communication load to some extent, but generally employ fixed parameters or a single operational level.

[0004] Existing methods have significant limitations, making it difficult to achieve an ideal trade-off in diverse distributed scenarios. Sparsification techniques lack adaptability, potentially discarding crucial gradient information or introducing excessive noise, impacting convergence speed. Quantization methods rely on fixed bit widths, failing to dynamically adapt to changes in gradient distribution, leading to early training accuracy loss or insufficient compression ratio. Low-rank decomposition processes are computationally expensive and struggle to capture complex tensor structures, affecting model stability. More importantly, existing techniques lack a systematic integration mechanism, resulting in an insufficient balance between compression ratio and information preservation, making it difficult to meet the long-term training needs in low-bandwidth environments. Summary of the Invention

[0005] The present invention provides a segmentation learning system and a bandwidth-aware neural subspace compression method, apparatus, device and medium thereof, to improve at least one of the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a method for bandwidth-aware neural subspace compression of a segmented learning system, comprising steps S1 to S4.

[0007] S1. Obtain the data to be compressed.

[0008] S2. Based on the data to be compressed, the adaptive rank selection module estimates the tensor singular value spectrum using the randomized subspace method, and performs pruning by combining the energy coverage threshold, bandwidth budget, and rank upper limit to obtain the compressed rank.

[0009] S3. Based on the data to be compressed, the compression rank, and the error feedback term from the previous iteration, the alternating orthogonal approximation module alternately updates the left and right factor matrices and orthogonally transforms the subspaces in the row and column directions to obtain the orthogonal basis and low-rank approximation of the row and column subspaces. The orthogonal basis of the row and column subspaces is the compressed data suitable for transmission.

[0010] S4. Based on the data to be compressed and the low-rank approximation result, obtain the error feedback term for the current iteration.

[0011] Preferably, the data to be compressed includes the tensor to be compressed and / or bandwidth and / or data precision and / or energy coverage threshold.

[0012] Preferably, the adaptive rank selection module is suitable for rank selection in both single tensors and multi-layer / multi-tensor scenarios.

[0013] Preferably, in the single tensor case, the constrained optimal rank selection includes:

[0014] .

[0015] .

[0016] In the formula, To retain the previous The cumulative energy at each singular value. The compressed rank that is currently retained. It serves as a summation index, without a fixed physical meaning, and is used to traverse a sequence during the summation process. For the first A singular value. This refers to the activation matrix or gradient matrix in segmentation learning. Let be the total energy of the matrix. is the row number of the matrix. is the number of columns in the matrix.

[0017] The cost of one communication for low-rank factorization is:

[0018] .

[0019] .

[0020] In the formula, When the compressed rank is The number of bytes required to be transmitted. The number of bytes stored for a single floating-point number, if using but If using or but .

[0021] Combination with energy threshold:

[0022] .

[0023] In the formula, To meet the energy coverage threshold The least rank.

[0024] Continuous relaxation and KKT threshold:

[0025] .

[0026] In the formula, Relaxation variables introduced for continuous relaxation. This is the maximum bandwidth budget for a single communication.

[0027] First-order optimality under Lagrange multipliers gives the marginal threshold:

[0028] .

[0029] In the formula, This is the optimal solution for slack variables. It is a Lagrange multiplier.

[0030] Preferably, the joint allocation operation for multi-layer / multi-tensor scenarios is as follows:

[0031] .

[0032] Greedy approximation:

[0033] .

[0034] In the formula, For the first The final rank assigned to the layer. This represents the total number of network layers. For the first Layer rank preservation The cumulative energy at that time. For the first Rank of layers Communication overhead during the process. The total bandwidth budget for a single communication of multiple layers / tensors. For the first The maximum available rank in the layer implementation. For the first Layer Energy gain per byte in a singular direction. For the first Layer A singular value. For the first Layer The square of a singular value. Indicates the first The number of rows in the layer matrix. Indicates the first The number of columns in the layer matrix.

[0035] The preferred expression for the alternating orthogonal approximation is:

[0036] .

[0037] .

[0038] .

[0039] In the formula, For the first The subspace orthogonal basis of rows obtained by round iteration. For orthogonalization. This refers to the activation matrix or gradient matrix in segmentation learning. This is the error feedback term from the previous iteration. It is an orthogonal basis for the subspace of the columns obtained in the previous iteration. For the first The subspace orthogonal basis of the columns obtained by the iteration. This indicates transpose. For the first The low-rank approximation of the next iteration.

[0040] The preferred approach, using alternating orthogonal approximations, yields the following relative residuals and round-by-round complexity:

[0041] .

[0042] .

[0043] In the formula, express The relative residual of the next iteration. It is the Frobenius norm. This represents the computational complexity of each iteration in the alternating orthogonal iteration. This is a representation of a complex metric level. is the row number of the matrix. is the number of columns in the matrix. The compressed rank that is currently retained.

[0044] Preferably, the expression for error feedback is:

[0045] .

[0046] .

[0047] In the formula, Indicates the first Error feedback term for the next iteration. This is the attenuation coefficient. This is the error feedback term from the previous iteration. This refers to the activation matrix or gradient matrix in segmentation learning. For the first The low-rank approximation of the next iteration. The weighting coefficients for historical residuals. For the first The low-rank approximation of the next iteration.

[0048] Secondly, the present invention provides a bandwidth-sensing neural subspace compression device, which includes a data acquisition module, a rank acquisition module, an update module and an error module.

[0049] The data acquisition module is used to acquire the data to be compressed.

[0050] The rank acquisition module is used to obtain the compressed rank by having the adaptive rank selection module estimate the tensor singular value spectrum based on the randomized subspace method according to the data to be compressed, and then pruning it by combining the energy coverage threshold, bandwidth budget and rank upper limit.

[0051] The update module is used to alternately update the left and right factor matrices based on the data to be compressed, the compression rank, and the error feedback term from the previous iteration, using an alternating orthogonal approximation module, and orthogonalize the subspaces in the row and column directions to obtain the orthogonal basis and low-rank approximation of the row and column subspaces. The orthogonal basis of the row and column subspaces is the compressed data suitable for transmission.

[0052] The error module is used to obtain the error feedback term for the current iteration based on the data to be compressed and the low-rank approximation result.

[0053] Thirdly, the present invention provides a bandwidth-aware neural subspace compression device, comprising a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a bandwidth-aware neural subspace compression method for a segmentation learning system as described in any paragraph of the first aspect.

[0054] Fourthly, the present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform a bandwidth-aware neural subspace compression method for a segmentation learning system as described in any paragraph of the first aspect.

[0055] Fifthly, the present invention provides a segmentation learning system, comprising a client and a server connected by communication. The client and the server are adapted to use a bandwidth-aware neural subspace compression method for a segmentation learning system as described in any paragraph of the first aspect when transmitting data.

[0056] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0057] The bandwidth-aware neural subspace compression method of the segmentation learning system of the present invention can significantly reduce the communication overhead in segmentation learning while maintaining model accuracy and convergence performance. Attached Figure Description

[0058] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the specific embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some specific embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating the bandwidth-aware neural subspace compression method.

[0060] Figure 2 This is the architecture diagram of the segmentation learning system. Detailed Implementation

[0061] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0062] Example 1, please refer to Figures 1 to 2 The first embodiment of the present invention provides a bandwidth-aware neural subspace compression method for a segmentation learning system, which can be executed by a segmentation learning system. The segmentation learning system includes a client and a server connected by communication. Specifically, both the client and the server use the bandwidth-aware neural subspace compression method of the segmentation learning system to compress data before transmitting it to each other, and upon receiving compressed data, they re-approximate the data based on the received compressed data.

[0063] The bandwidth-aware neural subspace compression method of this invention achieves efficient compression of activations and gradients in segmentation learning through the collaborative work of four key modules: adaptive rank selection, alternating orthogonal approximation, error feedback loop, and symmetric compressed transmission. First, adaptive rank selection dynamically determines an appropriate low-rank size based on randomized subspace spectrum estimation and bandwidth constraints, thereby controlling communication load while ensuring energy coverage. Then, alternating orthogonal approximation obtains a stable low-rank approximation representation with low complexity by alternately updating the left and right factor matrices while maintaining orthogonality. Next, the error feedback loop re-injects residuals in each iteration to compensate for information loss caused by low-rank truncation and prevent error accumulation from affecting training convergence. Finally, symmetric compressed transmission simultaneously applies the above mechanisms between the client and server to compress and decompress activations and gradients, forming a closed loop. This multi-level collaborative framework balances communication efficiency and model performance, maintaining low distortion and stable convergence under different bandwidth environments.

[0064] The bandwidth-aware neural subspace compression method for segmented learning systems includes steps S1 to S4.

[0065] S1. Obtain the data to be compressed. The data to be compressed includes information such as the tensor to be compressed, bandwidth, data precision, and energy coverage threshold.

[0066] S2. Based on the data to be compressed, the adaptive rank selection module estimates the singular value spectrum of the tensor using the randomized subspace method, and performs pruning by combining the energy coverage threshold, bandwidth budget, and rank upper limit to obtain the compressed rank. The adaptive rank selection module is suitable for optimal rank selection in the case of a single tensor, and for joint allocation in the case of multiple layers / multiple tensors.

[0067] The adaptive rank selection module estimates the singular value spectrum of the tensor based on the randomized subspace method and performs pruning by combining an energy coverage threshold, bandwidth budget, and rank upper limit to determine the final rank. The energy coverage threshold ensures that the main spectral energy is preserved. The bandwidth constraint ensures that the transmission volume does not exceed the budget. The rank upper limit avoids excessive overhead. This mechanism dynamically adapts to the energy distribution of tensors in different layers and batches, achieving a balance between compression ratio and information preservation.

[0068] In the case of a single tensor, the constrained optimal rank selection includes:

[0069] .

[0070] .

[0071] In the formula, To retain the previous The cumulative energy at each singular value. The current compressed rank (i.e., the number of singular values ​​retained after artificial truncation, which is also the dimension used in the low-rank approximation). It serves as a summation index, without a fixed physical meaning, and is used to traverse a sequence during the summation process. For the first The singular value represents the matrix at the nth singular value. The magnitude of energy in each principal direction. For the first The square of a singular value. This is used to segment the activation matrix or gradient matrix (i.e., the data to be compressed) in the learning process. The total energy of the matrix (sum of squares of all singular values). The number of rows in the matrix (corresponding to the feature dimension). This represents the number of columns in the matrix (corresponding to the number of channels). This indicates taking the smaller value.

[0072] The overhead (in bytes) of a single communication is factored by low rank:

[0073] .

[0074] .

[0075] In the formula, When the compressed rank is The number of bytes required to be transmitted. The number of bytes stored for a single floating-point number, if using but If using or but .

[0076] Combining with energy threshold (two-stage pruning):

[0077] .

[0078] In the formula, To meet the energy coverage threshold The least rank. To select the minimum rank that satisfies the conditions.

[0079] Energy coverage threshold This is used to control the energy coverage of the low-rank approximation in the adaptive rank selection module, thereby balancing compression ratio and information preservation. It is a hyperparameter that is preset by the user according to the specific application scenario and needs.

[0080] Continuous relaxation and KKT threshold:

[0081] .

[0082] In the formula, To select the maximum rank that satisfies the conditions. The slack variable introduced for continuous relaxation represents the th... Should the singular direction be preserved? This direction should be retained completely. Completely discard that direction. This is an intermediate state, representing "partial retention" or "fractional rank". (i.e., subject to) means "to satisfy..." or "to be bound by...". The maximum bandwidth budget for a single communication (in bytes).

[0083] First-order optimality under Lagrange multipliers gives the marginal threshold:

[0084] .

[0085] In the formula, This is the optimal solution for slack variables. It is a Lagrange multiplier.

[0086] The joint assignment operation for multi-level / multi-tensor scenarios is as follows:

[0087] .

[0088] Greedy approximation:

[0089] .

[0090] In the formula, For the first The final rank of the layer, determined by the greedy algorithm, is the rank of the layer. The number of singular directions preserved in the layer. This represents the total number of network layers, i.e., the total number of multiple layers / tens. For the first Layer (or the first) (each tensor) preserves its rank The cumulative energy at that time. For the first Layer (or the first) Take the rank of each tensor. Communication overhead (in bytes). Total bandwidth budget for a single multi-layer / multi-tensor communication (in bytes). For the first The maximum available rank in the layer implementation. For the first Layer Energy gain (density) per byte in a singular direction. For the first Layer A singular value. For the first Layer The square of a singular value. Indicates the first The number of rows in the layer matrix. Indicates the first The number of columns in the layer matrix.

[0091] Greedy approximation is the core implementation method in the scenario of "multi-layer / multi-tensor joint allocation". Its role is to "quickly find an approximately optimal rank allocation scheme that satisfies the total bandwidth constraint with low computational complexity".

[0092] S3. Based on the data to be compressed, the compression rank, and the error feedback term from the previous iteration, the alternating orthogonal approximation module alternately updates the left and right factor matrices and orthogonally transforms the subspaces in the row and column directions to obtain the orthogonal basis and low-rank approximation of the row and column subspaces. The orthogonal basis of the row and column subspaces is the compressed data suitable for transmission.

[0093] Based on a determined rank, the alternating orthogonal approximation iteratively updates the left and right factor matrices alternately, applying orthogonalization after each update to obtain a stable low-rank approximation. Compared with one-time truncated singular value decomposition, the alternating orthogonal approximation has advantages such as low computational complexity (approximately O(mnr)) and good numerical stability, making it particularly suitable for fast approximation of high-dimensional tensors.

[0094] The expression for the alternating orthogonal approximation is:

[0095] .

[0096] .

[0097] .

[0098] In the formula, For the first The subspace orthogonal basis of rows obtained by round iteration. For orthogonalization. This refers to the activation matrix or gradient matrix in segmentation learning. This is the error feedback term from the previous iteration. It is an orthogonal basis for the subspace of the columns obtained in the previous iteration. For the first The subspace orthogonal basis of the columns obtained by the iteration. This indicates transpose. For the first The low-rank approximation of the next iteration (i.e.) (Approximate).

[0099] , , Represent real numbers, is the row number of the matrix. is the number of columns in the matrix.

[0100] The relative residuals and round-by-round complexity of the alternating orthogonal approximation are:

[0101] .

[0102] .

[0103] In the formula, express The relative residual of each iteration is used to measure... and The approximation error between them. It is the Frobenius norm, used to measure the overall energy of a matrix. This represents the computational complexity of each iteration in the alternating orthogonal iteration. This is a representation of a complex metric level. is the row number of the matrix. is the number of columns in the matrix. The compressed rank that is currently retained.

[0104] S4. Based on the data to be compressed and the low-rank approximation result, obtain the error feedback term for the current iteration.

[0105] To avoid permanent information loss caused by low-rank truncation, the error feedback module injects residuals back into the compression process as momentum in each iteration. This mechanism dynamically compensates for compression errors, suppresses long-range cumulative bias, and thus enhances the stability and final accuracy of model convergence. In this embodiment, residual accumulation adopts an exponential decay form to ensure a balance between the utilization of historical information and stability.

[0106] The expression for error feedback is:

[0107] .

[0108] .

[0109] In the formula, Indicates the first The error feedback term of the next iteration is used to accumulate the residual information generated by the low-rank truncation. This is the attenuation coefficient, used to control the proportion of historical errors retained in the current iteration. This is the error feedback term from the previous iteration. This refers to the activation matrix or gradient matrix in segmentation learning. For the first The low-rank approximation of the next iteration. The weighting coefficients for historical residuals. For the first The low-rank approximation of the next iteration. This refers to the sequence number of the historical iteration.

[0110] The core objective of error feedback correction is to accumulate the residuals generated by low-rank truncation and avoid permanent information loss. The two expressions for error feedback fully describe the calculation process of error feedback from two dimensions: "recursive update logic" and "the essence of historical residual accumulation." They are equivalent but express different perspectives, and together they help to understand how errors are dynamically propagated and accumulated.

[0111] In the segmentation learning system, the bandwidth-aware neural subspace compression method described above is used to perform symmetrical compression and transmission of the transmitted data. Specifically, in the communication between the client and server of the segmentation learning system, the bandwidth-aware neural subspace compression method of the segmentation learning system applies the same low-rank compression and decompression process to both the uplink activation and the downlink gradient, maintaining a closed-loop training process. The client sends the compressed factor matrix and a small amount of metadata, and the server reconstructs the approximate representation and completes subsequent training. After receiving the gradient, the server compresses it using the same mechanism and sends it down, and the client reconstructs the gradient and updates its local parameters.

[0112] This symmetrical design ensures information consistency while further improving communication efficiency.

[0113] The expression for symmetric compressed transmission is as follows:

[0114] Uplink: .

[0115] Downlink: .

[0116] In the formula, This is used to segment the activation matrix or gradient matrix (i.e., the data to be compressed) in the learning process. for The subspace of the rows is an orthogonal basis. for The subspaces of the columns are orthogonal bases. for The low-rank approximation. This indicates transpose. For the process The gradient obtained during training. for The subspace of the rows is an orthogonal basis. for The subspaces of the columns are orthogonal bases. for The low-rank approximation.

[0117] Complexity and Communication Analysis of Bandwidth-Aware Neural Subspace Compression: In large-scale distributed training, the computational and communication complexity of an algorithm are key indicators of its practicality. The computational complexity of the neural subspace compressor mainly comes from randomized spectral estimation and alternating orthogonal approximation, both of which are in the low polynomial order. The communication complexity is proportional to the rank and can be flexibly controlled by the bandwidth budget. Overall, this method balances approximation quality and practical efficiency, and can adapt to diverse distributed environments.

[0118] Randomized spectral estimation complexity:

[0119] .

[0120] Communication load and compression ratio:

[0121] .

[0122] In the formula, This represents the total computational complexity of the randomized spectrum estimation process. The power is the number of iterations. This is a representation of a complex metric level. is the row number of the matrix. is the number of columns in the matrix. The compressed rank that is currently retained. This represents the oversampling dimension for randomized spectrum estimation. The payload (number of bytes transmitted) for a single communication. The number of bytes stored for a single floating-point number. This is the compression ratio.

[0123] The bandwidth-aware neural subspace compression method of this invention can construct a low-rank compression pipeline oriented towards segmentation learning. The low-rank compression pipeline sequentially comprises four stages: adaptive rank selection, alternating orthogonal approximation, error feedback correction, and symmetric compression transmission. First, the adaptive rank selection module receives information such as the tensor to be compressed, bandwidth, data precision, and energy coverage threshold, and dynamically prunes the rank using lightweight spectral estimation combined with budget and threshold. Then, the alternating orthogonal approximation module takes the selected rank and the current tensor as input, alternately updating and orthogonalizing the subspaces in the row and column directions to quickly obtain a stable low-rank factor representation. Next, the error feedback correction module reads the residual of the current approximation and the historical error buffer, accumulating the residual information as momentum and reinjecting it into the compression process in the next round. Finally, the symmetric compression transmission applies the above mechanism uniformly in both uplink and downlink. Uplink: The client compresses the activation into a factor along with a small amount of metadata and sends it to the server. The server receives it, reconstructs the approximate activation, and continues calculation. Downlink: The server obtains the gradient and compresses it according to the same process before sending it to the client. The client receives it, reconstructs the approximate gradient, and completes the local update.

[0124] Through the synergy of "adaptive rank determination, low-rank factorization, error backinjection, and symmetric transmission," the scheme significantly reduces the number of bytes transmitted with approximately linear, rank-controlled communication overhead under different bandwidth conditions, while maintaining low distortion and stable convergence, thus adapting to the actual deployment needs of resource-constrained devices. The entire pipeline optimizes the implementation of tensor decomposition and residual accumulation, ensuring low computational overhead and adaptability to resource-constrained edge devices. The four core methods are described below.

[0125] The bandwidth-aware neural subspace compression method of the segmentation learning system in this invention integrates four core modules: adaptive rank selection, alternating orthogonal approximation, error feedback correction, and symmetric compressed transmission. This significantly reduces communication overhead in segmentation learning while maintaining model accuracy and convergence performance. The method utilizes the spectral energy distribution characteristics of tensors to achieve adaptive rank pruning, effectively avoiding information loss by combining low-rank approximation and residual backinjection, and employs a symmetric compression mechanism in both uplink and downlink to ensure closed-loop consistency during training. Compared to traditional single compression methods, this invention not only balances compression ratio and information preservation but also possesses good adaptability and versatility, making it particularly suitable for resource-constrained distributed training environments with low bandwidth and high latency.

[0126] Example 2: The present invention provides a bandwidth-aware neural subspace compression device, which includes a data acquisition module, a rank acquisition module, an update module and an error module.

[0127] The data acquisition module is used to acquire the data to be compressed.

[0128] The rank acquisition module is used to obtain the compressed rank by having the adaptive rank selection module estimate the tensor singular value spectrum based on the randomized subspace method according to the data to be compressed, and then pruning it by combining the energy coverage threshold, bandwidth budget and rank upper limit.

[0129] The update module is used to alternately update the left and right factor matrices based on the data to be compressed, the compression rank, and the error feedback term from the previous iteration, using an alternating orthogonal approximation module, and orthogonalize the subspaces in the row and column directions to obtain the orthogonal basis and low-rank approximation of the row and column subspaces. The orthogonal basis of the row and column subspaces is the compressed data suitable for transmission.

[0130] The error module is used to obtain the error feedback term for the current iteration based on the data to be compressed and the low-rank approximation result.

[0131] Example 3: This invention provides a bandwidth-aware neural subspace compression device, which includes a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a bandwidth-aware neural subspace compression method for a segmentation learning system as described in any paragraph of Example 1.

[0132] Example 4: This invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform a bandwidth-aware neural subspace compression method for a segmentation learning system as described in any paragraph of Example 1.

[0133] Obviously, the embodiments described above are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0134] In the several embodiments provided in this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0135] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0136] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0138] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0139] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0140] The terms "first" and "second" used in the embodiments are merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0141] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for bandwidth-aware neural subspace compression in a segmentation learning system, characterized in that, Include: Obtain the data to be compressed; Based on the data to be compressed, the adaptive rank selection module estimates the tensor singular value spectrum using the randomized subspace method, and performs pruning by combining the energy coverage threshold, bandwidth budget, and rank upper limit to obtain the compressed rank; Based on the data to be compressed, the compression rank, and the error feedback term from the previous iteration, the alternating orthogonal approximation module alternately updates the left and right factor matrices and orthogonalizes the subspaces in the row and column directions to obtain the orthogonal basis and low-rank approximation of the subspaces of the rows and columns; wherein, the orthogonal basis of the subspaces of the rows and columns is the compressed data suitable for transmission. Based on the data to be compressed and the low-rank approximation result, obtain the error feedback term for the current iteration; The data to be compressed includes the tensor to be compressed and / or bandwidth and / or data precision and / or energy coverage threshold; The adaptive rank selection module is suitable for rank selection in single tensors and multi-layer / multi-tensor scenarios. In the case of a single tensor, the constrained optimal rank selection includes: ； ； In the formula, To retain the previous The cumulative energy at each singular value; The compressed rank that is currently retained; It serves as a summation index, without a fixed physical meaning, and is used to traverse a sequence during the summation process; For the first One singular value; This refers to the activation matrix or gradient matrix used in segmentation learning. The total energy of the matrix; The number of rows in the matrix; is the number of columns in the matrix; The cost of one communication for low-rank factorization is: ； ； In the formula, When the compressed rank is The number of bytes required to be transmitted at that time; The number of bytes stored for a single floating-point number, if using but If using or but ; Combination with energy threshold: ； In the formula, To meet the energy coverage threshold The least rank; Continuous relaxation and KKT threshold: ； In the formula, Relaxation variables introduced for continuous relaxation; The maximum bandwidth budget for a single communication; First-order optimality under Lagrange multipliers gives the marginal threshold: ； In the formula, The optimal solution for slack variables; For Lagrange multipliers; The joint assignment operation for multi-level / multi-tensor scenarios is as follows: ； Greedy approximation: ； In the formula, For the first The final rank assigned to the layer; This represents the total number of network layers. For the first Layer rank preservation The cumulative energy at that time; For the first Rank of layers Communication overhead at that time; Budget for total bandwidth in a single multi-layer / multi-tensor communication; For the first Maximum available rank in layer implementation; For the first Layer Energy gain per byte in a singular direction; For the first Layer One singular value; For the first Layer The square of each singular value; Indicates the first The number of rows in the layer matrix; Indicates the first The number of columns in the layer matrix; The expression for the alternating orthogonal approximation is: ； ； ； In the formula, For the first The subspace orthogonal basis of rows obtained by round iteration; To orthogonalize; This refers to the activation matrix or gradient matrix used in segmentation learning. This is the error feedback term from the previous iteration; This forms an orthogonal basis for the subspace of the columns obtained in the previous iteration; For the first The subspace orthogonal basis of the columns obtained in the next iteration; Indicates transpose; For the first The low-rank approximation of the next iteration.

2. The bandwidth-aware neural subspace compression method for a segmentation learning system according to claim 1, characterized in that, The relative residuals and round-by-round complexity of the alternating orthogonal approximation are: ； ； In the formula, express The relative residual of the next iteration; It is the Frobenius norm; This represents the computational complexity of each iteration in the alternating orthogonal iteration; For complex metric level representation; The number of rows in the matrix; is the number of columns in the matrix; The compressed rank that is currently retained.

3. The bandwidth-aware neural subspace compression method for a segmentation learning system according to claim 1, characterized in that, The expression for error feedback is: ； ； In the formula, Indicates the first Error feedback term for the next iteration; The attenuation coefficient; This is the error feedback term from the previous iteration; This refers to the activation matrix or gradient matrix used in segmentation learning. For the first The low-rank approximation in the next iteration; The weighting coefficients for historical residuals; For the first The low-rank approximation of the next iteration.

4. A bandwidth-sensing neural subspace compression device, characterized in that, A bandwidth-aware neural subspace compression method for executing a segmentation learning system according to any one of claims 1 to 3; the bandwidth-aware neural subspace compression device comprises: The data acquisition module is used to acquire the data to be compressed; The rank acquisition module is used to obtain the compressed rank by having the adaptive rank selection module estimate the tensor singular value spectrum based on the randomized subspace method according to the data to be compressed, and then pruning it by combining the energy coverage threshold, bandwidth budget and rank upper limit. The update module is used to alternately update the left and right factor matrices by the alternating orthogonal approximation module according to the data to be compressed, the compression rank, and the error feedback term of the previous iteration, and orthogonalize the subspaces in the row and column directions to obtain the orthogonal basis and low-rank approximation of the subspaces of the rows and columns; wherein, the orthogonal basis of the subspaces of the rows and columns is the compressed data suitable for transmission; The error module is used to obtain the error feedback term for the current iteration based on the data to be compressed and the low-rank approximation result.

5. A bandwidth-sensing neural subspace compression device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement a bandwidth-aware neural subspace compression method for a segmentation learning system as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform a bandwidth-aware neural subspace compression method for a segmentation learning system as described in any one of claims 1 to 3.

7. A segmentation learning system, characterized in that, It includes a client and a server with a communication connection; the client and the server are adapted to use a bandwidth-aware neural subspace compression method for a segmentation learning system as described in any one of claims 1 to 3 when transmitting data.

Images