A neural network pre-training method and system in an edge computing environment
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-04-01
- Publication Date
- 2026-06-09
Smart Images
Figure CN122176447A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a neural network pre-training method and system in an edge computing environment. Background Technology
[0002] With the continuous expansion of deep learning model scale, convolutional neural networks, Transformers, and their variants have been widely applied in fields such as industrial vision analysis, intelligent manufacturing, autonomous driving perception, and multimodal information processing. To obtain model parameters with good generalization ability, pre-training of the model from scratch on large-scale datasets is typically required. However, during pre-training, the first and second moments maintained by adaptive optimizers (such as Adam and AdamW) usually occupy a major portion of GPU memory resources, and their memory overhead may even exceed that of the model parameters themselves. This has become a key bottleneck limiting the efficient training of large-scale models in resource-constrained computing environments.
[0003] The aforementioned problems are particularly pronounced in resource-constrained scenarios such as single-GPU setups, small-scale computing clusters, and edge computing devices. In these environments, limited GPU memory capacity makes it difficult to support the massive optimizer state storage requirements of traditional pre-training processes, directly limiting model size, training stability, and final performance. To address this issue, existing technologies have proposed several memory optimization schemes:
[0004] (1) The first type of scheme adopts an optimizer state offloading mechanism, typically represented by ZeRO-Offload and ZeRO-Infinity techniques in the Microsoft DeepSpeed framework. This type of method offloads the optimizer state, gradients, or some model parameters from GPU memory to host CPU memory or NVMe storage devices, and achieves on-demand loading through pipeline scheduling, thereby reducing GPU memory usage to some extent. However, this scheme introduces frequent host-device data transfer, significantly increasing system communication overhead, and is highly dependent on I / O bandwidth and hardware environment, making it difficult to stably reproduce high-performance training results on general hardware platforms and edge computing environments.
[0005] (2) The second type of approach focuses on low-precision numerical representation, such as quantizing the optimizer state to 8 bits or lower precision to compress memory usage. Although this type of approach has good framework compatibility, in long-cycle, zero-starting pre-training tasks, low-precision quantization is prone to introducing numerical error accumulation, affecting the stability of the optimization trajectory, and thus leading to model convergence failure or significant performance degradation, which is particularly evident in high-dimensional non-convex optimization problems.
[0006] (3) The third type of scheme uses a structured approximation strategy to compress the optimizer state. For example, methods such as Adafactor reduce the optimizer state complexity from O(n) by sharing second-order statistics in the row or column dimensions of the weight matrix. The efficiency is reduced to O(d). However, this type of approximation method inherently sacrifices the adaptability at the parameter level. In pre-training scenarios with uneven gradient distribution or high task complexity, it is often difficult to maintain a convergence speed and final performance comparable to the standard Adam-type optimizer.
[0007] (4) The fourth type of approach reduces the state cache requirement by simplifying the optimizer design, for example by using lightweight or second-moment-free optimizers such as SGD with momentum or Lion. Although this type of method has low memory overhead, its ability to model gradient history information is limited. In complex pre-training tasks such as large-scale language models or visual basic models, it is usually difficult to achieve the optimization effect of Adam-type optimizers.
[0008] Compared to the aforementioned approaches, memory-efficient subspace optimization (MeSO) methods, proposed in recent years, exhibit unique advantages over existing technologies. These methods leverage the low-rank characteristic of neural network gradients during training, projecting high-dimensional gradients onto a low-dimensional subspace. A complete optimizer state update is then performed within this subspace, and the updated result is mapped back to the original parameter space. MeSO methods do not rely on external storage unloading, do not introduce low-precision quantization errors, and do not modify the core update logic of the optimizer. They only require adding forward and backward projection operations to the standard optimization process, significantly reducing memory usage without substantially altering the original training code. With appropriately set projection rank, these methods can achieve high memory compression efficiency while maintaining relatively unchanged model performance.
[0009] While MeSO methods offer significant advantages in reducing optimizer state memory overhead, they suffer from fundamental technical limitations in large-scale pre-training scenarios starting from scratch, severely restricting their applicability and reliability in practical industrial applications, especially in resource-constrained and edge training scenarios. Specifically, to achieve higher memory compression ratios, existing MeSO methods typically project gradients to a low-rank subspace with dimensions much lower than the original parameter space. When the projected rank decreases below a certain critical threshold, the model's final performance experiences a sharp and irreversible decline, and in some extreme configurations, it may even fail to converge to a practically useful solution. This problem is particularly prominent in high-dimensional non-convex pre-training tasks such as large-scale language models and visual foundational models. Further analysis reveals that this performance degradation is not caused by numerical approximation errors or engineering implementation defects, but rather stems from the structural constraints imposed on optimization dynamics by low-rank projection. Within the MeSO framework, each parameter update is confined to a fixed low-dimensional subspace spanned by the current projection matrix, causing the update directions that the model can explore throughout the training process to be consistently limited to the linear manifold of this subspace. When the subspace fails to cover the key directions leading to the global optimum or high-quality local optima in the loss function landscape, the optimization process prematurely loses directional diversity and falls into an optimization trajectory with limited expressive power. This problem is particularly severe in the early stages of pre-training. Since the model starts from random initialization and has not yet formed stable and semantically meaningful feature representations, its gradient signals typically exhibit high noise and directional dispersion. Forcibly compressing the gradient to an extremely low-dimensional subspace at this stage not only filters out a large number of potentially beneficial exploration directions but may also cause initial alignment deviations in the factor space basis vectors, systematically leading the optimization process into suboptimal regions. Once the model falls into a poor local minimum in the early stages, even extending the training time or adjusting the learning rate will make it difficult to effectively escape, resulting in irrecoverable performance loss.
[0010] In summary, existing MeSO methods, while pursuing memory efficiency, sacrifice the degree of freedom in the optimization process, making it difficult to simultaneously achieve the two core objectives of "memory compression" and "pre-training performance" under low-rank conditions. This contradiction is particularly pronounced in training scenarios where memory resources are extremely limited but model capacity requirements are high, especially in model pre-training and deep adaptation tasks in edge computing environments.
[0011] Therefore, how to achieve high-performance, high-efficiency, and high-reliability neural network pre-training under limited hardware resources, so as to provide solid technical support for artificial intelligence applications in cloud computing and edge computing environments, has become a key technical issue of concern to technical personnel. Summary of the Invention
[0012] In view of this, the purpose of this invention is to provide a neural network pre-training method and system in an edge computing environment, which can achieve high-performance, high-efficiency and high-reliability neural network pre-training under limited hardware resources, thereby providing solid technical support for artificial intelligence applications in cloud computing and edge computing environments.
[0013] To achieve the above objectives, this invention provides a neural network pre-training method in an edge computing environment, wherein training each weight matrix in the neural network includes the following steps:
[0014] Step 1: Initialize the weight matrix: initialize the weight matrix. Initialize with two learnable factor matrices and The product of: , , , It is the rank of the parameter and satisfies :
[0015] Step 2, Gradient Calculation and Decomposition: During backpropagation, the loss function is calculated relative to the factor matrix. and gradient and ;
[0016] Step 3, Inner Projection: Calculate the factor matrix respectively. The corresponding right projection matrix sum factor matrix The corresponding left projection matrix Through the right projection matrix gradient Projecting onto a low-rank right subspace yields the compressed low-rank gradient. Meanwhile, through the left projection matrix gradient Projecting onto a low-rank left subspace yields the compressed low-rank gradient. ;
[0017] Step 4: Optimizer State Update: Update the compressed low-rank gradient and The data is input into each of their respective optimizers, and the low-rank state parameters of each optimizer are updated.
[0018] Step 5, Parameter Update and Upprojection: Update the factor matrix based on the low-rank parameter update values returned by the optimizer. and : , , , It is the low-rank parameter update returned by the optimizer, i.e., the updated... , , It's the learning rate. , It is the updated factor matrix , Thus, the updated weight matrix is obtained: , This is the updated weight matrix;
[0019] Step 6: Convergence Judgment: Determine whether the updated weight matrix meets the preset convergence condition. If yes, training terminates; otherwise, return to Step 2 and repeat the iterative training.
[0020] To achieve the above objectives, the present invention also provides a neural network pre-training system in an edge computing environment, comprising:
[0021] A weight matrix initialization device, used to initialize the weight matrix of the neural network. Initialize with two learnable factor matrices and The product of: , , , It is the rank of the parameter and satisfies :
[0022] The gradient calculation and decomposition apparatus is used to calculate the loss function relative to the factor matrix during backpropagation. and gradient and ;
[0023] Internal projection apparatus, used to calculate the factor matrix separately. The corresponding right projection matrix sum factor matrix The corresponding left projection matrix Through the right projection matrix gradient Projecting onto a low-rank right subspace yields the compressed low-rank gradient. Meanwhile, through the left projection matrix gradient Projecting onto a low-rank left subspace yields the compressed low-rank gradient. ;
[0024] The optimizer update mechanism is used to update the compressed low-rank gradients. or The inputs are fed into the corresponding optimizers, and the low-rank state parameters of each optimizer are updated.
[0025] The parameter update and upprojection mechanism is used to update the factor matrix based on the low-rank parameter update values returned by the optimizer. and : , , , It is the low-rank parameter update returned by the optimizer, i.e., the updated... , , It's the learning rate. , It is the updated factor matrix , Thus, the updated weight matrix is obtained: , This is the updated weight matrix;
[0026] The convergence determination device is used to determine whether the updated weight matrix meets the preset convergence condition. If it does, the training terminates; otherwise, the gradient calculation and decomposition device is triggered to repeat the subsequent steps.
[0027] To achieve the above objectives, the present invention also provides a computing device, comprising:
[0028] Memory and processor;
[0029] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the neural network pre-training method in the edge computing environment.
[0030] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the neural network pre-training method in the edge computing environment.
[0031] Compared with existing technologies, the beneficial effects of this invention are as follows: While inheriting the advantages of simplicity, framework compatibility, and memory compression of existing MeSO methods, this invention fundamentally breaks through the rigid constraint of fixed low-rank subspace on parameter update direction. In resource-constrained environments, it can maintain optimizer state memory usage comparable to or even lower than that of MeSO methods, dynamically enhance or expand the diversity of model parameter update directions, support adaptive exploration of richer parameter subspaces during training, and ensure that the model can still stably traverse complex non-convex loss landscapes and converge to high-quality solutions under highly compressed optimization conditions. This invention combines a two-factor network structure with an internal projection mechanism: for the first time, it proposes to decompose the original weight matrix into a two-factor product form and design independent left / right projection strategies for each of the two factors. This internal projection mechanism is the core of this invention, enabling the two low-rank gradient components to carry complementary information. This invention can also decouple the gradient projection rank from the parameter update rank: through the two-factor structure, it successfully achieves an effective increase in the rank of the final parameter update while maintaining the low gradient projection rank (for memory compression). (For enhancing model exploration capabilities), this invention addresses the fundamental shortcomings of existing MeSO methods. It proposes a dedicated optimization scheme for pre-training scenarios, resolving the performance degradation of MeSO methods in pre-training tasks starting from scratch, effectively filling the application gap of this technology in the pre-training field. This invention achieves high performance: with a low memory budget (low projected rank), it can maintain or even surpass the performance of models trained to full rank, effectively solving the performance bottleneck of existing memory compression methods in pre-training scenarios. This invention achieves high memory efficiency: through dual compression using a two-factor structure and subspace optimization, it achieves extreme memory savings for model parameters and optimizer states, enabling larger-scale models to be pre-trained on single cards or small-scale clusters. This invention has strong versatility: it does not depend on a specific network architecture and can be widely applied to the pre-training process of various mainstream models such as Convolutional Neural Networks (CNN) and Transformers. This invention has good orthogonality: it is orthogonal to other model compression techniques such as quantization and pruning, and can be easily combined to obtain synergistic memory and computational benefits. Attached Figure Description
[0032] Figure 1 This is a flowchart illustrating a neural network pre-training method in an edge computing environment, as shown in an exemplary embodiment of the present invention.
[0033] Figure 2 This is a comparison diagram of the technical solutions of the method of the present invention and the existing MeSO method.
[0034] Figure 3 This is a schematic diagram comparing the performance of the method of this invention with the existing MeSO baseline method.
[0035] Figure 4This is a schematic diagram illustrating the structure of a neural network pre-training system in an edge computing environment, as shown in an exemplary embodiment of the present invention.
[0036] Figure 5 This is a comparison chart of the experimental results of image classification on the CIFAR-10 / 100 dataset using the method of this invention and the existing MeSO baseline method based on VGG.
[0037] Figure 6 This is a comparison chart of the experimental results of image classification using the method of this invention and the existing MeSO baseline method, respectively based on ResNet on the Tiny ImageNet dataset and based on ViT on the CIFAR-10 / 100 dataset.
[0038] Figure 7 This is a comparison chart of experimental results using the method of this invention and the existing MeSO baseline method, based on U-ViT, on the CIFAR-10 dataset.
[0039] Figure 8 These are display images generated using the U-ViT model by keeping parameter p constant and changing parameter r, according to the method of this invention.
[0040] Figure 9 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of the present invention. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.
[0042] To address the performance degradation of existing MeSO methods in low-projection-rank and pre-training scenarios, this invention proposes a neural network pre-training method and system for edge computing environments. By reconstructing the network weight structure and combining it with subspace optimization, an effective balance between memory efficiency and model performance is achieved.
[0043] like Figure 1 As shown, this invention relates to a neural network pre-training method in an edge computing environment. The training of each weight matrix in the neural network includes the following steps:
[0044] Step 1: Initialize the weight matrix: initialize the weight matrix. Initialize with two learnable factor matrices and The product of: , express It is a matrix space with all real numbers as elements, containing m rows and n columns. , , It is the rank of the parameter and satisfies :
[0045] Step 2, Gradient Calculation and Decomposition: During backpropagation, the loss function is calculated relative to the factor matrix. and gradient and ;
[0046] Step 3, Inner Projection: Calculate the factor matrices respectively. The corresponding right projection matrix sum factor matrix The corresponding left projection matrix Through the right projection matrix gradient Projecting onto a low-rank right subspace yields the compressed low-rank gradient. Meanwhile, through the left projection matrix gradient Projecting onto a low-rank left subspace yields the compressed low-rank gradient. ;
[0047] Step 4: Optimizer State Update: Update the compressed low-rank gradient and The inputs are fed into their respective optimizers (such as SGD or Adam) to update the low-rank state parameters of each optimizer.
[0048] Since the gradients have been compressed, the memory usage of the optimizer state has also been significantly reduced.
[0049] Step 5, Parameter Update and Upprojection: Update the factor matrix based on the low-rank parameter update values returned by the optimizer. and : , , , It is the low-rank parameter update returned by the optimizer (i.e., the updated parameter value). , ), It's the learning rate. , It is the updated factor matrix , Thus, the updated weight matrix is obtained: , This is the updated weight matrix;
[0050] Step 6: Convergence Judgment: Determine whether the updated weight matrix meets the preset convergence condition. If yes, training terminates; otherwise, return to Step 2 and repeat the iterative training.
[0051] In a preferred embodiment, the parameter rank in step one It can be dynamically adjusted based on the actual memory capacity, computing power utilization, or model training stage of the edge computing node, thereby further adapting to resource-constrained hardware environments.
[0052] This invention utilizes low-rank decomposition of the weight matrix ( The dual mechanism of gradient projection and optimization can achieve dual memory compression of parameters and optimizer state. Specifically, the number of parameters in the weight matrix itself is reduced from O(mn) to O(p(m + n)); the memory usage of the optimizer state is reduced from O(mn) in the traditional method to O(r(m + n)).
[0053] Step three may further include:
[0054] Step 31: Set the update frequency of the subspace, and determine whether the current time is the start time of each round of subspace update cycle. If so, use the MeSO method to calculate the factor matrix respectively. The corresponding right projection matrix sum factor matrix The corresponding left projection matrix , , , If it is the projection rank, then proceed to the next step; otherwise, go to step 33; where the subspace update frequency is used to set the update period of the right projection matrix and the left projection matrix, and the right projection matrix and the left projection matrix remain unchanged in each round of subspace update period;
[0055] In a specific embodiment of the present invention, the right projection matrix and the left projection matrix employ independent computational branches, thereby capturing gradient information of the weight matrix from different dimensions. By projecting the gradient to a low-rank subspace, the present invention enables the rank of the final synthesized weight update ΔW to reach 2r, thereby further improving gradient compression efficiency while maintaining the original expressive power. Compared with the traditional MeSO method, the present invention, through two-factor decomposition and independent projection mechanisms, enables the rank of the update to reach 2r. This significantly improves performance, better preserving gradient information and thus achieving more stable model convergence even under limited memory conditions in edge computing environments.
[0056] Step 32: Update the factor matrix respectively and Each optimizer state: When the optimizer state is a first-order momentum, the update formula for the optimizer state is as follows: When the optimizer state is a second-order momentum, the update formula for the optimizer state is as follows: ,in, Representation of factor matrix or The optimizer state, This is the updated optimizer status. This represents the right projection matrix calculated within the current subspace update cycle. or left projection matrix , This represents the right projection matrix calculated within the previous subspace update cycle. or left projection matrix ;
[0057] The optimizer state should be different for different projection matrices;
[0058] Step 33: Using the right projection matrix and the left projection matrix, respectively, the gradient is... , Project the gradient onto a low-rank right and left subspace, and calculate the compressed low-rank gradient. , : , .
[0059] In one specific embodiment, the present invention can also periodically update the update frequency of the subspace based on the model training progress, gradient stability, or subspace similarity factors, and set the subspace update cycle to several batches or several steps, so that the projection matrix remains constant within each subspace update cycle, reducing high-frequency computation overhead, and recalculating the MeSO projection at the end of the subspace update cycle, balancing efficiency and accuracy.
[0060] In a preferred embodiment, the projection rank The system can be dynamically adjusted based on the actual memory capacity, computing power utilization, or model training stage of the edge computing node, thereby further adapting to resource-constrained hardware environments. This invention can also adjust the projected rank... A real-time dynamic configuration mechanism is adopted, which monitors metrics such as memory usage, computational load, and throughput during training and automatically updates the projected rank based on preset thresholds. The value can be set. For example, when memory usage exceeds a threshold, it can be reduced. To reduce memory pressure; when the model convergence speed is lower than expected, the memory pressure can be increased appropriately. To improve the accuracy of expression. By adjusting the projection rank The dynamic adjustment makes this invention particularly suitable for edge computing scenarios where hardware resources fluctuate significantly.
[0061] Step four may further include:
[0062] Step 41, Parameter update calculation: Factor matrix The corresponding optimizer is based on the low-rank gradient of the input. Right projection matrix and optimizer state Calculate the corresponding parameter update amount and factor matrix. The corresponding optimizer is based on the low-rank gradient of the input. Left projection matrix and optimizer state Calculate the corresponding parameter update amount. The formula for calculating the parameter update amount is as follows: ,in, Represents the compressed low-rank gradient or , Represents the right projection matrix within the current subspace update cycle. or left projection matrix , This represents the function for calculating the optimizer's own parameter update. This represents the parameter update amount calculated by the optimizer, i.e., the updated parameter value. or ;
[0063] Step 42, Optimizer State Optimization Update: Factor Matrix and The corresponding optimizers perform state optimization and update operations respectively: ,in, This indicates that the optimizer itself optimizes and updates its state. This is the updated optimizer status.
[0064] Figure 2 A comparison diagram of the technical solutions of the method of the present invention and the existing MeSO method is shown, wherein (a) represents the key technical points of the existing MeSO method, and (b) represents the key technical points of the method of the present invention (hereinafter referred to as ViLaX). Figure 2 It is clear that existing MeSO methods complete training through low-rank gradients and low-rank updates, while the method of this invention completes training through low-rank gradients and high-rank updates.
[0065] Figure 3 A performance comparison diagram of the method of this invention with existing MeSO baseline methods (such as FLORA, GaLore, and GRASS) is shown. Compared with existing MeSO methods, the method of this invention has the following performance advantages:
[0066] (1) Improvement of update rank: The key advantage of this invention is that, although the gradient projection rank of each factor matrix is 1, the update rank is improved. However, the final synthesized weight matrix Update volume The rank can reach This is far higher than the update rank not exceeding that in traditional MeSO methods. The limitations greatly enhance the model's ability to explore in the parameter space.
[0067] (2) Improved memory efficiency: The memory advantages of this invention are twofold. First, the two-factor structure itself reduces the original weights The number of parameters from Reduce to Secondly, based on this, MeSO is applied to further reduce the memory of the optimizer state from... Reduce to The combination of these two factors results in significant overall memory compression.
[0068] (3) Theoretical compatibility: For networks with low-rank gradient characteristics (such as invertible networks), their corresponding two-factor network structures can also maintain the low-rank nature of the gradient. Therefore, this invention can be directly and seamlessly applied to the MeSO framework, and has a solid theoretical foundation.
[0069] like Figure 4 As shown, the present invention also relates to a neural network pre-training system in an edge computing environment, which may include:
[0070] A weight matrix initialization device, used to initialize the weight matrix of the neural network. Initialize with two learnable factor matrices and The product of: , , , It is the rank of the parameter and satisfies :
[0071] The gradient calculation and decomposition apparatus is used to calculate the loss function relative to the factor matrix during backpropagation. and gradient and ;
[0072] Internal projection apparatus, used to calculate the factor matrix separately. The corresponding right projection matrix sum factor matrix The corresponding left projection matrix Through the right projection matrix gradient Projecting onto a low-rank right subspace yields the compressed low-rank gradient. Meanwhile, through the left projection matrix gradient Projecting onto a low-rank left subspace yields the compressed low-rank gradient. ;
[0073] The optimizer update mechanism is used to update the compressed low-rank gradients. or The inputs are fed into the corresponding optimizers, and the low-rank state parameters of each optimizer are updated.
[0074] The parameter update and upprojection mechanism is used to update the factor matrix based on the low-rank parameter update values returned by the optimizer. and : , , , It is the low-rank parameter update returned by the optimizer, i.e., the updated... , , It's the learning rate. , It is the updated factor matrix , Thus, the updated weight matrix is obtained: , This is the updated weight matrix;
[0075] The convergence determination device is used to determine whether the updated weight matrix meets the preset convergence condition. If it does, the training terminates; otherwise, the gradient calculation and decomposition device is triggered to repeat the subsequent steps.
[0076] To verify the technical effects of this invention, the applicant conducted an experimental comparison between the method of this invention and the existing MeSO method in a training environment with a single general-purpose GPU (NVIDIA GeForce RTX 3090). To simulate the model training process under resource-constrained computing scenarios, the experiment artificially set an upper limit on the video memory budget, limited the storage scale of the optimizer state, and adjusted the size of the projection rank, thereby constructing various low-resource training conditions.
[0077] Given that edge computing devices and the aforementioned restricted GPU training environment share highly similar key constraints (including limitations on video memory capacity and optimizer state storage capacity), the technical effects obtained by this invention under the experimental settings described above can be equivalently characterized in its actual performance in edge computing training scenarios.
[0078] To systematically verify the effectiveness of the proposed method, the experiment further constructed multiple sets of gradient resource constraint configurations by adjusting the projection rank and memory budget, comprehensively evaluating the training stability and model performance of the invention under low memory conditions. This experimental design is not only applicable to data center-level GPU platforms, but can also be equivalently mapped to resource-constrained training scenarios such as edge computing devices or local computing nodes.
[0079] The applicant conducted experimental comparisons of the ViLax method of this invention with existing MeSO baseline methods (such as FLORA, GaLore, GRASS) on multiple vision pre-training tasks, including image classification on CIFAR-10 / 100 and Tiny ImageNet datasets using VGG, ResNet and Vision Transformer (ViT), and image generation using U-ViT. Figure 5 The figure shows a comparison of experimental results for image classification on the CIFAR-10 / 100 dataset using the method of this invention and the existing MeSO baseline method based on VGG. Figure 6 The figure shows a comparison of experimental results for image classification using the method of this invention and existing MeSO baseline methods, respectively based on ResNet on the Tiny ImageNet dataset and based on ViT on the CIFAR-10 / 100 dataset. Figure 7 The figure shows a comparison of experimental results using the method of this invention and existing MeSO baseline methods, based on U-ViT, on the CIFAR-10 dataset. Figure 8 The images shown are generated by the U-ViT model using the method of the present invention, while keeping the parameter p constant and changing the parameter r. In (a), p=180 and r=64, and in (b), p=180 and r=128.
[0080] The experimental results above all demonstrate that, under the same low projective rank setting, the method of this invention significantly outperforms existing MeSO baseline methods (such as GALORE, GRASS, and FLORA) on all tasks, and successfully avoids the model non-convergence or performance collapse problems caused by the low rank of the baseline methods. Meanwhile, the total memory usage of this invention (including parameters, gradients, optimizer states, and projection matrices) is much lower than all the comparative methods, effectively proving that it achieves a better balance between memory efficiency and model performance.
[0081] See Figure 9 , Figure 9 This is a structural block diagram of a computing device 900 illustrated in an exemplary embodiment of this specification. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.
[0082] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.
[0083] In one embodiment of this specification, the aforementioned components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0084] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 900 can also be a mobile or stationary server or cloud server, etc.
[0085] The processor 920 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the neural network pre-training method described above in the edge computing environment.
[0086] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the neural network pre-training method in the edge computing environment described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the neural network pre-training method in the edge computing environment described above.
[0087] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the neural network pre-training method described above in an edge computing environment.
[0088] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the neural network pre-training method in the edge computing environment described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the neural network pre-training method or system in the edge computing environment described above.
[0089] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the neural network pre-training method in the edge computing environment described above.
[0090] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solution of the neural network pre-training method in the edge computing environment described above. Details not described in detail in the computer program's technical solution can be found in the description of the technical solution of the neural network pre-training method or system in the edge computing environment described above.
[0091] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0092] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.
[0093] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.
[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A neural network pre-training method in an edge computing environment, characterized in that, Training each weight matrix in a neural network involves the following steps: Step 1: Initialize the weight matrix: initialize the weight matrix. Initialize with two learnable factor matrices and The product of: , , , It is the rank of the parameter and satisfies : Step 2, Gradient Calculation and Decomposition: During backpropagation, the loss function is calculated relative to the factor matrix. and gradient and ; Step 3, Inner Projection: Calculate the factor matrix respectively. The corresponding right projection matrix sum factor matrix The corresponding left projection matrix Through the right projection matrix gradient Projecting onto a low-rank right subspace yields the compressed low-rank gradient. Meanwhile, through the left projection matrix gradient Projecting onto a low-rank left subspace yields the compressed low-rank gradient. ; Step 4: Optimizer State Update: Update the compressed low-rank gradient and The data is input into each of their respective optimizers, and the low-rank state parameters of each optimizer are updated. Step 5, Parameter Update and Upprojection: Update the factor matrix based on the low-rank parameter update values returned by the optimizer. and : , , , It is the low-rank parameter update returned by the optimizer, i.e., the updated... , , It's the learning rate. , It is the updated factor matrix , Thus, the updated weight matrix is obtained: , This is the updated weight matrix; Step 6: Convergence Judgment: Determine whether the updated weight matrix meets the preset convergence condition. If yes, training terminates; otherwise, return to Step 2 and repeat the iterative training.
2. The method according to claim 1, characterized in that, In step one, the parameter rank Adjustments are made based on the actual memory capacity, computing power utilization, or model training stage of the edge computing node.
3. The method according to claim 1, characterized in that, Step three includes: Step 31: Set the update frequency of the subspace, and determine whether the current time is the start time of each round of subspace update cycle. If so, use the MeSO method to calculate the factor matrix respectively. The corresponding right projection matrix sum factor matrix The corresponding left projection matrix , , , If it is the projected rank, proceed to the next step; otherwise, go to step 33. Step 32: Update the factor matrix respectively and Each optimizer state: When the optimizer state is a first-order momentum, the update formula for the optimizer state is as follows: When the optimizer state is a second-order momentum, the update formula for the optimizer state is as follows: ,in, Representation of factor matrix or The optimizer state, This is the updated optimizer status. This represents the right projection matrix calculated within the current subspace update cycle. or left projection matrix , This represents the right projection matrix calculated within the previous subspace update cycle. or left projection matrix ; Step 33: Using the right projection matrix and the left projection matrix, respectively, the gradient is... , Project the gradient onto a low-rank right and left subspace, and calculate the compressed low-rank gradient. , : , .
4. The method according to claim 3, characterized in that, It also includes: Based on factors such as model training progress, gradient stability, or subspace similarity, the subspace update frequency is periodically updated, and the subspace update period is set to a number of batches or a number of steps.
5. The method according to claim 3, characterized in that, In step three, the right projection matrix and the left projection matrix are calculated using independent branches, and the projection rank... Adjustments are made based on the actual memory capacity, computing power utilization, or model training stage of the edge computing node.
6. The method according to claim 5, characterized in that, It also includes: For projected rank A real-time dynamic configuration mechanism is adopted, which monitors memory usage, computational load, or throughput metrics during training and automatically updates the projected rank based on preset thresholds. The value of .
7. The method according to claim 1, characterized in that, Step four includes: Step 41, Parameter update calculation: Factor matrix The corresponding optimizer is based on the low-rank gradient of the input. Right projection matrix and optimizer state Calculate the corresponding parameter update amount and factor matrix. The corresponding optimizer is based on the low-rank gradient of the input. Left projection matrix and optimizer state Calculate the corresponding parameter update amount. The formula for calculating the parameter update amount is as follows: ,in, Represents the compressed low-rank gradient or , Represents the right projection matrix within the current subspace update cycle. or left projection matrix , This represents the function for calculating the optimizer's own parameter update. This represents the parameter update amount calculated by the optimizer, i.e., the updated parameter value. or ; Step 42, Optimizer State Optimization Update: Factor Matrix and The corresponding optimizers perform state optimization and update operations respectively: ,in, This indicates that the optimizer itself optimizes and updates its state. This is the updated optimizer status.
8. A neural network pre-training system for edge computing environments, characterized in that, Including: A weight matrix initialization device, used to initialize the weight matrix of the neural network. Initialize with two learnable factor matrices and The product of: , , , It is the rank of the parameter and satisfies : The gradient calculation and decomposition apparatus is used to calculate the loss function relative to the factor matrix during backpropagation. and gradient and ; Internal projection apparatus, used to calculate the factor matrix separately. The corresponding right projection matrix sum factor matrix The corresponding left projection matrix Through the right projection matrix gradient Projecting onto a low-rank right subspace yields the compressed low-rank gradient. Meanwhile, through the left projection matrix gradient Projecting onto a low-rank left subspace yields the compressed low-rank gradient. ; The optimizer update mechanism is used to update the compressed low-rank gradients. or The inputs are fed into the corresponding optimizers, and the low-rank state parameters of each optimizer are updated. The parameter update and upprojection mechanism is used to update the factor matrix based on the low-rank parameter update values returned by the optimizer. and : , , , It is the low-rank parameter update returned by the optimizer, i.e., the updated... , , It's the learning rate. , It is the updated factor matrix , Thus, the updated weight matrix is obtained: , This is the updated weight matrix; The convergence determination device is used to determine whether the updated weight matrix meets the preset convergence condition. If it does, the training terminates; otherwise, the gradient calculation and decomposition device is triggered to repeat the subsequent steps.
9. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the neural network pre-training method in the edge computing environment according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, It stores computer-executable instructions, which, when executed by a processor, implement the steps of the neural network pre-training method in the edge computing environment as described in any one of claims 1-7.