A large model distributed fine-tuning federated split learning method and system for a wireless edge network and a medium

Abstract

The embodiment of the application provides a large model distributed fine-tuning federated segmentation learning method and system for a wireless edge network and a medium, and relates to the technical field of computers. The method comprises the following steps: each local device in a plurality of local devices inputs local training data into a device-side model of the local device, obtains an intermediate output result and transmits the intermediate output result to a server; the server processes a plurality of intermediate output results by using a server-side model to obtain a device-side model gradient and a server-side model gradient; the server updates parameters of the server-side model by using the server-side model gradient, and each local device updates parameters of the device-side model of the local device by using the device-side model gradient; at least according to a neuron importance score, the server-side model and the device-side model are pruned, the pruned server-side model and the pruned device-side model are used for continued collaborative training, and a trained large model is obtained, so that the large model fine-tuning in a wireless network environment is efficiently realized.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a large-model distributed fine-tuning federated segmentation learning method, system and medium for wireless edge networks. Background Technology

[0002] Currently, large-scale models are increasingly widely used across various fields. These models typically have hundreds of millions of parameters, requiring massive computing resources and data for training. However, in wireless network environments, the training and application of large-scale models face numerous challenges due to issues such as network bandwidth, transmission latency, and device heterogeneity. First, bandwidth limitations in wireless networks can lead to inefficient data transmission for large models, increasing training time. Second, transmission latency in wireless networks can cause synchronization problems during model training, affecting training results. Furthermore, heterogeneity between different devices can also lead to inconsistencies in model training and performance degradation. Therefore, how to efficiently fine-tune large-scale models in wireless network environments is a pressing issue that needs to be addressed.

[0003] While some methods have attempted to address the fine-tuning of large models in wireless networks, several drawbacks and issues remain. First, traditional centralized machine learning model fine-tuning methods require uploading all data to the cloud or data center for processing, placing immense strain on wireless networks with limited bandwidth and violating increasingly stringent privacy requirements. Second, large models, due to their massive and complex network structures and vast numbers of parameters, are unsuitable for fine-tuning on mobile terminals and local devices with limited computing power. Third, federated learning methods require downloading the entire model or dataset to each node for training, which not only increases the burden of network transmission but also poses a risk of data privacy breaches. Summary of the Invention

[0004] Based on the above-mentioned technical problems, embodiments of the present invention provide a distributed fine-tuning federated segmentation learning method, system, and medium for large models in wireless edge networks, so as to more effectively manage the use of computing and storage resources and achieve the goal of efficiently training high-performance large models in resource-constrained environments.

[0005] This invention provides a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks, the method comprising:

[0006] Each of the multiple local devices inputs local training data into its local device-side model to obtain intermediate output results; each device-side model is a part of a larger model, and the local training data of the multiple local devices includes: text training data, image training data, video training data, and audio training data, and the intermediate output results represent the device-side model's understanding and representation of the currently input local training data;

[0007] The multiple local devices transmit their respective intermediate output results to the server;

[0008] The server uses the server-side model to process multiple intermediate output results to obtain the device-side model gradient and the server-side model gradient. The server-side model gradient is used to update the parameters of the server-side model. The server-side model is the part of the large model other than the device-side model.

[0009] The server sends the device-side model gradient as a shared device-side model gradient among the multiple local devices to each of the multiple local devices.

[0010] Each of the plurality of local devices updates the parameters of its local device-side model using the device-side model gradient;

[0011] Based at least on the neuron importance score, the server prunes the server-side model to obtain a pruned server-side model, and each of the plurality of local devices prunes its local device-side model to obtain a pruned device-side model.

[0012] The multiple local devices and the server continue to train collaboratively based on the pruned server-side model and the pruned device-side model to obtain a fully trained large model.

[0013] The trained large model is used to perform multimodal generation tasks, including text generation, image generation, video generation, and audio generation.

[0014] A second aspect of this invention provides a large-model distributed fine-tuning federated segmentation learning system for wireless edge networks, the system comprising:

[0015] The data processing module is deployed on each of the multiple local devices to input local training data into the local device-side model and obtain intermediate output results. Each device-side model is a part of a larger model. The local training data of the multiple local devices includes: text training data, image training data, video training data, and audio training data. The intermediate output results represent the device-side model's understanding and representation of the currently input local training data.

[0016] A data transmission module, deployed on the multiple local devices, is used to transmit the intermediate output results of each device to the server.

[0017] The result processing module, deployed on the server, is used to process multiple intermediate output results using the server-side model to obtain the device-side model gradient and the server-side model gradient, and to update the parameters of the server-side model using the server-side model gradient. The server-side model is the remaining part of the large model excluding the device-side model.

[0018] A gradient delivery module, deployed on the server, is used to send the device-side model gradient as a shared device-side model gradient among the multiple local devices to each of the multiple local devices.

[0019] A parameter update module is deployed on each of the plurality of local devices to update the parameters of the local device-side model using the device-side model gradient;

[0020] A pruning module, deployed on the server and each of the plurality of local devices, is used to prune the server-side model at least based on neuron importance scores to obtain a pruned server-side model, and to prune the local device-side model to obtain a pruned device-side model.

[0021] The training module, deployed on the multiple local devices and the server, is used to continue collaborative training based on the pruned server-side model and the pruned device-side model to obtain a fully trained large model.

[0022] The trained large model is used to perform multimodal generation tasks, including text generation, image generation, video generation, and audio generation.

[0023] A third aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the large-model distributed fine-tuning federated segmentation learning method for wireless edge networks according to the first aspect of the present invention.

[0024] The distributed fine-tuning federated segmentation learning method for large models in wireless edge networks, as described in this embodiment, uses federated segmentation learning to divide a large model into device-side models and server-side models suitable for processing by various devices (such as local devices and servers). Each device is only responsible for training a portion of the large model, reducing the computational burden on individual devices. Furthermore, since this embodiment adopts a distributed model fine-tuning framework, it allows devices to fine-tune the model on local data and only transmits the intermediate output results of the model to the server, rather than the original data. This not only reduces communication volume but also achieves large model fine-tuning while protecting user privacy. In addition, this embodiment utilizes the computing power of local devices and the centralized processing power of servers to collaboratively train the model, which also improves the resource utilization of the network, thereby enabling efficient large model fine-tuning in a wireless network environment. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks, as shown in an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram illustrating a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks, as shown in an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of the structure of a multimodal large model provided in an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram illustrating an example of applying federated segmentation learning to distributed fine-tuning of a multimodal large model, as provided in an embodiment of the present invention.

[0030] Figure 5 This is a block diagram of a large-model distributed fine-tuning federated segmentation learning system for wireless edge networks provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] As mentioned earlier, in traditional machine learning, model training typically requires transmitting large amounts of data to a central server. This not only poses privacy risks but also places high demands on network bandwidth and storage resources. While federated learning allows data training on local devices, with only model parameters uploaded to the server for aggregation, even the transmission of model parameters can lead to significant network load and latency for large models. Therefore, how to efficiently fine-tune large models in a wireless network environment is a pressing issue that needs to be addressed.

[0033] While some current federated learning methods can perform distributed fine-tuning while protecting data privacy, they require each device to have sufficient computing and storage capabilities to train the entire model. This often ignores the bandwidth limitations and transmission latency of wireless network environments, leading to low training efficiency. For fine-tuning large models, how to reduce communication overhead and computing resource requirements while ensuring performance is a critical problem that urgently needs to be solved.

[0034] Federated learning-based distributed training methods typically assume that all clients can independently train the entire machine learning model. Each device trains the model locally using its own data, only uploading model updates (such as gradients) to a central server for aggregation, thus achieving distributed training and data privacy protection. However, even so, for large models, the number of parameters updated each time remains enormous, potentially exceeding the capacity of wireless networks. Furthermore, the dedicated memory of local devices may be insufficient to store all the weights of a large model during training. This means that the device's memory availability will limit the size of the neural network model during training, potentially compromising information integrity. To address memory constraints, the model size is often reduced (potentially resulting in a loss of representational power), and the input data is often dimensionality-reduced, placing higher demands on developing more memory-efficient training algorithms. However, most techniques for reducing deployment memory are not suitable for reducing training memory, as they typically require training a full-size model first and then compressing it. Moreover, techniques for reducing training memory used in data centers / supercomputers are not suitable for federated learning. Therefore, addressing the computational and storage challenges faced by local devices training large models in federated learning is an important and urgent task.

[0035] The existence of the above problems is mainly due to the following two reasons: (1) Device hardware limitations: Federated learning usually requires each device or server to have a complete copy of the model, while IoT devices in wireless networks usually only have limited computing power and storage space. Large models need to process massive amounts of data, which places very high demands on hardware, including high-performance GPUs, large-capacity hard drives and memory. IoT devices often cannot meet these requirements and cannot run and train large models directly on the local machine, resulting in slow training or failure to complete the training process. (2) Wireless communication and privacy issues: The bandwidth and stability of wireless communication networks are lower than those of wired networks. Distributed training of large models requires a large amount of data transmission and synchronization. Under limited wireless bandwidth conditions, the communication cost of transmitting model parameters is extremely high. The training data collected by local devices may contain users' privacy information, so it is also necessary to consider how to conduct effective model training while ensuring that the data does not leave the local machine.

[0036] Therefore, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, embodiments of the present invention propose a distributed fine-tuning federated segmentation learning method for large models in wireless edge networks. In this method, each device is only responsible for training a part of the large model, which reduces the computational burden of individual devices and allows data to be partially processed on local devices (such as smartphones, sensors, etc.) before the intermediate processing results are transmitted to the server for further processing, thereby reducing the amount of data transmission. Since the original data does not need to leave the local device, it helps to protect the privacy of the data and improve data security, thereby enabling efficient fine-tuning of large models in wireless network environments.

[0037] Reference Figure 1 , Figure 1 This is a flowchart illustrating a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks, as shown in an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0038] Step S11: Each of the multiple local devices inputs its local training data into its local device-side model to obtain intermediate output results; each device-side model is a part of a larger model, and the local training data of the multiple local devices includes: text training data, image training data, video training data, and audio training data, and the intermediate output results represent the device-side model's understanding and representation of the currently input local training data.

[0039] In this embodiment, for the fine-tuning of training a large model, the large model can be segmented using federated segmentation learning into a device-side model and a server-side model. The device-side model is a part of the large model, and the server-side model is the remaining part of the large model excluding the device-side model. For example, the segmented device-side model represents the network structure of layers 1 to n of the large model, and the server-side model represents the network structure of layers n+1 to N of the large model. The segmented device-side model is deployed on one or more local devices, each containing the same device-side model. Each local device has a certain computing power and can be an electronic device (such as a mobile phone), a sensor, etc. The local devices are responsible for performing the forward propagation and backward propagation of the device-side model.

[0040] In this embodiment, "large model" refers to a machine learning or deep learning model with a huge number of parameters, such as a large-scale pre-trained model under the Transformer architecture (e.g., BERT, GPT-3). Based on deep learning principles, it uses a large amount of data and computing resources to train a neural network model with a large number of parameters. Due to their large parameter scale, these models often have high expressive power and generalization ability, but they also require a large amount of computing resources, storage space, and data for training and deployment.

[0041] Federated learning, as described in this embodiment, is a distributed machine learning technique. Its core idea is to perform distributed model training across multiple devices with local data, allowing multiple devices or nodes (such as smartphones and IoT devices) to collaboratively train a shared model without uploading the original data to a central server or cloud. Each device updates its model (i.e., calculates gradients) on its local data and then sends only the updates (not the original data) to the central server. These local updates are aggregated at the central server to iteratively improve the global model. During model training, the local training data does not leave the local device; instead, a global model based on virtual fused data is built by exchanging model parameters or intermediate results. This leverages large amounts of distributed data to improve model performance while effectively protecting user privacy. Similarly, segmentation learning is a distributed training method for deep learning models. Its core idea is to divide the network structure into two or more parts, with each device retaining a portion of the sub-network structure. During training, different devices only perform forward or backward computation on their local network structure and pass the results to the next device. This reduces the computational and storage requirements of individual devices, and each participant is unaware of the complete network structure, helping to protect data privacy.

[0042] In this embodiment, each of the multiple local devices can input its local training data into its local device-side model, perform forward propagation calculations, and obtain an intermediate output result from the local device-side model. This intermediate output result represents the device-side model's understanding and representation of the currently input local training data. The local training data of the multiple local devices in this embodiment includes: text training data, image training data, video training data, and audio training data, used for training and fine-tuning a large model.

[0043] In one example, the device-side model can be represented as Where u represents the parameter dimension of the device-side model, and the set of local training data in the local devices participating in the training is denoted as C = {1, 2, ..., C}, where C represents the total number of devices, and each device c has its own unique dataset D of local training data. c .

[0044] In this embodiment, the local device processes only its own local training data without relying on training data from other local devices, thereby protecting the privacy of local data from being leaked.

[0045] Step S12: The multiple local devices transmit their respective intermediate output results to the server.

[0046] In this embodiment, after multiple local devices obtain the intermediate output results of their respective local device-side models, they transmit their intermediate output results to the server through a wireless edge network.

[0047] Step S13: The server processes multiple intermediate output results using the server-side model to obtain the device-side model gradient and the server-side model gradient, and updates the parameters of the server-side model using the server-side model gradient. The server-side model is the remaining part of the large model excluding the device-side model.

[0048] In this embodiment, the server can be an edge server, connected to the base station, possessing powerful computing capabilities, and responsible for training and fine-tuning the server-side model it deploys. After receiving multiple intermediate output results transmitted from multiple local devices, the server can continue forward propagation calculations using the server-side model, further processing these intermediate output results to obtain the gradients of the device-side model and the server-side model. Then, the server uses the obtained server-side model gradients to update the parameters of the server-side model. Through the backpropagation algorithm, the server can precisely adjust the parameters of the server-side model to optimize its performance.

[0049] In one example, the server-side model can be represented as Here, z represents the parameter dimension of the server-side model. In addition, the server is responsible for collecting critical network information, such as channel state information and device computing power, in order to implement efficient resource management strategies.

[0050] Step S14: The server sends the device-side model gradient as a shared device-side model gradient among the multiple local devices to each of the multiple local devices.

[0051] In this embodiment, after obtaining the device-side model gradient, the server sends this gradient as a shared device-side model gradient to each local device via the wireless edge network. In other words, in this embodiment, the server sends the same device-side model gradient to each of all local devices to ensure that the device-side model on the local device can be synchronized with the server-side model on the server.

[0052] Step S15: Each of the plurality of local devices updates the parameters of its local device-side model using the device-side model gradient.

[0053] In this embodiment, after each local device receives the device-side model gradient transmitted by the server, it can use the backpropagation algorithm to update the parameters of the local device-side model, adjust its model parameters to a state that is more compatible with the server-side model, thereby improving the performance of the entire large model.

[0054] Step S16: Based at least on the neuron importance score, the server prunes the server-side model to obtain a pruned server-side model, and each of the plurality of local devices prunes its local device-side model to obtain a pruned device-side model.

[0055] In this embodiment, after one round of parameter updates for the device-side model and the server-side model, i.e. after one round of collaborative training of the device-side model and the server-side model is completed, the neuron importance scores corresponding to the device-side model and the server-side model are obtained respectively. Then, a dynamic pruning strategy can be used to guide the pruning process of the device-side model and the server-side model, based at least on the neuron importance scores: the server prunes the server-side model based at least on its corresponding neuron importance scores to obtain the pruned server-side model; and each local device prunes its local device-side model based at least on its corresponding neuron importance scores to obtain the pruned device-side model.

[0056] This embodiment relates to a dynamic pruning technique for deep neural networks, which aims to reduce the computation and storage requirements during model fine-tuning, and is particularly suitable for scenarios where client resources are limited.

[0057] Step S17: The multiple local devices and the server continue to train collaboratively based on the pruned server-side model and the pruned device-side model to obtain a fully trained large model.

[0058] In this embodiment, after obtaining the pruned server-side model and the pruned device-side model, multiple local devices and servers can continue to perform collaborative training through the aforementioned steps S11 to S16 based on their respective deployed pruned server-side models and pruned device-side models until the model converges or reaches a preset accuracy, thereby obtaining the trained device-side model and the trained server-side model, and thus obtaining the trained large model.

[0059] In this embodiment, the trained large model can be used to perform multimodal generation tasks, including text generation, image generation, video generation, and audio generation.

[0060] This embodiment utilizes distributed fine-tuning to make targeted adjustments to a pre-trained large model using local training datasets distributed across different geographical locations or computing nodes. This means fine-tuning the large model in a distributed environment, allowing for parallel training across multiple computing nodes and thus accelerating the training process. Distributed fine-tuning is extremely useful for handling large-scale datasets because it fully utilizes computing resources and improves training efficiency. Furthermore, distributed fine-tuning means that each local device optimizes the parameters of its device-side model based on its local training data, ultimately integrating the updates from all local devices to better adapt the large model to the overall data distribution while avoiding the privacy risks associated with directly transmitting raw data.

[0061] Specifically, the distributed fine-tuning process of the large model in this embodiment involves the following key steps: First, a large model pre-trained on a large-scale dataset is selected as the base model; then, according to the task requirements of the multimodal generation task, the large model is split so that each local device and server can process a part of the network structure (i.e., the device-side model and the server-side model). During training, each local device performs forward and backward computation locally and passes the intermediate results to other devices; finally, the model is aggregated through the server to obtain a global large model with optimized performance on the multimodal generation task.

[0062] In this embodiment, federated segmentation learning is used to divide the large model into device-side models and server-side models suitable for processing by each device. Each device is only responsible for training a part of the large model, which reduces the computational burden of a single device. Furthermore, since this embodiment adopts a distributed model fine-tuning framework, it allows devices to fine-tune the model on local data and only transmits the intermediate output results of the model to the server, rather than the original data. This not only reduces the amount of communication but also protects user privacy while enabling fine-tuning of the large model. In addition, this embodiment utilizes the computing power of local devices and the centralized processing power of the server to train the model collaboratively, which also improves the utilization rate of network resources, thereby enabling efficient fine-tuning of large models in a wireless network environment.

[0063] In conjunction with the above embodiments, in one implementation, the present invention also provides a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks. In this method, the large model includes multiple neurons, and the parameters of the server-side model and the device-side model include at least: neuron weights, where the neuron importance score is used for pruning neurons, and the neuron importance score reflects the importance of the neuron to the overall network output.

[0064] After updating the parameters of both the device-side and server-side models, a neuron importance score can be calculated for each neuron in the large model. Then, based on the neuron importance score, it's determined whether to prune the corresponding neuron. If a neuron is pruned, the edges corresponding to that neuron will also be pruned. In one example, this could involve pruning all neurons with importance scores below a first threshold (which can be freely set), or pruning one or more neurons with the lowest importance scores (the number can be freely set); there are no restrictions on this.

[0065] Specifically, regarding neuron importance scoring: for the j-th neuron in the i-th layer, its neuron importance score s i,j It is determined based on the absolute values ​​of the weights of all neurons in the (i+1)th layer connected to it, and the neuron importance scores of all neurons in the (i+1)th layer connected to it.

[0066] In one optional example, the neuron importance score s of the j-th neuron in the i-th layer. i,j It is the sum of the absolute weights of all neurons in the next layer (layer i+1) connected to it, and the neuron importance scores of these neurons (i.e., all neurons in layer i+1 connected to it). Thus, this recursive calculation quantifies the total influence of each neuron on the final output. Importance propagates backward from the output layer, reflecting the importance of each neuron to the overall network output. The neuron importance score s of the j-th neuron in layer i is... i,jIt can be shown in the following formula (1):

[0067]

[0068] in, S is the weight connecting the j-th neuron in the i-th layer to the k-th neuron in the (i+1)-th layer. i+1,k It is the neuron importance score that connects the j-th neuron in the i-th layer to the k-th neuron in the (i+1)-th layer.

[0069] In conjunction with the above embodiments, in one implementation, the present invention also provides a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks. In this method, step S16 may specifically include step S21:

[0070] Step S21: Based on the neuron importance score and weight importance score, the server prunes the server-side model to obtain a pruned server-side model, and each of the plurality of local devices prunes its local device-side model to obtain a pruned device-side model.

[0071] In this embodiment, the importance score of the edge connecting two neurons can be determined based on the neuron importance score. Therefore, the pruning process of the device-side model and the server-side model is guided by the neuron importance score and the weight importance score.

[0072] The server prunes the server-side model based on the weight importance score corresponding to the server-side model and the neuron importance score corresponding to the neurons in the server-side model, resulting in a pruned server-side model; and each local device prunes its local device-side model based on the weight importance score corresponding to the device-side model and the neuron importance score corresponding to the neurons in the device-side model, resulting in a pruned device-side model.

[0073] The weight importance score is used to prune the edges connecting two neurons; the weight importance score connecting the j-th neuron in the i-th layer and the k-th neuron in the (i+1)-th layer. It is determined based on the absolute value of the weights connecting the j-th neuron in layer i to the k-th neuron in layer i+1, and the neuron importance score of the k-th neuron in layer i+1.

[0074] In an optional example, the weight importance score connecting the j-th neuron in layer i to the k-th neuron in layer i+1. This is the product of the absolute value of the weight connecting the j-th neuron in layer i to the k-th neuron in layer (i+1), and the neuron importance score of the k-th neuron in layer (i+1). This weight importance score considers not only the magnitude of the weight but also its potential impact on subsequent neurons and the network's overall response. It directly links the importance of the weight to its absolute value and its correlation with the neurons it affects, comprehensively measuring the weight's contribution to the overall network performance and providing a more comprehensive assessment of weight importance.

[0075] Therefore, the weight importance of connecting the j-th neuron in the i-th layer to the k-th neuron in the (i+1)-th layer. The weight is obtained by multiplying the absolute value of the weight by the importance score of the target neuron, and its calculation method is shown in the following formula (2):

[0076]

[0077] in, S is the weight connecting the j-th neuron in the i-th layer to the k-th neuron in the (i+1)-th layer. i+1，k It is the neuron importance score that connects the j-th neuron in the i-th layer to the k-th neuron in the (i+1)-th layer.

[0078] In conjunction with the above embodiments, in one implementation, the present invention also provides a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks. In this method, during each round of collaborative training, the pruning process of the device-side model and the server-side model is guided by the neuron importance scores propagating from the output layer to the input layer of the large model. As collaborative training continues, the number of pruned neurons in each round of collaborative training gradually decreases.

[0079] This embodiment employs a dynamic pruning strategy, guiding the pruning process based on neuron importance scores propagating from the output layer to the input layer, or neuron importance scores and weight importance scores. In the early stages of model fine-tuning, redundant connections in the network are pruned more aggressively. As the network becomes sparser, the number of pruning operations per round is gradually reduced. This method improves efficiency without compromising performance, meeting the growing demand for lightweight models and making it suitable for deployment in resource-constrained environments. Furthermore, the core of this pruning method lies in assessing the importance of neurons based on their contribution to the network's final output or response using both neuron and weight importance scores. This contrasts sharply with simple pruning methods that rely solely on the absolute value of weights (pruning when the weight falls below a weight threshold).

[0080] In another implementation, model pruning can be performed after the last round of co-training based on neuron importance scores, or neuron importance scores and weight importance scores; alternatively, pruning rounds can be set (e.g., every 3 rounds, every 5 rounds, etc.), and model pruning can be performed based on neuron importance scores, or neuron importance scores and weight importance scores, when the pruning round is reached.

[0081] In conjunction with the above embodiments, in one implementation, the present invention also provides a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks. In this method, the local training data includes multiple data samples, each data sample including input data and corresponding labels; step S12 above may specifically include step S31, and the step S13 above, "the server uses the server-side model to process multiple intermediate output results to obtain the device-side model gradient and the server-side model gradient," may specifically include steps S32 to S35.

[0082] Step S31: The multiple local devices transmit their respective intermediate output results and the tags corresponding to the intermediate output results to the server via a wireless channel.

[0083] In this embodiment, the local training data may include multiple data samples, such as those containing D. c There are 1 data sample, each data sample includes input data x. c,C and the corresponding label y c,C Composition. The forward propagation of the device-side model is computed independently on the local training data dataset of the corresponding local device, and includes the following aspects:

[0084] Data preparation: For each local device participating in the fine-tuning of the large model, a mini-batch of data is randomly drawn from its local training dataset. This mini-batch typically contains b data samples. The input data and corresponding labels can then be represented as follows: and Where t represents the current training round and d is the feature dimension of the input data.

[0085] Model loading: Each local device has a local device-side model w c (t-1), this device-side model is updated and saved at the end of the previous training round (t-1).

[0086] Forward propagation: During forward propagation, the local device transmits small batches of data X. c (t) Input to the device-side model w cIn (t-1), the device-side model performs a series of calculations according to its defined neural network structure, generating intermediate outputs (also called activation values) at specific model segmentation layers. These intermediate outputs represent the device-side model's understanding and representation of the data features under the current input, and they will be transmitted to the server for further processing. The generation process of intermediate results is represented as follows:

[0087] Parallel processing: During the forward propagation phase of the device-side model, all local devices participating in the fine-tuning of the large model will execute the above steps in parallel. This means that different local devices can perform forward propagation simultaneously on different local training datasets, thereby accelerating the fine-tuning process of the entire large model.

[0088] After the local device completes the forward propagation and generates intermediate output results, it can transmit the intermediate output results and their corresponding tags to the server via a wireless channel. This process ensures data privacy because the original data does not leave the local device; at the same time, transmitting intermediate results instead of the original data reduces network transmission load.

[0089] Step S32: The server uses the server-side model to predict the multiple intermediate output results to obtain the prediction result.

[0090] In this embodiment, after the server receives the intermediate outputs from all local devices participating in training and fine-tuning, the server can perform forward propagation, using the intermediate outputs to make predictions. Multiple intermediate outputs are input into the server-side model until the server-side model outputs the prediction result. This prediction result represents the server's prediction of the input data based on the intermediate outputs from all local devices. The dimensions and format of the prediction result depend on the specific design of the server-side model and the task requirements.

[0091] Step S33: The server calculates the loss corresponding to each local device based on the prediction result and the tag transmitted by each local device.

[0092] In this embodiment, after obtaining the prediction results, the server can perform subsequent processing and analysis: the server can calculate the loss corresponding to each local device in parallel based on the prediction results and the tags transmitted by each local device, using the loss function corresponding to each local device.

[0093] Step S34: The server averages the multiple losses corresponding to each of the multiple local devices and then calculates the partial derivatives to obtain the aggregated gradient.

[0094] In this embodiment, after obtaining the losses corresponding to multiple local devices, the server can average these losses and then calculate their partial derivatives to obtain the aggregated gradient. The averaging process in this embodiment can be either direct averaging or weighted averaging, depending on the requirements or application scenario. The main purpose of calculating the aggregated gradient in this embodiment is to reduce the communication and computational overhead during subsequent backpropagation by local devices, thereby improving the system's resource utilization efficiency.

[0095] In one embodiment, the server calculates aggregated gradients, which are the partial derivatives of the model parameters with respect to the loss function. These aggregated gradients reflect the direction of the model's adjustments during training. To fully utilize loss functions from multiple local devices, the server first weights and aggregates these loss functions from different local devices, then calculates the partial derivatives to form a comprehensive gradient update.

[0096] Specifically, the server first computes the loss function of all local devices participating in the fine-tuning of the large model in parallel, and then calculates their average. The specific loss function aggregation strategy may vary depending on the application scenario. Common strategies include average aggregation (averaging all loss functions) or weighted average aggregation (assigning different weights based on the amount of data from each edge device; for example, if the first local device used 2 data samples and the second local device used 3 data samples, then the weight of the first local device is 2 / 5 and the weight of the second local device is 3 / 5). Then, the server calculates the partial derivative of the averaged loss function to obtain the aggregated gradient.

[0097] Step S35: The server obtains the device-side model gradient and the server-side model gradient based on the aggregated gradient.

[0098] In this embodiment, after obtaining the aggregated gradient, the server can determine the device-side model gradient G based on the aggregated gradient. c (t) and the gradient G of the server-side model s (t), and then the server transmits the device-side model gradient G via a wireless channel. c (t) is distributed to all local devices, with the aim of enabling local devices to contribute knowledge features to each other.

[0099] Subsequently, the backpropagation of the server-side model is based on the gradient G of the server-side model. s (t) is used to update the parameters of the server-side model so as to better fit the training data in the next iteration. The specific process is shown in the following formula (3):

[0100] W s (t)=W s (t-1)-η s GS (t) (3)

[0101] Where, η s W is the learning rate for server-side model updates. s (t) represents the server-side model updated in this training round (round t), W s (t-1) represents the server-side model at the end of the previous training round (round t-1), G s (t) represents the gradient of the server-side model.

[0102] Sharing gradients between local devices not only prevents overfitting but also eliminates the need for model swapping as required in federated learning. This significantly reduces the number of backpropagations needed, optimizes the training process, and improves training efficiency and resource utilization. Specifically, the backpropagation of the device-side model is based on the shared gradient G received by the local device. c (t) is used to update the parameters of the device-side model, and the specific process is shown in the following formula (4):

[0103] W c (t)=W c (t-1)-η c G c (t) (4)

[0104] Where, η c W is the learning rate for device-side model updates. c (t) represents the updated device-side model in this training round (round t), W c (t-1) represents the device-side model at the end of the previous training round (round t-1), G c (t) represents the gradient of the device-side model.

[0105] In conjunction with the above embodiments, the present invention also provides a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks. In this method, step S32 may specifically include steps S41 and S42:

[0106] Step S41: The server concatenates the multiple intermediate output results based on the number of intermediate output results transmitted by each local device and the activation dimension of the server-side model to obtain an intermediate concatenation result.

[0107] In this embodiment, after receiving the intermediate output results from all local devices, the server concatenates these intermediate output results. Since each local device may have a different amount of data and different model output dimensions, the concatenation operation needs to ensure data consistency and alignment. Based on the number of intermediate output results transmitted by each local device and the activation dimension of the server-side model, the server concatenates multiple intermediate output results to obtain an intermediate concatenated result.

[0108] For example, the intermediate splicing result can be represented as follows: Where t represents the current training round, C represents the number of local devices participating in the training, b is the mini-batch data size of each local device (i.e., the number of intermediate output results transmitted by each device), and q is the activation dimension of the server-side model after the segmentation layer (i.e., the number of features of each intermediate output result).

[0109] Step S42: The server uses the server-side model to predict the intermediate splicing result and obtain the prediction result.

[0110] In this embodiment, after obtaining the intermediate stitching result, the server can use the server-side model to predict the intermediate stitching result R(t) to obtain the prediction result. The process of determining the prediction result based on the intermediate stitching result can be represented as follows: in, R(t) represents the predicted result, and W represents the intermediate splicing result. s (t-1) represents the server-side model at the end of the previous training round (round t-1).

[0111] In conjunction with the above embodiments, in one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram illustrating a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks, as shown in an embodiment of the present invention. Figure 2 In this model, the large model is divided into a device-side model and a server-side model. The device-side model is deployed on multiple devices, and the server-side model is deployed on an edge server connected to the base station. A total of C devices are deployed on the edge server, and each device has a local dataset.

[0112] The process of distributed fine-tuning of large models includes the following steps:

[0113] Step 1, Forward Propagation of Device-Side Model: On the local device, perform forward propagation calculations using local data and the device-side model to generate intermediate results.

[0114] Step 2, Uplink Transmission of Intermediate Results: The local device transmits the calculated intermediate results to the server.

[0115] Step 3, Forward Propagation of Server-Side Model: The server receives the intermediate results and continues the forward propagation calculation using the server-side model.

[0116] Step 4: Calculate the aggregated gradient: The server aggregates the loss functions of different devices and then calculates the aggregated gradient.

[0117] Step 5, Backpropagation of the server-side model: The server uses the aggregated gradients to update the parameters of the server-side model.

[0118] Step 6, Aggregated Gradient Downstream Broadcast: The server broadcasts the aggregated device-side model gradients to all local devices.

[0119] Step 7, Device-side model backpropagation: After receiving the aggregated gradient broadcast by the server, the local device uses the gradient to update the local model parameters.

[0120] Step 8, Large Model Structured Pruning: Model pruning is guided by the importance scores of neurons and weights from the output layer to the input layer.

[0121] In conjunction with the above embodiments, the present invention also provides a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks. In this method, in addition to the steps described above, steps S51 and S52 may also be included:

[0122] Step S51: The server divides the large model according to its functional structure to obtain the device-side model and the server-side model.

[0123] In this embodiment, the server can divide the large model according to its functional structure to obtain a device-side model and a server-side model. The server-side model is the last part of the large model and is used to produce the final prediction result.

[0124] Step S52: The server deploys the device-side model to multiple local devices and deploys the server-side model to the server.

[0125] In this embodiment, after the server obtains a device-side model and a server-side model, it can deploy the device-side model to multiple local devices and deploy the server-side model to the server, so that each local device deploys a device-side model, representing the same part of the large model.

[0126] In conjunction with the above embodiments, the present invention also provides a large-model distributed fine-tuning federated segmentation learning method for wireless edge networks. In this method, the large model is a multimodal large model, which includes: a modality encoder, an input projector, a pedestal large model, an output projector, and a modality generator. For example, ... Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a multimodal large model provided in an embodiment of the present invention. The multimodal large model includes multiple potential segmentation layers, which can be segmented according to requirements.

[0127] In this embodiment, the model is first structurally pruned based on the neuron importance scores of the multimodal large model, and then the multimodal large model is segmented according to its functional structure. The modality encoder is considered as the input layer processing specific modal data, and therefore it is deployed as a device-side model on the local device. The input projector, pedestal large model, output projector, and modality generator are responsible for higher-level feature extraction and generation than processing specific modal data, and are therefore deployed as server-side models on the edge server. Figure 4 As shown, Figure 4 This is a schematic diagram illustrating an example of applying federated segmentation learning to distributed fine-tuning of a multimodal large model, as provided in an embodiment of the present invention. Figure 4 The strategy involves dividing the large multimodal model into two parts and deploying them to an edge server and a local device for fine-tuning: the modal encoder is deployed to the local device, while the remaining modules of the large model (including the input projector, the base model, the output projector, and the modal generator) are deployed to the edge server.

[0128] On the local device, a modal encoder is deployed. This encoder processes the specific modal data collected on the local device and transforms it into feature representations suitable for input into the large base model. This specific modal data includes at least image data, text data, speech data, and audio data. After completing the forward propagation, the local device sends the resulting feature representations to the edge server for further processing.

[0129] On the edge server, an input projector, a pedestal large model, an output projector, and a modality generator are deployed. Upon receiving feature representations from the local device, these modules work together to perform backpropagation and update model parameters. During backpropagation, the edge server first calculates the device-side model gradient and the server-side model gradient based on the loss function, and then propagates the aggregated device-side model gradient to the local device. The local device then uses the received gradients to update its local modality encoder parameters, and the edge server also uses the aggregated server-side model gradient to update the parameters of the input projector, pedestal large model, output projector, and modality generator.

[0130] Finally, an iterative training approach is used to fine-tune the large multimodal model: in each iteration cycle, the local device processes new data and sends the resulting feature representations to the edge server; the edge server uses these feature representations for backpropagation and parameter updates, and sends the aggregated gradient information back to the local device. In this way, the local device can achieve distributed fine-tuning and optimization of the large multimodal model without sharing the original data.

[0131] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0132] Based on the same inventive concept, one embodiment of the present invention provides a large-model distributed fine-tuning federated segmentation learning system 500 for wireless edge networks. (Reference) Figure 5 , Figure 5 This is a block diagram of a large-model distributed fine-tuning federated segmentation learning system for wireless edge networks provided in an embodiment of the present invention. Figure 5 As shown, the system 500 includes:

[0133] The data processing module 501 is deployed on each of the multiple local devices and is used to input local training data into the local device-side model to obtain intermediate output results. Each device-side model is a part of a larger model. The local training data of the multiple local devices includes: text training data, image training data, video training data, and audio training data. The intermediate output results represent the device-side model's understanding and representation of the currently input local training data.

[0134] The data transmission module 502 is deployed on the multiple local devices and is used to transmit the intermediate output results of each device to the server.

[0135] The result processing module 503 is deployed on the server and is used to process multiple intermediate output results using the server-side model to obtain the device-side model gradient and the server-side model gradient. The server-side model is used to update the parameters of the server-side model, where the server-side model is the remaining part of the large model excluding the device-side model.

[0136] Gradient delivery module 504, deployed on the server, is used to send the device-side model gradient as a shared device-side model gradient among the multiple local devices to each of the multiple local devices.

[0137] The parameter update module 505 is deployed in each of the plurality of local devices and is used to update the parameters of the local device-side model using the device-side model gradient.

[0138] The pruning module 506 is deployed on the server and each of the plurality of local devices, and is used to prune the server-side model at least based on the neuron importance score to obtain a pruned server-side model, and to prune the local device-side model to obtain a pruned device-side model.

[0139] Training module 507, deployed on the multiple local devices and the server, is used to continue collaborative training based on the pruned server-side model and the pruned device-side model to obtain a fully trained large model;

[0140] The trained large model is used to perform multimodal generation tasks, including text generation, image generation, video generation, and audio generation.

[0141] Optionally, the large model includes multiple neurons, and the parameters include at least: the weights of the neurons; the neuron importance score is used to prune the neurons;

[0142] The neuron importance score s of the j-th neuron in the i-th layer. i,j It is determined based on the absolute values ​​of the weights of all neurons in the (i+1)th layer connected to it, and the neuron importance scores of all neurons in the (i+1)th layer connected to it.

[0143] Optionally, the pruning module 506 includes:

[0144] A pruning submodule is deployed on the server and each of the plurality of local devices to prune the server-side model based on the neuron importance score and the weight importance score to obtain a pruned server-side model, and to prune the local device-side model to obtain a pruned device-side model.

[0145] The weight importance score is used to prune the edges connecting two neurons; the weight importance score connecting the j-th neuron in the i-th layer and the k-th neuron in the (i+1)-th layer is... It is determined based on the absolute value of the weight connecting the j-th neuron in the i-th layer and the k-th neuron in the (i+1)-th layer, and the neuron importance score of the k-th neuron in the (i+1)-th layer.

[0146] Optionally, the local training data includes multiple data samples, each data sample including input data and corresponding labels; the data transmission module 502 includes:

[0147] A data transmission submodule, deployed on the multiple local devices, is used to transmit the intermediate output results and the tags corresponding to the intermediate output results to the server via a wireless channel.

[0148] The result processing module 503 includes:

[0149] The result prediction module, deployed on the server, is used to predict the multiple intermediate output results using the server-side model to obtain the prediction result;

[0150] A loss calculation module, deployed on the server, is used to calculate the loss corresponding to each local device based on the prediction results and the tags transmitted by each local device.

[0151] The gradient aggregation module, deployed on the server, is used to average the multiple losses corresponding to multiple local devices and then calculate the partial derivatives to obtain the aggregated gradient.

[0152] A gradient determination module, deployed on the server, is used to obtain the device-side model gradient and the server-side model gradient based on the aggregated gradient.

[0153] Optionally, the result prediction module includes:

[0154] The result splicing module, deployed on the server, is used to splice the multiple intermediate output results based on the number of intermediate output results transmitted by each local device and the activation dimension of the server-side model to obtain an intermediate splicing result.

[0155] The result determination module, deployed on the server, is used to predict the intermediate splicing result using the server-side model to obtain the prediction result.

[0156] Optionally, in each round of collaborative training, the pruning process of the device-side model and the server-side model is guided by the neuron importance score propagating from the output layer to the input layer of the large model. As the collaborative training continues, the number of neurons pruned in each round of collaborative training gradually decreases.

[0157] Optionally, the system 500 further includes:

[0158] A model segmentation module, deployed on the server, is used to segment the large model according to its functional structure to obtain the device-side model and the server-side model, wherein the server-side model is the last part of the large model.

[0159] The model deployment module, deployed on the server, is used to deploy the device-side model to multiple local devices and the server-side model to the local server.

[0160] Optionally, the large model is a multimodal large model, which includes: a modal encoder, an input projector, a base large model, an output projector, and a modal generator;

[0161] The modal encoder is considered as an input layer for processing specific modal data. The modal encoder is a device-side model deployed on the local device, used to process the specific modal data collected on the local device into a feature representation that conforms to the input to the base model. The specific modal data includes at least: image data, text data, speech data, and audio data.

[0162] The input projector, the pedestal large model, the output projector, and the modality generator are deployed to an edge server for higher-level feature extraction and generation than processing the specific modality data.

[0163] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the large-model distributed fine-tuning federated segmentation learning method for wireless edge networks as described in any of the above embodiments of the present invention.

[0164] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0165] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0166] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0171] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0172] The foregoing has provided a detailed description of a large-model distributed fine-tuning federated segmentation learning method, system, and medium for wireless edge networks provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A distributed fine-tuning federated segmentation learning method for large-scale models in wireless edge networks, characterized in that, The method includes: Each of the multiple local devices inputs local training data into its local device-side model to obtain intermediate output results; each device-side model is a part of a larger model, and the local training data of the multiple local devices includes: text training data, image training data, video training data, and audio training data, and the intermediate output results represent the device-side model's understanding and representation of the currently input local training data; The multiple local devices transmit their respective intermediate output results to the server; The server uses the server-side model to process multiple intermediate output results to obtain the device-side model gradient and the server-side model gradient. The server-side model gradient is used to update the parameters of the server-side model. The server-side model is the part of the large model other than the device-side model. The server sends the device-side model gradient as a shared device-side model gradient among the multiple local devices to each of the multiple local devices. Each of the plurality of local devices updates the parameters of its local device-side model using the device-side model gradient; Based at least on the neuron importance score, the server prunes the server-side model to obtain a pruned server-side model, and each of the plurality of local devices prunes its local device-side model to obtain a pruned device-side model. The multiple local devices and the server continue to train collaboratively based on the pruned server-side model and the pruned device-side model to obtain a fully trained large model. The trained large model is used to perform multimodal generation tasks, including text generation, image generation, video generation, and audio generation.

2. The method according to claim 1, characterized in that, The large model comprises multiple neurons, and the parameters include at least: the weights of the neurons; the neuron importance score is used to prune the neurons; The neuron importance score s of the j-th neuron in the i-th layer. i ,j It is determined based on the absolute values ​​of the weights of all neurons in the (i+1)th layer connected to it, and the neuron importance scores of all neurons in the (i+1)th layer connected to it.

3. The method according to claim 2, characterized in that, The process of pruning the server-side model based at least on neuron importance scores to obtain a pruned server-side model, and pruning the local device-side model by each of the plurality of local devices to obtain a pruned device-side model, includes: Based on the neuron importance score and weight importance score, the server prunes the server-side model to obtain a pruned server-side model, and each of the plurality of local devices prunes its local device-side model to obtain a pruned device-side model. The weight importance score is used to prune the edges connecting two neurons; the weight importance score connecting the j-th neuron in the i-th layer and the k-th neuron in the (i+1)-th layer is... It is determined based on the absolute value of the weight connecting the j-th neuron in the i-th layer and the k-th neuron in the (i+1)-th layer, and the neuron importance score of the k-th neuron in the (i+1)-th layer.

4. The method according to claim 1, characterized in that, The local training data includes multiple data samples, each of which includes input data and corresponding labels; The multiple local devices transmit their respective intermediate output results to the server, including: The multiple local devices transmit their respective intermediate output results and the corresponding tags of the intermediate output results to the server via a wireless channel; The server processes multiple intermediate output results using a server-side model to obtain the device-side model gradient and the server-side model gradient, including: The server uses the server-side model to predict the multiple intermediate output results to obtain the prediction result; The server calculates the loss for each local device based on the prediction results and the tags transmitted by each local device. The server averages the multiple losses corresponding to each of the multiple local devices and then calculates the partial derivatives to obtain the aggregated gradient. The server obtains the device-side model gradient and the server-side model gradient based on the aggregated gradient.

5. The method according to claim 4, characterized in that, The server uses the server-side model to predict the multiple intermediate output results, obtaining prediction results including: The server concatenates the multiple intermediate output results based on the number of intermediate output results transmitted by each local device and the activation dimension of the server-side model to obtain an intermediate concatenation result. The server uses the server-side model to predict the intermediate splicing result to obtain the prediction result.

6. The method according to any one of claims 1 to 5, characterized in that, In each round of collaborative training, the pruning process of the device-side model and the server-side model is guided by the neuron importance score propagating from the output layer to the input layer of the large model. As collaborative training continues, the number of neurons pruned in each round of collaborative training gradually decreases.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The server divides the large model according to its functional structure to obtain the device-side model and the server-side model, with the server-side model being the last part of the large model. The server deploys the device-side model to multiple local devices and the server-side model to the server itself.

8. The method according to any one of claims 1 to 5, characterized in that, The large model is a multimodal large model, which includes: a modal encoder, an input projector, a base large model, an output projector, and a modal generator; The modal encoder is considered as an input layer for processing specific modal data. The modal encoder is a device-side model deployed on the local device, used to process the specific modal data collected on the local device into a feature representation that conforms to the input to the base model. The specific modal data includes at least: image data, text data, speech data, and audio data. The input projector, the pedestal large model, the output projector, and the modality generator are deployed to an edge server for higher-level feature extraction and generation than processing the specific modality data.

9. A distributed fine-tuning federated segmentation learning system for large-scale models in wireless edge networks, characterized in that, The system includes: The data processing module is deployed on each of the multiple local devices to input local training data into the local device-side model and obtain intermediate output results. Each device-side model is a part of a larger model. The local training data of the multiple local devices includes: text training data, image training data, video training data, and audio training data. The intermediate output results represent the device-side model's understanding and representation of the currently input local training data. A data transmission module, deployed on the multiple local devices, is used to transmit the intermediate output results of each device to the server. The result processing module, deployed on the server, is used to process multiple intermediate output results using the server-side model to obtain the device-side model gradient and the server-side model gradient, and to update the parameters of the server-side model using the server-side model gradient. The server-side model is the remaining part of the large model excluding the device-side model. A gradient delivery module, deployed on the server, is used to send the device-side model gradient as a shared device-side model gradient among the multiple local devices to each of the multiple local devices. A parameter update module is deployed on each of the plurality of local devices to update the parameters of the local device-side model using the device-side model gradient; A pruning module, deployed on the server and each of the plurality of local devices, is used to prune the server-side model at least based on neuron importance scores to obtain a pruned server-side model, and to prune the local device-side model to obtain a pruned device-side model. The training module, deployed on the multiple local devices and the server, is used to continue collaborative training based on the pruned server-side model and the pruned device-side model to obtain a fully trained large model. The trained large model is used to perform multimodal generation tasks, including text generation, image generation, video generation, and audio generation.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the large-model distributed fine-tuning federated segmentation learning method for wireless edge networks as described in any one of claims 1 to 8.

Images