Training method and device of federal large model, equipment and storage medium
By adding an adapter network or fast Fourier transform network to the federated large model and keeping other network parameters unchanged during the training process, the problem of poor comprehensiveness of the federated large model when processing data of different data types is solved, achieving better training results.
Patent Information
- Application Number
- CN202411551392.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-03-18
AI Technical Summary
Existing federal big models have poor comprehensive processing when processing data of different data types, resulting in poor training results.
By obtaining the pre-trained federated model, based on the data type of data to be processed by the target terminal, an adapter network or a fast Fourier transform network is added to the model to obtain the adapted federated model, and the network parameters except the adapter and fast Fourier transform network are kept unchanged during the training process.
This improves the comprehensiveness of the data processing of different data types by the federated large models, thereby improving the training effect of the model.
Smart Images
Figure CN119647558A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of model training technology, and in particular to a training method, apparatus, device and storage medium for a federated large model. Background Art
[0002] At present, the training process of large models requires a large amount of useful data and user data privacy protection considerations. Generally, federated learning is used to train large models to obtain the corresponding federated large models.
[0003] However, in the related art, the federated big model is generally trained based on data of a certain data type. Accordingly, the federated big model can only process data of the same data type during use, which may cause the federated big model to be incomplete in processing data of different data types, resulting in poor training effect of the federated big model. Summary of the invention
[0004] The main purpose of this application is to provide a training method, device, equipment and storage medium for a federal big model, aiming to improve the comprehensiveness of the federal big model in processing data of different data types, so as to improve the training effect of the federal big model.
[0005] In a first aspect, the present application provides a method for training a federated large model, comprising:
[0006] Obtaining a first federated large model, where the first federated large model is a pre-trained federated large model;
[0007] According to the data type of the data to be processed of the target terminal, at least one of an adapter network and a fast Fourier transform network is added to the first federated large model to obtain a second federated large model; the data to be processed includes text data or image data;
[0008] The second federal big model is trained according to the data to be processed, and the trained second federal big model is determined as the target federal big model corresponding to the target terminal; wherein, during the training process, the preset network parameters of the networks other than the adapter network and the fast Fourier transform network in the second federal big model are kept unchanged; the target federal big model is used by the target terminal to process the data to be processed.
[0009] In a second aspect, the present application further provides a training device for a federated large model, the training device comprising:
[0010] A model acquisition module, used to acquire a first federated large model, where the first federated large model is a pre-trained federated large model;
[0011] an architecture adjustment module, configured to add at least one of an adapter network and a fast Fourier transform network to the first federated large model according to the data type of the data to be processed of the target terminal, so as to obtain a second federated large model; the data to be processed includes text data or image data;
[0012] A model training module is used to train the second federal big model according to the data to be processed, and determine the trained second federal big model as the target federal big model corresponding to the target terminal; wherein during the training process, the preset network parameters of the network other than the adapter network and the fast Fourier transform network in the second federal big model are kept unchanged; the target federal big model is used by the target terminal to process the data to be processed.
[0013] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor;
[0014] The memory is used to store computer programs;
[0015] The processor is used to execute the computer program and implement the steps of the training method of the federated large model as described above when executing the computer program.
[0016] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the training method of the federated large model as described above are implemented.
[0017] The present application provides a training method, device, equipment and storage medium for a federal big model, the training method comprising: obtaining a first federal big model, the first federal big model being a pre-trained federal big model; adding at least one of an adapter network and a fast Fourier transform network to the first federal big model according to the data type of the data to be processed of the target terminal, to obtain a second federal big model; the data to be processed comprises text data or image data; the second federal big model is trained according to the data to be processed, and the trained second federal big model is determined as the target federal big model corresponding to the target terminal; wherein, during the training process, the preset network parameters of the network other than the adapter network and the fast Fourier transform network in the second federal big model are kept unchanged; the target federal big model is used for the target terminal to process the data to be processed, so as to improve the comprehensiveness of the federal big model in processing data of different data types, so as to improve the training effect of the federal big model.
[0018] For example, in the process of training the federated large model, the network architecture of the federated large model can be adjusted according to the data type of the data to be processed of the target terminal, such as adding at least one of an adapter network and a fast Fourier transform network to the first federated large model. The data types of the data to be processed of different target terminals can be different, such as the data type of text data is text, the data type of image data is image, and the data types of text data and image data are different. During the training process, the federated large model can learn how to process the data to be processed of different data types according to the data to be processed of different data types, and obtain different target federated large models corresponding to different target terminals. For example, the federated large model can learn how to process text data based on text data. For another example, the federated large model can learn how to process image data based on image data. Accordingly, different target federated large models can be adapted to process data to be processed of different data types, so that the data processing of the federated large model can be cross-modal, which is conducive to improving the comprehensiveness of the processing of data of different data types by the federated large model, so as to improve the training effect of the federated large model. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0020] Figure 1 It is a flowchart of a method for training a federated large model provided in an embodiment of the present application;
[0021] Figure 2 is a schematic block diagram of a training method for a federated large model involved in an embodiment of the present application;
[0022] Figure 3 This is a schematic diagram of the network architecture of the second federation model involved in an embodiment of the present application;
[0023] Figure 4 It is a schematic block diagram of a training device for a federated large model provided in an embodiment of the present application;
[0024] Figure 5 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0027] The embodiments of the present application provide a method, apparatus, device and storage medium for training a federated large model. The training method of the federated large model can be applied to a computer device, which can be a tablet computer, a laptop computer, a desktop computer and the like. It can also be applied to a server, which can be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0028] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0029] See also Figure 1 , Figure 1 It is a flowchart of a method for training a federal large model provided in an embodiment of the present application. It should be noted that the method for training a federal large model provided in an embodiment of the present application can be used in a computer device, and of course can also be used in a server. For example, a computer device can obtain a first federal large model from a server, and perform corresponding processing on the first federal large model according to the training method of the federal large model to obtain a target federal large model. Exemplarily, the computer device can, for example, send the target network parameters of the target federal large model to the server; of course, it is not limited to this and is not limited here. The computer device is also called a target terminal, for example, and is not limited here.
[0030] In specific implementation, the computer device includes but is not limited to: any one of a tablet computer, a laptop computer, and a desktop computer; the server can be a single server or a server cluster, or a cloud server that provides cloud computing services.
[0031] like Figure 1As shown, the training method of the federated large model includes steps S101 to S103.
[0032] Step S101: Obtain a first federal large model, where the first federal large model is a pre-trained federal large model.
[0033] For example, when the federated big model is trained by adopting the federated learning method, the network architecture of the federated big model can be determined by the server side.
[0034] For example, the federated big model may include a natural language processing (CV) big model, a computer vision (CV) big model, and the like, which are not limited here. The NLP big model includes, for example, a generative pre-trained Transformer (GPT) model, a bidirectional encoder representation from Transformers (BERT) model based on Transformer, and the like, which are not limited here. The CV big model includes, for example, a vision transformer (ViT) model based on the transformer architecture, a data-efficient image transformer (DeiT) model, and the like, which are not limited here. The server side can determine different NLP big models and CV big models, and the network architectures they adopt can be the same. Based on this, when determining the network architecture of the federated big model, the server side can design the network architecture of the federated big model as a network architecture that is simultaneously adapted to the natural language processing requirements and the computer vision processing requirements based on the consideration that the federated big model obtained after training can process data of different data types. For example, the network architecture of the federated big model may include a transformer architecture. Of course, the network architecture that is adapted to both natural language processing and computer vision processing needs is not limited to the Transformer architecture and is not restricted here.
[0035] For example, when the network architecture of the federated large model is determined, the server can pre-train the federated large model to obtain the pre-trained federated large model, i.e., the first federated large model. Accordingly, the server can publish the first federated large model to different target terminals. Figure 2 As shown, the server can pre-train the federated large model of the corresponding network architecture according to the preset training data to obtain the first federated large model. The server can send the first federated large model to different target terminals.
[0036] The first federal big model can be used by different target terminals to train the first federal big model based on their local data, such as the data to be processed of the target terminal, so as to subsequently determine the target federal big model corresponding to the target terminal.
[0037] Step S102: according to the data type of the data to be processed of the target terminal, at least one of an adapter network and a fast Fourier transform network is added to the first federated large model to obtain a second federated large model; the data to be processed includes text data or image data.
[0038] For example, the text data included in the data to be processed may be determined based on text information such as emails and short messages acquired by the target terminal. The image data included in the data to be processed may be determined based on image information such as videos and pictures acquired by the target terminal.
[0039] During the training of the federated model, the target terminal can also continue to send and receive emails, pictures, etc., so the data to be processed of the target terminal will change. Based on this, the data to be processed can include both the current local data of the target terminal before training the federated model and the newly added local data of the target terminal during or after the training of the federated model is completed.
[0040] In some embodiments, the target terminal may obtain the file format corresponding to the data to be processed, and determine the data type of the data to be processed according to the file format corresponding to the data to be processed. For example, when the file format corresponding to the data to be processed is jpg, the target terminal may determine that the data type of the data to be processed is image. For another example, when the file format corresponding to the data to be processed is eml, the target terminal may determine that the data type of the data to be processed is text. Of course, the file format corresponding to the data to be processed is not limited to this, and is not limited here.
[0041] The data types of the data to be processed of different target terminals may be different. Before the target terminal trains the first federal large model based on the data to be processed, the target terminal may fine-tune the network architecture of the first federal large model so that the first federal large model after fine-tuning the network architecture can adapt to the processing requirements of the data to be processed of the data type corresponding to the target terminal. Based on this, the target terminal can determine the data type of the data to be processed, and fine-tune the network architecture of the first federal large model based on the data type of the data to be processed, such as adding at least one of an adapter network and a fast Fourier transform network to the first federal large model to obtain a second federal large model. Figure 2As shown, when the target terminal receives the first federal large model sent by the server, it can fine-tune the network architecture of the first federal large model according to the data type of the data to be processed to obtain the second federal large model, and the second federal large model can adapt to different training needs. Training needs include, for example, object and sentiment classification, text and image generation, etc., which are not limited here. Accordingly, the target terminal can train the second federal large model based on the data to be processed, and determine the trained second federal large model as the target federal large model corresponding to the target terminal. The target federal large model can be deployed on the target terminal for the target terminal to subsequently process the data to be processed.
[0042] Exemplarily, the first federated large model includes a multi-head self-attention network and a multi-layer perceptron network. Figure 3 As shown, both the multi-head self-attention network and the multi-layer perceptron network can be provided with corresponding summation and normalization layers. The input end of the multi-head self-attention network is used to receive the data to be processed of the target terminal, and the output end of the multi-head self-attention network is connected to the summation and normalization layer corresponding to the multi-head self-attention network. The input end of the multi-layer perceptron network is connected to the summation and normalization layer corresponding to the multi-head self-attention network, and the output end of the multi-layer perceptron network is connected to the summation and normalization layer corresponding to the multi-layer perceptron network.
[0043] For example, the multi-head self-attention network can apply the multi-head self-attention mechanism to the input data of the multi-head self-attention network, such as the data to be processed of the target terminal, to capture the global dependency of the data to be processed. Correspondingly, the multi-layer perceptron network can perform feature extraction and nonlinear transformation on the data to be processed after being processed by the multi-head self-attention network.
[0044] In some implementations, when the data type of the data to be processed is text, an adapter network is added to the output end of the multilayer perceptron network.
[0045] For example, when the data type of the data to be processed is text, the target terminal can add an adapter network, such as an adapter network, to the output end of the multi-layer perceptron network based on the text processing requirements to determine the second federated large model. The adapter network can be used to learn the feature representation of the data to be processed after being processed by the multi-head self-attention network and the multi-layer perceptron network to improve the processing performance of the federated large model for text.
[0046] like Figure 3 As shown, the adapter network may include a downsampling layer, a nonlinear activation function layer, and an upsampling layer. The input end of the downsampling layer is connected to the output end of the multilayer perceptron network, and the output end of the downsampling layer is connected to the input end of the nonlinear activation function layer. The input end of the upsampling layer is connected to the output end of the nonlinear activation function layer.
[0047] The downsampling layer can perform dimensionality reduction processing on the data to be processed after being processed by the multi-head self-attention network and the multi-layer perceptron network to reduce the data dimension, thereby reducing the computational complexity. The nonlinear activation function layer can perform nonlinear transformation on the output data of the downsampling layer to introduce nonlinear capabilities. Correspondingly, the upsampling layer can perform data dimension recovery processing on the output data of the nonlinear activation function layer to maintain the data dimension consistency between the input data of the adapter network and the output data of the adapter network, thereby realizing seamless connection of the network architecture between the downsampling layer, the nonlinear activation function layer and the upsampling layer.
[0048] In an exemplary embodiment, the shallow structure of the downsampling layer includes, for example, a first preset number of convolutional neural networks with a first preset convolutional kernel size. The deep structure of the downsampling layer includes, for example, a second preset number of convolutional neural networks with a first preset convolutional kernel size. The first preset number includes, for example, 2 layers, the second preset number includes, for example, 6 layers, and the second preset convolutional kernel size includes, for example, 3×3, which is not limited here.
[0049] In an exemplary embodiment, the nonlinear activation function layer may be provided with a nonlinear activation function. The nonlinear activation function includes, for example, a hyperbolic tangent (tanh) activation function. The calculation formula of the tanh activation function may be expressed as a function f(x):
[0050]
[0051] Among them, x is used to indicate the input data of the nonlinear activation function layer, such as the data to be processed after being processed by the downsampling layer of the multi-head self-attention network, the multi-layer perceptron network, and the adapter network.
[0052] Of course, the hierarchical architecture of the adapter network may include more or fewer levels, which is not limited here.
[0053] In some embodiments, when the data type of the data to be processed is an image, an adapter network is added to the output end of the multi-layer perceptron network, a first fast Fourier transform network is added between the input end and the output end of the multi-head self-attention network, and a second fast Fourier transform network is added between the input end and the output end of the multi-layer perceptron network.
[0054] When the data type of the data to be processed is an image, the target terminal can, based on image processing requirements, add an adapter network, such as an adapter network, to the output end of the multi-layer perceptron network, add a first fast Fourier transform network between the input end and the output end of the multi-head self-attention network, and add a second fast Fourier transform network between the input end and the output end of the multi-layer perceptron network to determine the second federated large model. The adapter network can be used to learn the feature representation of the data to be processed after being processed by the multi-head self-attention network and the multi-layer perceptron network, and the first fast Fourier transform network and the second fast Fourier transform network can assist the second federated large model to pay attention to global information and local information at the same time, so as to improve the processing performance of the federated large model for images.
[0055] For example, the first fast Fourier transform network is connected between the input and output of the multi-head self-attention network in the form of a residual.
[0056] For example, the second fast Fourier transform network is connected between the input and output ends of the multilayer perceptron network in the form of a residual.
[0057] The hierarchical architecture involved in the first fast Fourier transform network and the second fast Fourier transform network may be the same. Figure 3 As shown, the first fast Fourier transform network and the second fast Fourier transform network both include an imaginary fast Fourier transform layer, a convolution layer, an activation function layer and a real fast Fourier transform layer. The number of convolution layers may include at least one, which is not limited here. Wherein, the imaginary fast Fourier transform layer, the convolution layer, the activation function layer, the convolution layer and the real fast Fourier transform layer can be connected in sequence. Accordingly, in the case where the first fast Fourier transform network is added between the input and output ends of the multi-head self-attention network, the input end of the imaginary fast Fourier transform layer included in the first fast Fourier transform network is connected to the input end of the multi-head self-attention network, and the output end of the real fast Fourier transform layer is connected to the output end of the multi-head self-attention network. In the case where the second fast Fourier transform network is added between the input and output ends of the multi-layer perceptron network, the input end of the imaginary fast Fourier transform layer included in the second fast Fourier transform network is connected to the input end of the multi-layer perceptron network, and the output end of the real fast Fourier transform layer is connected to the output end of the multi-layer perceptron network.
[0058] The imaginary fast Fourier transform layer can convert frequency domain data into time domain data. For example, the imaginary fast Fourier transform layer can convert the input data of the multi-head self-attention network and the input data of the multi-layer perceptron network from the frequency domain to the time domain to obtain the corresponding time domain data, so as to focus on low-frequency features, thereby improving the ability of the multi-head self-attention network and the multi-layer perceptron network to focus on the global model, thereby improving the ability of the federated large model to focus on the global model. The convolution layer can be used to extract local features of the input data of the convolution layer to capture local dependencies in the input data of the convolution layer. For example, the convolution layer can be used to extract local features of the input data of the multi-head self-attention network and the input data of the multi-layer perceptron network to improve the local feature extraction capabilities of the multi-head self-attention network and the multi-layer perceptron network, thereby improving the local feature extraction capabilities of the federated large model. The activation layer can be used to introduce nonlinear factors to improve the expression capabilities of the federated large model. The real fast Fourier transform layer can convert time domain data into frequency domain data. For example, the real fast Fourier transform layer can convert the output data of the multi-head self-attention network and the output data of the multi-layer perceptron network from the time domain to the frequency domain to obtain the corresponding frequency domain data for subsequent processing.
[0059] In an exemplary embodiment, the convolution kernel size of the convolution layer includes, for example, a second preset convolution kernel size. The second preset convolution kernel size includes, for example, 1×1, which is not limited herein.
[0060] In an exemplary embodiment, the fast Fourier transform expressions of the first fast Fourier transform network and the second fast Fourier transform network can be expressed as:
[0061]
[0062] Where X[k] represents the frequency The spectrum at time , j is an imaginary unit, and x[j] represents the sequence of pixel points corresponding to the data to be processed.
[0063] Of course, the hierarchical architecture of the fast Fourier transform network may include more or fewer levels, which is not limited here.
[0064] The functions and setting information of the adapter network can refer to the above description related to the adapter network, which will not be repeated here.
[0065] In this way, by adding at least one of an adapter network and a fast Fourier transform network to the first federal large model according to the data type of the data to be processed of the target terminal, a second federal large model is obtained, and different target terminals can subsequently process their respective data to be processed according to their respective second federal large models. For example, in the case where the data type of the data to be processed of the target terminal is text, the second federal large model corresponding to the target terminal can subsequently be used by the target terminal to process other data to be processed that belong to the same data type as text. In the case where the data type of the data to be processed of the target terminal is an image, the second federal large model corresponding to the target terminal can subsequently be used by the target terminal to process other data to be processed that belong to the same data type as image. Based on this, when the second federal large model corresponding to the target terminal is determined, it is beneficial to subsequently improve the comprehensiveness of the federal large model's processing of data of different data types, thereby helping to improve the training effect of the federal large model.
[0066] Step S103: train the second federal large model according to the data to be processed, and determine the trained second federal large model as the target federal large model corresponding to the target terminal; wherein, during the training process, the preset network parameters of the network other than the adapter network and the fast Fourier transform network in the second federal large model are kept unchanged; the target federal large model is used for the target terminal to process the data to be processed.
[0067] When the network architecture of the first federated large model is fine-tuned according to the data type of the data to be processed to obtain the second federated large model, the target terminal can train the second federated large model according to the data to be processed. For example, the target terminal can train the second federated large model according to the data to be processed to determine the target federated large model corresponding to the target terminal.
[0068] In some embodiments, during the training of the second federated large model, the preset network parameters of the networks other than the adapter network and the fast Fourier transform network in the second federated large model may be frozen to keep the preset network parameters of the networks other than the adapter network and the fast Fourier transform network in the second federated large model unchanged. During the training of the second federated large model, the adapter network and the fast Fourier transform network are parameter-adjustable networks in the second federated large model to reduce the communication or computational overhead generated during the training of the second federated large model and improve the training efficiency of the second federated large model.
[0069] The target terminal can change the preset network parameters of at least one of the adapter network and the fast Fourier transform network according to the data to be processed, until the second federal large model converges to obtain the trained second federal large model. Accordingly, the target terminal can determine the trained second federal large model as the target federal large model corresponding to the target terminal. The target federal large model can be used for the target terminal to process the data to be processed. For example, the data to be processed may include current local data and newly added local data, then the target terminal can train the second federal large model according to the current local data to determine the target federal large model corresponding to the target terminal. Accordingly, the target terminal can perform corresponding processing on the newly added local data according to the target federal large model to obtain an output result. Take the target terminal including terminal 1 and terminal 2 as an example. When the data to be processed of terminal 1 includes text data, such as email data, the target federal large model corresponding to terminal 1 can perform sentiment classification on the email data to obtain a corresponding sentiment classification result. When the data to be processed of terminal 2 includes image data, the target federal large model corresponding to terminal 2 can perform at least one of text generation processing and image recognition processing on the image data to obtain a corresponding recognition result. No limitation is made here.
[0070] In this way, by training the second federal big model with the data to be processed of the target terminal, the target federal big model corresponding to the target terminal can have the processing capability of the data type to be processed corresponding to the target terminal. For different target terminals, when the preset types of the data to be processed of different target terminals are different, the different target federal big models corresponding to different target terminals have the processing capability of the data to be processed of different data types. Based on this, in the process of training the federal big model, different target federal big models can be obtained according to the data to be processed of different data types in combination with the same training process, so that the cross-modality of the data processing of the federal big model can be realized, which is conducive to improving the comprehensiveness of the federal big model in processing data of different data types, thereby helping to improve the training effect of the federal big model.
[0071] Exemplarily, after training the second federal big model to obtain the target federal big model corresponding to the target terminal, the target terminal can upload the target network parameters of the target federal big model to the server side so that the server side can assist the training process of the target federal big models corresponding to other target terminals.
[0072] In some embodiments, the target network parameters of the target federal big model and the data type of the data to be processed of the target terminal are uploaded to the server side, so that the server side determines a network parameter set based on the target network parameters and the data type; the target network parameters are determined based on at least one of the network parameters of the adapter network and the network parameters of the fast Fourier transform network in the target federal big model; the network parameter set includes at least one network parameter to be configured corresponding to each different data type, and the network parameter to be configured is determined based on the target network parameters of the target federal big model.
[0073] Take the case where the target terminals include terminals 1 to 4, the data type of the to-be-processed data of terminals 1 and 3 is text, and the data type of the to-be-processed data of terminals 2 and 4 is image. Terminals 1 to 4 can add at least one of an adapter network and a fast Fourier transform network to the acquired first federated large model according to the data type of their respective to-be-processed data, and obtain the second federated large model corresponding to each of terminals 1 to 4. Terminals 1 to 4 can train their respective corresponding second federated large models according to their respective to-be-processed data, so as to determine the target federated large model corresponding to each of terminals 1 to 4. Accordingly, the to-be-processed data of terminals 1 to 4 can be different, and the target network parameters of the target large models corresponding to each of terminals 1 to 4 can also be different. For example, the network parameters of the adapter network included in the target network parameters corresponding to terminal 1 can be different from the network parameters of the adapter network included in the target network parameters corresponding to terminals 2, terminal 3 and terminal 4. For another example, the network parameters of the fast Fourier transform network included in the target network parameters corresponding to terminal 2 can be different from the network parameters of the fast Fourier transform network included in the target network parameters corresponding to terminal 4. Terminals 1 to 4 can upload the target network parameters of their respective target federated large models and the data types of their respective data to be processed to the server side, so that the server side determines the corresponding network parameter set.
[0074] For example, the server side can set corresponding identification information for the target network parameters corresponding to each target terminal according to the data type of the to-be-processed data of each target terminal. For example, the server side can set the identification information of the target network parameter corresponding to terminal 1 as parameter Text1 according to the data type corresponding to terminal 1 being text. The server side can set the identification information of the target network parameter corresponding to terminal 2 as parameter Image2 according to the data type corresponding to terminal 2 being image. By analogy, the server side can determine the identification information of the target network parameter corresponding to terminal 3 as parameter Text3, and determine the identification information of the target network parameter corresponding to terminal 4 as parameter Image4. A corresponding relationship is established between the identification information of the target network parameter and the target network parameter. Based on this, the server side can determine a network parameter set according to the identification information of the target network parameter corresponding to each target terminal. For example, the network parameter set can be expressed as (Text1, Image2, Text3, Image4). Of course, the network parameter set is not limited to this, and is not limited here.
[0075] When determining the network parameter set, the server side can determine the target network parameters of the target federated large model corresponding to each of the different target terminals involved in the network parameter set as the configuration basis for at least one of the network parameters of the adapter network and the fast Fourier transform network when other target terminals train the acquired federated large model based on their to-be-processed data. The server side can determine the target network parameters corresponding to each of the different target terminals in the network parameter set as the network parameters to be configured. For example, the network parameters to be configured include parameters Text1, Image2, Text3, and Image4. The data types corresponding to parameters Text1 and Text3 are text, and the data types corresponding to parameters Image2 and Image4 are images. The network parameter set can include at least one network parameter to be configured corresponding to different data types.
[0076] In this way, when the network parameter set is determined, other target terminals can be assisted in configuring the network parameters of the federal big model based on the network parameters to be configured included in the network parameter set, so as to facilitate the subsequent training of the federal big model, which is conducive to improving the training effect of the federal big model.
[0077] Exemplarily, the target terminal may train the federated large model based on the network parameters to be configured in the network parameter set and the data to be processed of the target terminal.
[0078] In some implementations, according to the data type of the to-be-processed data of the target terminal, at least one to-be-configured network parameter corresponding to the data type is acquired from a network parameter set provided by the server.
[0079] For example, the second federated large model is trained based on the data to be processed and at least one network parameter to be configured.
[0080] For example, the target terminal includes terminal 5, and the network parameter set includes parameters Text1, Image2, Text3, and Image4. When the data type of the data to be processed by terminal 5 is text, since the data type of terminal 5 is the same as the data type corresponding to parameters Text1 and Text3, terminal 5 can obtain parameters Text1 and Text3 from the network parameter set based on its data type being text. Accordingly, terminal 5 can train the obtained federated large model in combination with the data to be processed by terminal 5, parameters Text1, and Text3.
[0081] Of course, it is not limited to this. For example, the target terminal includes terminal 6, and the network parameter set includes parameter Text1, parameter Image2, parameter Text3, and parameter Image4. When the data type of the data to be processed by terminal 6 is an image, since the data type of terminal 6 is the same as the data type corresponding to parameters Image2 and Image4, terminal 6 can obtain parameter Image2 and parameter Image4 from the network parameter set based on its data type being an image. Accordingly, terminal 6 can train the obtained federated large model in combination with the data to be processed by terminal 6, parameter Image2, and parameter Image4. There is no limitation here.
[0082] In some embodiments, according to the network parameters to be configured, the preset network parameters of at least one of the adapter network and the fast Fourier transform network in the second federal large model are configured to obtain a third federal large model; based on the third federal large model, the data to be processed is processed to obtain a prediction result corresponding to the data to be processed; according to the preset labels corresponding to the data to be processed and the prediction result, the cross entropy loss value corresponding to the network parameter to be configured is determined; from at least one network parameter to be configured, the network parameter to be configured with the smallest cross entropy loss value is determined as the target network parameter of the third federal large model.
[0083] For example, when the data type of the data to be processed by the terminal 5 is text, the terminal 5 can add the adapter network to the acquired first federated large model to obtain the second federated large model. Accordingly, the terminal 5 can configure the preset network parameters of the adapter network in the second federated large model according to the acquired network parameters to be configured, such as parameter Text1 or parameter Text3, to obtain the third federated large model.
[0084] For example, the terminal 5 can configure the preset network parameters of the adapter network in the second federated large model according to the parameter Text1 to obtain the third federated large model. The terminal 5 can input the data to be processed into the third federated large model so that the third federated large model processes the data to be processed and obtains the corresponding prediction result. Accordingly, the terminal 5 can determine the cross entropy loss value corresponding to the parameter Text1 according to the prediction result corresponding to the data to be processed and the preset label corresponding to the data to be processed.
[0085] For another example, the terminal 5 can configure the preset network parameters of the adapter network in the second federal large model according to the parameter Text3 to obtain the third federal large model. The terminal 5 can input the data to be processed into the third federal large model so that the third federal large model processes the data to be processed and obtains the corresponding prediction result. Accordingly, the terminal 5 can determine the cross entropy loss value corresponding to the parameter Text3 according to the prediction result corresponding to the data to be processed and the preset label corresponding to the data to be processed.
[0086] Based on this, the terminal 5 can compare the cross entropy loss values corresponding to the parameters Text1 and Text3, and select the network parameter to be configured with the smallest cross entropy loss value from the parameters Text1 and Text3, and determine it as the target network parameter of the third federal large model. When the target network parameter of the third federal large model is determined, the third federal large model corresponding to the target network parameter can be determined as the target federal large model corresponding to the terminal 5.
[0087] For example, when the data type of the data to be processed by the terminal 6 is an image, the terminal 6 can add an adapter network and a fast Fourier transform network to the acquired first federated large model to obtain a second federated large model. Accordingly, the terminal 6 can configure the preset network parameters of the adapter network and the fast Fourier transform network in the second federated large model according to the acquired network parameters to be configured, such as parameter Image2 or parameter Image4, to obtain a third federated large model.
[0088] For example, the terminal 6 can configure the preset network parameters of the adapter network and the fast Fourier transform network in the second federal large model according to the parameter Image2 to obtain the third federal large model. The terminal 6 can input the data to be processed into the third federal large model so that the third federal large model processes the data to be processed and obtains the corresponding prediction result. Accordingly, the terminal 6 can determine the cross entropy loss value corresponding to the parameter Image2 according to the prediction result corresponding to the data to be processed and the preset label corresponding to the data to be processed.
[0089] For another example, the terminal 6 can configure the preset network parameters of the adapter network and the fast Fourier transform network in the second federal large model according to the parameter Image4 to obtain the third federal large model. The terminal 6 can input the data to be processed into the third federal large model so that the third federal large model processes the data to be processed and obtains the corresponding prediction result. Accordingly, the terminal 6 can determine the cross entropy loss value corresponding to the parameter Image4 according to the prediction result corresponding to the data to be processed and the preset label corresponding to the data to be processed.
[0090] Based on this, terminal 6 can compare the cross entropy loss values corresponding to parameter Image42 and parameter Image4, and select the network parameter to be configured with the smallest cross entropy loss value from parameter Image2 and parameter Image4, and determine it as the target network parameter of the third federal large model. When the target network parameter of the third federal large model is determined, the third federal large model corresponding to the target network parameter can be determined as the target federal large model corresponding to terminal 6.
[0091] In this way, the target terminal can configure the network parameters of the federated big model according to the network parameters to be configured included in the network parameter set and the data type of the data to be processed, and train the federated big model, so that the trained federated big model can be used to process the data to be processed of the corresponding data type. Accordingly, when the data types corresponding to different target terminals are different, the target federated big models corresponding to different target terminals can process the data to be processed of the corresponding data type respectively, which is conducive to improving the comprehensiveness of the federated big model's processing of data of different data types and improving the training effect of the federated big model.
[0092] Exemplarily, before training the second federated large model based on the data to be processed, corresponding data preprocessing may be performed on the data to be processed.
[0093] In some implementations, feature extraction is performed on the data to be processed according to the data type of the data to be processed to obtain feature information corresponding to the data to be processed.
[0094] For example, the feature information is input into the second federated large model to train the second federated large model.
[0095] In the case where the data type of the data to be processed is text, the target terminal may embed the data to be processed through a preset embedding layer, so as to perform feature extraction on the data to be processed and obtain corresponding feature information.
[0096] When the data type of the data to be processed is an image, the target terminal can perform convolution processing on the data to be processed through a preset convolution layer to perform feature extraction processing on the data to be processed to obtain corresponding feature information. The convolution kernel size of the preset convolution layer includes, for example, 3×3, which is not limited here.
[0097] When the feature information applicable to the data to be processed is determined, the target terminal may input the feature information into the second federated large model to train the second federated large model.
[0098] In this way, by preprocessing the data to be processed and obtaining the feature information corresponding to the data to be processed, the feature information corresponding to the data to be processed can be adapted to the form requirements of the second federal large model for the input data, which is conducive to the subsequent improvement of the training effect of the federal large model.
[0099] The training method of the federal big model provided in the above embodiment obtains a first federal big model, which is a pre-trained federal big model; according to the data type of the data to be processed of the target terminal, at least one of an adapter network and a fast Fourier transform network is added to the first federal big model to obtain a second federal big model; the data to be processed includes text data or image data; according to the data to be processed, the second federal big model is trained, and the trained second federal big model is determined as the target federal big model corresponding to the target terminal; wherein, during the training process, the preset network parameters of the network other than the adapter network and the fast Fourier transform network in the second federal big model are kept unchanged; the target federal big model is used for the target terminal to process the data to be processed, so as to improve the comprehensiveness of the federal big model in processing data of different data types, so as to improve the training effect of the federal big model.
[0100] For example, in the process of training the federated large model, the network architecture of the federated large model can be adjusted according to the data type of the data to be processed of the target terminal, such as adding at least one of an adapter network and a fast Fourier transform network to the first federated large model. The data types of the data to be processed of different target terminals can be different, such as the data type of text data is text, the data type of image data is image, and the data types of text data and image data are different. During the training process, the federated large model can learn how to process the data to be processed of different data types according to the data to be processed of different data types, and obtain different target federated large models corresponding to different target terminals. For example, the federated large model can learn how to process text data based on text data. For another example, the federated large model can learn how to process image data based on image data. Accordingly, different target federated large models can be adapted to process data to be processed of different data types, so that the data processing of the federated large model can be cross-modal, which is conducive to improving the comprehensiveness of the processing of data of different data types by the federated large model, so as to improve the training effect of the federated large model.
[0101] See also Figure 4 , Figure 4 This is a schematic block diagram of a training device for a federated large model provided in an embodiment of the present application. The training device can be configured in a server or a computer device to execute the aforementioned training method for a federated large model.
[0102] like Figure 4 As shown, the training device of the federated large model includes: a model acquisition module 110, an architecture adjustment module 120 and a model training module 130.
[0103] The model acquisition module 110 is used to acquire a first federated large model, where the first federated large model is a pre-trained federated large model.
[0104] The architecture adjustment module 120 is used to add at least one of an adapter network and a fast Fourier transform network to the first federal large model according to the data type of the data to be processed of the target terminal, so as to obtain a second federal large model; the data to be processed includes text data or image data.
[0105] The model training module 130 is used to train the second federal big model according to the data to be processed, and determine the trained second federal big model as the target federal big model corresponding to the target terminal; wherein during the training process, the preset network parameters of the network other than the adapter network and the fast Fourier transform network in the second federal big model are kept unchanged; the target federal big model is used by the target terminal to process the data to be processed.
[0106] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0107] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0108] Exemplarily, the above method and apparatus may be implemented in the form of a computer program, which may be run on a computer device.
[0109] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server or an electronic device.
[0110] like Figure 5 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.
[0111] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any training method of the federated large model.
[0112] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0113] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any training method of the federated large model.
[0114] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0115] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0116] In one embodiment, the processor is used to execute a computer program and implement the following steps when executing the computer program:
[0117] Obtaining a first federated large model, where the first federated large model is a pre-trained federated large model;
[0118] According to the data type of the to-be-processed data of the target terminal, at least one of an adapter network and a fast Fourier transform network is added to the first federated large model to obtain a second federated large model;
[0119] The second federal big model is trained according to the data to be processed, and the trained second federal big model is determined as the target federal big model corresponding to the target terminal; wherein, during the training process, the preset network parameters of the networks other than the adapter network and the fast Fourier transform network in the second federal big model are kept unchanged; the target federal big model is used by the target terminal to process the data to be processed.
[0120] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the training of the federal large model described above can refer to the corresponding process in the aforementioned federal large model training method embodiment, and will not be repeated here.
[0121] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the training method of the federated large model of the present application.
[0122] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.
[0123] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0124] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.
[0125] It should also be understood that the non-Company software tools or components appearing in the embodiments of the present application are merely examples and do not represent actual use.
[0126] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for training a federated large model, characterized in that: include: Obtaining a first federated large model, where the first federated large model is a pre-trained federated large model; According to the data type of the data to be processed of the target terminal, at least one of an adapter network and a fast Fourier transform network is added to the first federated large model to obtain a second federated large model; the data to be processed includes text data or image data; The second federated large model is trained according to the data to be processed, and the trained second federated large model is determined as the target federated large model corresponding to the target terminal; wherein, during the training process, the preset network parameters of the networks other than the adapter network and the fast Fourier transform network in the second federated large model are kept unchanged; The target federation large model is used by the target terminal to process the data to be processed.
2. The training method according to claim 1, characterized in that: The first federated large model includes a multi-head self-attention network and a multi-layer perceptron network; The adding at least one of an adapter network and a fast Fourier transform network to the first federated large model according to the data type of the to-be-processed data of the target terminal comprises: When the data type of the data to be processed is text, adding an adapter network at the output end of the multilayer perceptron network; When the data type of the data to be processed is an image, an adapter network is added to the output end of the multi-layer perceptron network, a first fast Fourier transform network is added between the input end and the output end of the multi-head self-attention network, and a second fast Fourier transform network is added between the input end and the output end of the multi-layer perceptron network.
3. The training method according to claim 2, characterized in that: The method of adding a first fast Fourier transform network between the input end and the output end of the multi-head self-attention network comprises: Connecting the first fast Fourier transform network between the input and output of the multi-head self-attention network in the form of a residual; The method of adding a second fast Fourier transform network between the input terminal and the output terminal of the multilayer perceptron network comprises: The second fast Fourier transform network is connected between the input end and the output end of the multilayer perceptron network in the form of a residual.
4. The training method according to any one of claims 1 to 3, characterized in that: After training the second federated large model according to the data to be processed and determining the trained second federated large model as the target federated large model corresponding to the target terminal, the method further includes: The target network parameters of the target federal big model and the data type of the to-be-processed data of the target terminal are uploaded to the server side, so that the server side determines a network parameter set according to the target network parameters and the data type; the target network parameters are determined according to at least one of the network parameters of the adapter network and the network parameters of the fast Fourier transform network in the target federal big model; the network parameter set includes at least one to-be-configured network parameter corresponding to each different data type, and the to-be-configured network parameter is determined according to the target network parameters of the target federal big model.
5. The training method according to claim 4, characterized in that: After obtaining the first federal large model, the method further includes: According to the data type of the to-be-processed data of the target terminal, acquiring at least one to-be-configured network parameter corresponding to the data type from the network parameter set provided by the server; The step of training the second federated large model according to the data to be processed includes: The second federated large model is trained according to the data to be processed and at least one of the network parameters to be configured.
6. The training method according to claim 5, characterized in that: The training of the second federated large model according to the data to be processed and at least one of the network parameters to be configured includes: According to the network parameters to be configured, at least one of the preset network parameters of the adapter network and the fast Fourier transform network in the second federated large model is configured to obtain a third federated large model; Based on the third federal large model, the data to be processed is processed to obtain a prediction result corresponding to the data to be processed; Determine the cross entropy loss value corresponding to the network parameter to be configured according to the preset label corresponding to the data to be processed and the prediction result; From at least one of the network parameters to be configured, the network parameter to be configured with the smallest cross entropy loss value is determined as the target network parameter of the third federal large model.
7. The training method according to any one of claims 1 to 3, characterized in that: Before training the second federated large model according to the data to be processed, the method further includes: According to the data type of the data to be processed, feature extraction processing is performed on the data to be processed to obtain feature information corresponding to the data to be processed; The step of training the second federated large model according to the data to be processed includes: The feature information is input into the second federated large model to train the second federated large model.
8. A training device for a federated large model, characterized in that: The training device comprises: A model acquisition module, used to acquire a first federated large model, where the first federated large model is a pre-trained federated large model; an architecture adjustment module, configured to add at least one of an adapter network and a fast Fourier transform network to the first federated large model according to the data type of the data to be processed of the target terminal, so as to obtain a second federated large model; the data to be processed includes text data or image data; A model training module is used to train the second federal big model according to the data to be processed, and determine the trained second federal big model as the target federal big model corresponding to the target terminal; wherein during the training process, the preset network parameters of the network other than the adapter network and the fast Fourier transform network in the second federal big model are kept unchanged; the target federal big model is used by the target terminal to process the data to be processed.
9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the training method of the federated large model as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for training a federated large model as described in any one of claims 1 to 7 are implemented.