Model training method, electronic equipment and computer readable storage medium

Abstract

The invention provides a model training method, electronic equipment and a computer readable storage medium, the method can comprise participant equipment applied to a federated learning system, the model training method comprises the steps that model parameters of a local model are updated based on Nth model parameters to obtain N + 1th local model parameters, N is a positive integer, and N is a positive integer; the integer is a positive integer meeting a set rule; decomposing the (N + 1) th local model parameter to obtain at least two (N + 1) th sub-model parameters; sending the at least two (N + 1) th sub-model parameters to a central server, so that the central server performs aggregation processing on the received model parameters uploaded by each participant device to obtain (N + 1) th model parameters; receiving an (N + 1) th model parameter sent by the central server, and updating according to the (N + 1) th model parameter to obtain an (N + 2) th local model parameter; and repeating the steps until the convergence or updating times of the local model are greater than a set times threshold.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular, to a model training method, an electronic device, and a computer-readable storage medium. Background Art

[0002] User group classification has become the key for enterprises and research institutions to gain in-depth understanding of market demand and optimize personalized services. User data in various industries are distributed among various manufacturers. In order to better protect user data privacy, the traditional centralized data processing method to implement model training is not suitable for this type of environment. The federated learning method can achieve model training based on user data from various manufacturers while protecting the privacy of user data. However, this type of training environment also brings new problems. The amount of data transmitted between manufacturers is relatively large, and the required bandwidth is relatively large, which also affects the efficiency of model training. Summary of the invention

[0003] The purpose of this application is to provide a model training method, an electronic device and a computer-readable storage medium, which can reduce the bandwidth required in the training data process and improve the efficiency of model training.

[0004] In a first aspect, the present application provides a model training method, which is applied to a participant device of a federated learning system, wherein the participant device is connected to a central server of the federated learning system, and the model training method comprises: updating the model parameters of the local model based on the Nth model parameters to obtain the N+1th local model parameters, wherein N is a positive integer and is a positive integer that satisfies a set rule; decomposing the N+1th local model parameters to obtain at least two N+1th sub-model parameters; sending at least two N+1th sub-model parameters to the central server, so that the central server can aggregate the model parameters uploaded by each participant device to obtain the N+1th model parameters; receiving the N+1th model parameters sent by the central server, and performing model training according to the N+1th model parameters to obtain the N+2th local model parameters; repeating the above steps until the local model converges or the number of updates is greater than the set number threshold.

[0005] In the above-mentioned implementation mode of the present application, the larger parameters that need to be uploaded to the central server can be decomposed into relatively smaller parameters, and then the parameters can be uploaded. This allows the transmission of the parameters to be achieved through a smaller bandwidth, which can reduce bandwidth requirements and thus improve the effective transmission rate of the parameters.

[0006] In an optional embodiment, the decomposing the N+1th local model parameter to obtain at least two N+1th sub-model parameters includes: determining the N+1th singular value for decomposition according to the dimensionality value of the N+1th local model parameter; and decomposing the N+1th local model parameter according to the N+1th singular value to obtain at least two N+1th sub-model parameters.

[0007] In an optional embodiment, determining the N+1th singular value for decomposition based on the dimensional value of the N+1th local model parameter includes: determining the N+1th energy threshold based on the current training progress; determining the N+1th singular value for decomposition based on the N+1th energy threshold and the dimensional value of the N+1th local model parameter.

[0008] In an optional implementation, the energy threshold is determined by the following formula:

[0009] T(t)=T1+(T2-T1)*t:

[0010] Among them, T(t) represents the real-time energy threshold; T1 represents the first set threshold; T2 represents the second set threshold; t represents the progress ratio determined based on the current training progress; the progress ratio represents the ratio of the current training times to the set total training times, the first set threshold and the second set threshold are pre-set values, and the second set threshold is greater than the first set threshold; when the value of t is the progress ratio determined based on the N+1th training, T(t) represents the N+1th energy threshold.

[0011] In an optional embodiment, determining the N+1th singular value for decomposition based on the energy threshold and the dimensionality value of the N+1th local model parameter includes: constructing a singular value inequality based on the N+1th local model parameter, the dimensionality value of the N+1th local model parameter and the N+1th energy threshold; and determining a minimum value based on the singular value inequality, and using it as the N+1th singular value.

[0012] In an optional implementation, the singular value inequality constructed according to the N+1th local model parameter, the dimension value of the N+1th local model parameter, and the N+1th energy threshold is determined by the following formula:

[0013]

[0014] Where K represents the singular value; [σ1, σ2, ..., σ Q] represents the N+1th local model parameter; Q represents the dimensional value of the N+1th local model parameter; T represents the N+1th energy threshold.

[0015] In the above implementation, by selecting a suitable singular value based on the capability threshold, it is possible to achieve smaller parameters decomposed based on the singular value and to achieve more efficient data transmission.

[0016] In an optional embodiment, the local model includes: multiple convolution modules, one or more attention mechanism modules, and one or more fully connected layers; wherein the attention mechanism module is arranged between two adjacent convolution modules; and the multiple convolution modules include kernel ca t convolution kernel.

[0017] In the above implementation, by using kernel ca The t-convolution kernel can improve computational efficiency while better maintaining computational accuracy, thereby improving model training based on federated learning.

[0018] In an optional embodiment, updating the model parameters of the local model based on the Nth model parameters to obtain the N+1th local model parameters includes: using a training process to update the model parameters of the local model based on the Nth model parameters to obtain the N+1th local model parameters; decomposing the N+1th local model parameters to obtain at least two N+1th sub-model parameters includes: using the training process to decompose the N+1th local model parameters to obtain at least two N+1th sub-model parameters; sending at least two copies of the N+1th sub-model parameters to the central server includes: using a communication process to send at least two copies of the N+1th sub-model parameters to the central server.

[0019] In the above implementation of the present application, communication and model training can be achieved through two processes, so that data transmission and model training can be carried out in parallel, thereby improving the efficiency of model training.

[0020] In an optional embodiment, the decomposing the N+1th local model parameters to obtain at least two N+1th sub-model parameters includes: performing privacy processing on the N+1th local model parameters to obtain the N+1th privacy model parameters; and decomposing the N+1th privacy model parameters to obtain at least two N+1th sub-model parameters.

[0021] In the above implementation manner of the present application, privacy processing may be performed on the parameters before they are transmitted, so as to better protect the privacy of the parameters and improve the security of the parameters.

[0022] In a second aspect, the present application provides an electronic device, comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the above method.

[0023] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program executes the steps of the above method when executed by a processor.

[0024] In a fourth aspect, the present application provides a computer program product, wherein the computer program product comprises a computer program, and the computer program implements the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of the interaction between the server and the terminal device provided in the embodiment of the present application;

[0027] Figure 2 A block diagram of an electronic device provided in an embodiment of the present application;

[0028] Figure 3 A flowchart of a model training method provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of the model structure used in the model training method provided in the embodiment of the present application;

[0030] Figure 5 A schematic diagram of a portion of the model structure used in the model training method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application.

[0032] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0033] In today's big data era, in order to meet the needs of different users, various industries usually need to classify user groups to provide different personalized services for different user groups. In order to make user classification more accurate, it is usually necessary to use massive user data for model training to obtain a model that can achieve user classification. However, massive user data usually exists in different manufacturers, and for the privacy of user data, it is usually not recommended to transfer user data from one manufacturer to another. The traditional centralized data processing model can no longer meet the needs of user data privacy. Federated learning, as an emerging distributed machine learning paradigm, allows participants to collaborate on training models without directly exchanging original user data. It can meet the requirements of model accuracy while meeting data privacy protection.

[0034] However, the huge model and redundant parameters have increased the burden on the communication of federated learning, affecting the efficiency of the overall operation.

[0035] Based on this, the present application provides a model training method, an electronic device and a computer-readable storage medium, which can improve the efficiency of model training based on federated learning.

[0036] To facilitate understanding of this embodiment, the operating environment of a model training method disclosed in an embodiment of the present application is first introduced.

[0037] like Figure 1 , which is a schematic diagram of the interaction between the server 110 and the terminal device 120 provided in an embodiment of the present application. The server 110 is connected to one or more terminal devices 120 through a network for data communication or interaction. The server 110 may be a network server, a database server, etc. The terminal device 120 may be a server device. The terminal device 120 may be a personal computer (PC), a tablet computer, a smart phone, a personal digital assistant (PDA), etc. The terminal device 120 may also be a network server, a database server, etc.

[0038] In this embodiment, the server 110 may be used to receive user data sent by each terminal device 120, and implement user classification model training based on the user data. The user classification model may be used to implement user classification.

[0039] The terminal device 120 may be a device of each participant participating in the user classification model training. The terminal device 120 may store user data of each participant. For different application scenarios, each participant is different, and the content of the user data contained by each participant is also different. The participant may provide user data for use by the server to train the model.

[0040] Taking the dining scenario as an example, each participant can be an independent restaurant, a chain restaurant, etc. The user data of the participant can include information such as user ID, user dining frequency, user dining type, dining price, dining distance, etc. Taking the shopping mall scenario as an example, each participant can be a shopping mall in different geographical locations, shopping malls of different brands, shopping malls in different cities, etc. The user data of the participant can include information such as user ID, user shopping frequency, user shopping price range, etc.

[0041] In this embodiment, the operating environment of the server 110 and the terminal device 120 may be a federated learning scenario, and the server 110 may be a central server. Each terminal device 120 may be a participant device storing user data of the participant.

[0042] like Figure 2 , which is a block diagram of an electronic device. The electronic device 200 may include a memory 211 and a processor 213. A person skilled in the art may understand that Figure 2 The structure shown is only for illustration and does not limit the structure of the electronic device 200. For example, the electronic device 200 may further include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations are shown.

[0043] The above-mentioned memory 211 and processor 213 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The above-mentioned processor 213 is used to execute the executable module stored in the memory.

[0044] The memory 211 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 211 is used to store programs, and the processor 213 executes the program after receiving the execution instruction. The method executed by the electronic device 200 defined by the process disclosed in any embodiment of the present application can be applied to the processor 213, or implemented by the processor 213.

[0045] The processor 213 may be an integrated circuit chip with signal processing capability. The processor 213 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), a dedicated integrated circuit, or a processor.

[0046] (Application Specific Integrated Circuit, referred to as ASIC), Field Programmable Gate Array (Field Programmable Gate Array, referred to as FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0047] In this embodiment, the above Figure 1 The server 110 and the terminal device 120 in the operating environment shown may include the Figure 2 The components included in the electronic device 200 shown in FIG. 1 . Of course, based on different actual needs, the server 110 may also include more components than the electronic device 200, for example, the server 110 may also include a communication unit. The terminal device 120 may also include more components than the electronic device 200, for example, the terminal device 120 may also include a display unit, a positioning unit, a communication unit, and other components.

[0048] The electronic device 200 in this embodiment can be used to execute each step in each method provided in the embodiments of the present application. The implementation process of the model training method and the user identification method is described in detail below through several embodiments.

[0049] See also Figure 3 , is a flow chart of the model training method provided in the embodiment of the present application. The model training method provided in the embodiment of the present application can be applied to a terminal device, and the steps in the model training method are performed by the terminal device. The terminal device can be a participant device of the federated learning system, and the participant device is connected to the central server of the federated learning system. Figure 3 The specific process shown is described in detail.

[0050] Step 310: Update the model parameters of the local model based on the Nth model parameters to obtain the N+1th local model parameters.

[0051] Wherein, N is a positive integer that satisfies the set rule. When each training number is a positive integer that satisfies the set rule, the local model parameters can be processed and sent to the central server, and the central server aggregates the received model parameters.

[0052] In order to improve the training efficiency while meeting the training requirements, it is not necessary to send the local model parameters to the central server for aggregation every time they are updated. The frequency of interacting with the central server for model parameters can be set in advance, so as to improve the training efficiency while meeting the training requirements. The above setting rule can be the frequency of interacting with the central server for model parameters. For example, the model parameters can be interacted with the central server once every ten trainings, and the above N can be a multiple of ten. For example, the model parameters can be interacted with the central server once every fifty trainings, and the above N can be a multiple of fifty. Based on the actual training needs, the setting rule can be set adaptively.

[0053] Of course, in order to make the trained model more in line with the data rules of each participant, it can also be sent to the central server for aggregation after each update of the local model parameters.

[0054] In this embodiment, when the current number of training times meets the above-mentioned setting rules, the local model parameters need to be processed in the subsequent steps 330 and 340. If the current number of training times is not the number of times that meets the setting rules, steps 330 and 340 may not be performed, and the next round of model training and model parameter update may be performed directly.

[0055] Exemplarily, when N is 1, the Nth model parameter is a preset initial parameter.

[0056] When N is a positive integer greater than 1, if N-1 is a positive integer that satisfies the set rules, then the Nth model parameters may be model parameters obtained from the central server; if N-1 is not a positive integer that satisfies the set rules, then the Nth model parameters are updated model parameters obtained from the N-1th model training.

[0057] The local model can be a model trained by the participant device based on the actual user identification and user classification needs. Exemplarily, the local model can be a classification model for classifying users. In one instance, the participant device can be a data processing device of a catering merchant, and the classification model can be used as a model for classifying the user's diet type. In one instance, the participant device can be a data processing device of a catering merchant, and the classification model can be used as a model for classifying the user's diet type. In one instance, the participant device can be a data processing device of a food delivery platform, and the classification model can be used as a model for classifying the dining habits of users of the food delivery platform.

[0058] Step 320 , decomposing the N+1th local model parameters to obtain at least two N+1th sub-model parameters.

[0059] In this embodiment, the local model parameters may be parameters in a matrix form. By decomposing the matrix, the local model parameters with a larger size may be decomposed into a plurality of model parameters with smaller sizes.

[0060] Exemplarily, the N+1th local model parameter may be decomposed into two smaller matrices, and the product of the two smaller matrices is equal to the N+1th local model parameter.

[0061] Step 330, at least two copies of the N+1th sub-model parameters are sent to the central server, so that the central server can aggregate the model parameters uploaded by the devices of each participant to obtain the N+1th model parameters.

[0062] After obtaining the N+1th sub-model parameters, the central server can perform recovery processing to obtain the restored N+1th local model parameters. The central server can recover the local model parameters of each participant and aggregate them to obtain the N+1th model parameters.

[0063] In this embodiment, if the data sent by each participant to the central server are decomposed, the central server can restore the parameters sent by each participant to obtain the model parameters of each participant.

[0064] Exemplarily, the central server may use the FedAvg algorithm to aggregate the local model parameters sent by each participating device.

[0065] Step 340, receiving the N+1th model parameters sent by the central server, and performing model training according to the N+1th model parameters to obtain the N+2th local model parameters.

[0066] If N+2 satisfies the set rules, the N+2th local model parameters can be processed from step 320 to step 340. If N+2 does not satisfy the set rules, the latest local model determined based on the N+2th local model parameters can be trained to obtain the N+3th local model parameters.

[0067] Repeat the above steps until the local model converges or the number of updates exceeds the set threshold.

[0068] Through the above steps, by decomposing a large number of model parameters into parameters of smaller size, the transmission of model parameters can be achieved while reducing bandwidth requirements, thereby improving the success rate of model parameter transmission and thus improving the efficiency of model parameter transmission.

[0069] In order to maintain the uniformity of standards of all participants and the central server, the decomposition standards can be unified and can be decomposed based on singular values. The singular value can be determined based on the dimension value of the local model parameter. Based on this, the above step 320 may include: determining the N+1th singular value for decomposition according to the dimension value of the N+1th local model parameter; decomposing the N+1th local model parameter according to the N+1th singular value to obtain at least two N+1th sub-model parameters.

[0070] Exemplarily, the local model parameters may be decomposed into three sub-model parameters, and the singular value may be used as the number of rows and columns of one of the three sub-model parameters.

[0071] With the i-th local parameter g i is decomposed into an example, where g i It can be a gradient matrix with P rows and Q columns. i =U i V i W i , where U i ∈R PxH 、V i ∈R KxK , W i ∈R KxQ , K represents the singular value. U i 、V i , W i They represent the parameters of the three-molecule model obtained after decomposition.

[0072] Exemplarily, the above-mentioned N+1th singular value can be determined in the following manner: determine the N+1th energy threshold according to the current training progress; determine the N+1th singular value used for decomposition according to the N+1th energy threshold and the dimensionality value of the N+1th local model parameter.

[0073] The energy threshold is determined by the following formula: T(t)=T1+(T2-T1)*t;

[0074] Wherein, T(t) represents the real-time energy threshold; T1 represents the first set threshold; T2 represents the second set threshold; t Indicates the progress ratio determined based on the current training progress.

[0075] The first set threshold and the second set threshold are pre-set values, and the second set threshold is greater than the first set threshold. The first set threshold may represent the minimum value that the energy threshold can take, and the second set threshold may represent the maximum value that the energy threshold can take.

[0076] The progress ratio represents the ratio of the current training times to the set total training times, and the value range of the progress ratio is a value in the range of (0, 1). In an example, the current training times is 100, and the total training times is set to 1000, then the progress ratio corresponding to the current training progress can be 0.1. In an example, the current training times is 500, and the total training times is set to 10000, then the progress ratio corresponding to the current training progress can be 0.05.

[0077] Among them, when the value of t is the progress ratio determined based on the N+1th training, T(t) represents the N+1th energy threshold.

[0078] Determining the N+1th singular value based on the energy threshold may include: constructing a singular value inequality based on the N+1th local model parameter, the dimension value of the N+1th local model parameter and the N+1th energy threshold; and determining the minimum value based on the singular value inequality and using it as the N+1th singular value.

[0079] Exemplarily, constructing the singular value inequality is determined by the following formula:

[0080]

[0081] Where K represents the singular value; [σ1, σ2, ..., σ Q ] can represent the N+1th local model parameter; Q represents the dimensional value of the N+1th local model parameter; T represents the N+1th energy threshold.

[0082] Through the above processing logic, the local model parameters are decomposed based on singular values, so that they can be converted into parameters with smaller data volume, which means that less bandwidth is required for their transmission.

[0083] In order to improve the security of the transmitted model parameters, the model parameters may also be subjected to privacy processing. Based on this, the above step 320 may include: performing privacy processing on the N+1th local model parameters to obtain the N+1th privacy model parameters; performing decomposition processing on the N+1th privacy model parameters to obtain at least two N+1th sub-model parameters.

[0084] Optionally, the N+1th privacy model parameters may be obtained by superimposing the set noise on the N+1th local model parameters.

[0085] The above-mentioned set noise may be Gaussian noise.

[0086] Gaussian noise can be noise that satisfies the Gaussian distribution, and the noise that satisfies the Gaussian distribution can be calculated and determined in the following way: determine the sensitive parameters based on the number of samples in the training data used to train the local model and the preset gradient clipping coefficient; calculate the variance of the noise using the sensitive parameters, the preset relaxation factor and the preset privacy budget; and determine the noise that satisfies the Gaussian distribution based on the variance.

[0087] Among them, different participants may have different numbers of samples in their training data used to train local models.

[0088] The preset gradient clipping coefficient is a hyperparameter, and the specific value of the preset gradient clipping coefficient can be a value set according to actual user experience.

[0089] The above sensitive parameters can be determined by the following formula:

[0090]

[0091] in, Δ f represents a sensitive parameter, D k represents the number of samples in the training data used by the kth participant to train its local model; C represents the preset gradient clipping coefficient.

[0092] The above-mentioned preset relaxation factor and preset privacy budget are hyperparameters, and the specific values ​​of the preset relaxation factor and the preset privacy budget may be values ​​set according to actual user experience.

[0093] In one example, the variance can be determined by the following formula:

[0094]

[0095] Among them, σ represents the standard deviation; ε represents the preset privacy budget; δ represents the preset relaxation factor.

[0096] The square of the standard deviation σ can represent the above variance σ 2.

[0097] Optionally, the value of a specified point in the Gaussian distribution obtained based on the above variance may be selected as the noise satisfying the Gaussian distribution.

[0098] Exemplarily, the value of the zero point can be selected as the noise satisfying the Gaussian distribution. The above-mentioned noise satisfying the Gaussian distribution can be expressed as: N (0, σ 2).

[0099] Of course, based on different actual needs, other point values ​​may also be selected as the noise satisfying the Gaussian distribution, for example, the horizontal coordinates may be -1, 1, 3, 5, etc.

[0100] Taking the added noise as noise that satisfies Gaussian distribution as an example, the participating device can determine the privacy model parameters in the following way:

[0101]

[0102] Among them, N(0,σ 2 ) represents the noise that satisfies Gaussian distribution; and The distribution is represented by the weight parameters in the model parameters updated after the t-th and t+1-th training of the k-th participant; F k (ω k ) represents the latest loss function of the kth participant; represents the latest local model gradient in the model parameters of the kth participant; represents the gradient operator; R represents the learning rate of the gradient descent method.

[0103] Optionally, the above-mentioned step 320 may include: decomposing the N+1th local model parameters to obtain at least two N+1th decomposition model parameters; and performing privacy processing on each N+1th decomposition model parameter to obtain at least two N+1th sub-model parameters.

[0104] In this embodiment, the method of performing privacy processing on the N+1th decomposition model parameters is the same as the method of performing privacy processing on the N+1th local model parameters mentioned above. For details, please refer to the above description and will not be repeated here.

[0105] In the above implementation, by adding noise to the model parameters to achieve privacy processing, the privacy of the parameters of the participants can be better protected, the security of the parameters can be improved, and the possible leakage risk during the transmission of the model parameters can be reduced.

[0106] In order to make the obtained user classification model more accurate in classification, the initial training model can be designed so that it can better extract features. Figure 4 As shown, the initial classification model includes: multiple convolution modules, one or more attention mechanism modules, and one or more fully connected layers.

[0107] Among them, the attention mechanism module is set between two adjacent convolution modules; the output data of the last convolution module among multiple convolution modules is used as the input data of one or more fully connected layers.

[0108] The convolution module provided in this embodiment includes kernel ca t convolution kernel.

[0109] For example, the kernel ca The calculation formula of the convolution kernel t can be expressed as:

[0110] kernel cat :F(x)=G(Cat(w t (x)));

[0111] Among them, F(x) represents the output of the convolutional layer, G(x) represents point-by-point convolution, and w i Represents layer-by-layer convolution, and i represents a hyperparameter used to control the convolution kernel conversion.

[0112] exist Figure 4 In the example shown, the initial training model includes three convolution modules, two attention mechanism modules, and two fully connected layers. Among them, the attention mechanism module is set between the two convolution modules, and the fully connected layer is set after the convolution module. Therefore, the data flow of the initial training model is the first convolution module Cm1, the first attention mechanism module Se1, the second convolution module Cm2, the second attention mechanism module Se2, the third convolution module Cm3, the first fully connected layer Fc1, and the second fully connected layer Fc2.

[0113] In this embodiment, each convolution module includes at least two identical convolution layers. For example, each convolution module includes two identical convolution layers.

[0114] Optionally, the attention mechanism module may include: at least two convolution units, a compression unit connected to the convolution unit, an excitation unit connected to the compression unit, and an activation unit connected to the excitation unit.

[0115] like Figure 5 As shown, the attention mechanism module may include two convolution units (divided into cm1 and cm2), a compression unit Cu connected to the convolution unit, an excitation unit Iu connected to the compression unit, and an activation unit Au connected to the excitation unit. The data flow of the attention mechanism module is the first convolution unit cm1, the second convolution unit cm2, the compression unit Cu, the excitation unit Iu, and the activation unit Au.

[0116] In this embodiment, feature recalibration is achieved through compression, excitation, and reactivation to compensate for the limited receptive field of the convolution module and better capture global correlation information.

[0117] In the above implementation, a convolutional neural network is used as the main body of the deep learning model, and each convolution module can use two identical convolutional layers to extract features. In addition, an attention module is introduced to connect the various convolutional modules, and the "feature recalibration" operation is performed through compression, excitation, and reactivation to make up for the problem of limited receptive field of the convolutional layer and better capture global correlation information. Furthermore, the depth-separable convolution operator is improved, and the kernel_cat convolution kernel is used to replace the kernel_sum convolution kernel in the conventional convolution module, and the depth-separable convolution is improved. The extracted feature maps are spliced ​​and convolved layer by layer using a coupling method of feature splicing, which effectively compresses the model parameters.

[0118] When using the above model training method, since model parameters need to be shared, in the actual training process, the model training and model parameter sharing logic need to be performed separately. Inevitably, during the training process, the model parameters of the central server and other participating devices need to be waited for, which means that the communication of model parameter sharing may interfere with the model training performed by the participating devices. Based on this, the federated learning framework structure can be adjusted to implement model training with dual-process processing logic.

[0119] The above-mentioned step 310 may include: using a training process to update the model parameters of the local model based on the Nth model parameters to obtain the N+1th local model parameters.

[0120] The above-mentioned step 320 may include: using a training process to decompose the N+1th local model parameters to obtain at least two N+1th sub-model parameters;

[0121] The above-mentioned step 330 may include: using a communication process to send at least two copies of the N+1th sub-model parameters to the central server.

[0122] The above step 340 is an action of local model training, and the step 340 can also be executed by the training process.

[0123] In this embodiment, the above-mentioned training process and communication process may be created before model training.

[0124] Exemplarily, the training process and the communication process may be created before the model parameters of the local model are updated for the first time. The training process and the communication process may also be created before the local model is trained. The training process and the communication process may also be created before the training data used for the local model training is processed.

[0125] By implementing the above dual-process approach to model training and model parameter sharing, the impact of model parameter sharing on model training efficiency can be reduced, thereby improving model training efficiency.

[0126] The model training method provided in the embodiment of the present application realizes the training of the user group classification model under the federated learning framework, and provides a parameter sharing mechanism under the framework. In view of the problems existing in the practice of federated learning, such as the large scale of model parameters and huge communication overhead, the model training method provided in the embodiment of the present application decomposes the parameters to be transmitted and transmits the decomposed parameters, which can reduce the bandwidth required for transmitting data and improve the effectiveness of data transmission, thereby optimizing communication efficiency. It not only significantly reduces the volume of model parameters to be transmitted, but also reduces the communication resources required for transmitting model update information between multiple participants, thereby improving the practicality and scalability of the training system provided by federated learning while better meeting the performance of the trained model.

[0127] Furthermore, in an embodiment of the present application, the local model uses a convolution module, which includes a kernel_cat convolution kernel. Compared with a conventional convolution module, it reduces the computational complexity while satisfying the computational accuracy.

[0128] In addition, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model training method described in the above method embodiment are executed.

[0129] The computer program product of the model training method provided in the embodiment of the present application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the model training method described in the above method embodiment. Please refer to the above method embodiment for details, which will not be repeated here.

[0130] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0132] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of more restrictions, the elements defined by the sentence "includes..." do not exclude the presence of other identical elements in the process, method, article or apparatus that includes the elements.

[0133] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0134] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: A participant device applied to a federated learning system, wherein the participant device is connected to a central server of the federated learning system, and the model training method comprises: The model parameters of the local model are updated based on the Nth model parameters to obtain the N+1th local model parameters, where N is a positive integer that satisfies a set rule; Decomposing the N+1th local model parameters to obtain at least two N+1th sub-model parameters; Sending at least two copies of the N+1th sub-model parameters to the central server, so that the central server can aggregate the model parameters uploaded by the devices of each participant to obtain the N+1th model parameters; Receiving the N+1th model parameters sent by the central server, and performing model training according to the N+1th model parameters to obtain the N+2th local model parameters; Repeat the above steps until the local model converges or the number of updates is greater than a set threshold.

2. The method according to claim 1, characterized in that The decomposing the N+1th local model parameter to obtain at least two N+1th sub-model parameters includes: Determining an N+1th singular value for decomposition according to the dimension value of the N+1th local model parameter; The N+1th local model parameter is decomposed according to the N+1th singular value to obtain at least two N+1th sub-model parameters.

3. The method according to claim 2, characterized in that The step of determining the N+1th singular value for decomposition according to the dimension value of the N+1th local model parameter comprises: Determine the N+1th energy threshold according to the current training progress; An N+1th singular value for decomposition is determined according to the N+1th energy threshold and the dimension value of the N+1th local model parameter.

4. The method according to claim 3, characterized in that The energy threshold is determined by the following formula: T(t)=T1+(T2-T1)*t; Wherein, T(t) represents the real-time energy threshold; T1 represents the first set threshold; T2 represents the second set threshold; t represents the progress ratio determined based on the current training progress; the progress ratio represents the ratio of the current training times to the set total training times, the first set threshold and the second set threshold are pre-set values, and the second set threshold is greater than the first set threshold; When the value of t is the progress ratio determined based on the N+1th training, T(t) represents the N+1th energy threshold.

5. The method according to claim 3, characterized in that: The determining, according to the energy threshold and the dimension value of the N+1th local model parameter, the N+1th singular value for decomposition comprises: Constructing a singular value inequality according to the N+1th local model parameter, the dimension value of the N+1th local model parameter, and the N+1th energy threshold; Based on the singular value inequality, a minimum value is determined and used as the N+1th singular value.

6. The method according to claim 5, characterized in that The singular value inequality constructed according to the N+1th local model parameter, the dimension value of the N+1th local model parameter and the N+1th energy threshold is determined by the following formula: Where K represents the singular value; [σ1, σ2, ..., σ Q ] represents the N+1th local model parameter; Q represents the dimensional value of the N+1th local model parameter; T represents the N+1th energy threshold.

7. The method according to any one of claims 1 to 6, characterized in that: The local model includes: multiple convolution modules, one or more attention mechanism modules, and one or more fully connected layers; Wherein, the attention mechanism module is arranged between two adjacent convolution modules; The multiple convolution modules include the kernel_cat convolution kernel.

8. The method according to any one of claims 1 to 6, characterized in that: The updating of the model parameters of the local model based on the Nth model parameters to obtain the N+1th local model parameters includes: using a training process to update the model parameters of the local model based on the Nth model parameters to obtain the N+1th local model parameters; The decomposing the N+1th local model parameter to obtain at least two N+1th sub-model parameters comprises: using the training process to decompose the N+1th local model parameter to obtain at least two N+1th sub-model parameters; The sending at least two copies of the N+1th sub-model parameters to the central server includes: using a communication process to send at least two copies of the N+1th sub-model parameters to the central server.

9. The method according to any one of claims 1 to 6, characterized in that: The decomposing the N+1th local model parameter to obtain at least two N+1th sub-model parameters includes: Performing privacy processing on the N+1th local model parameters to obtain the N+1th privacy model parameters; The N+1th privacy model parameters are decomposed to obtain at least two N+1th sub-model parameters.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method according to any one of claims 1 to 9.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images